Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Sy03
|
04ae6aa009 | ||
|
Sy03
|
79c629885a | ||
|
Sy03
|
d8458b8417 | ||
|
Sy03
|
ec0feea2bd | ||
|
Sy03
|
daf91b4d7a | ||
|
Sy03
|
40a000de05 | ||
|
Sy03
|
dc3c7dbe16 |
No files matched your search
@@ -1,64 +0,0 @@
|
||||
---
|
||||
name: 3d-animation-short-generator
|
||||
description: Develop stylized 3D narrative shorts through story, character and environment references, timed shot tables, storyboards, generation, and assembly. Use for coherent animated storytelling or its planning stages, not isolated image edits or photorealistic live action.
|
||||
metadata:
|
||||
compatibility: File-based agent workflow with sibling h3-prompt-writing. Media generation and editing require available tools or a configured backend.
|
||||
---
|
||||
|
||||
# 3D Animation Short Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
For a complete film, progress from story to references, shot table, text storyboard,
|
||||
clips, and assembly. Keep these as local artifacts, using existing user decisions
|
||||
and authorization. If only a planning stage is requested, deliver that stage.
|
||||
|
||||
## Story and visual contract
|
||||
|
||||
Capture premise, desired emotional outcome, duration, ratio, model, and audio mode.
|
||||
Use warm stylized 3D as a default only when style is unspecified: readable geometric
|
||||
silhouettes, designed hair clumps, tactile materials, soft skin shading where
|
||||
appropriate, expressive brows/eyes, and elastic posing. User-selected 2D or other
|
||||
styles override this preset rather than receiving accidental 3D texture.
|
||||
|
||||
Create an active protagonist with a want, obstacle, meaningful choice, and payoff.
|
||||
Connect the ending to an earlier visual/emotional anchor. Use as many beats as the
|
||||
duration needs; do not pad a tiny story into an eight-beat formula. Dialogue should
|
||||
change a relationship or move the action, and use the user's requested language.
|
||||
|
||||
## Reference design
|
||||
|
||||
- Character references lock name, silhouette, proportions, face, hair, costume,
|
||||
signature props, and color. Label identity in the production document; clean
|
||||
rendering anchors should not carry labels that could leak into the video.
|
||||
- Scene references contain environments only unless the user requested a composite.
|
||||
Record landmarks, lighting direction/time of day, entrances, exits, and prop state.
|
||||
- A reference board with multiple views is for design review. Prefer appropriate
|
||||
clean views for model input so grids and pose sheets do not become final frames.
|
||||
- When an asset changes, update all affected prompts and clips, with a manifest
|
||||
identifying the current revision.
|
||||
|
||||
## Shot planning
|
||||
|
||||
Read [shot-table-spec.md](references/shot-table-spec.md) for the six-column table
|
||||
and continuity checks. Then read
|
||||
[storyboard-guidelines.md](references/storyboard-guidelines.md) for the matching
|
||||
text storyboard and optional pencil previews. The text storyboard is the source
|
||||
of shot timing; preview art does not silently replace it.
|
||||
|
||||
Vary shot size and elastic performance to serve the story. Camera tilts, close-ups,
|
||||
or exaggeration are choices for specific beats, not quotas. Track off-screen
|
||||
characters and spatial landmarks so they do not teleport back into later shots.
|
||||
|
||||
## Generation and assembly
|
||||
|
||||
Before rendering, read [model-selection.md](references/model-selection.md) and
|
||||
[fallback-policy.md](references/fallback-policy.md). Use the chosen model through
|
||||
available tools. Compile prompts using the H3 base or reference guide as appropriate.
|
||||
Remove storyboard-only labels, arrows, panel borders, and sketch styling from
|
||||
rendering inputs. Tie each clip to the current scene and character references.
|
||||
|
||||
Read [qc-checklist.md](references/qc-checklist.md) for assembly and final review.
|
||||
Keep the planned shot order and audio continuity. Use a continuous score when
|
||||
requested, duck it under dialogue/SFX, and do not add subtitles unasked. Deliver
|
||||
the actual film or the requested planning package, with reproducible prompts and
|
||||
any unresolved identity, scene, or timing defects.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "3D Animation Shorts"
|
||||
short_description: "Plan stories, character anchors, shots, and edits"
|
||||
default_prompt: "Use $3d-animation-short-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,21 +0,0 @@
|
||||
# Targeted generation recovery
|
||||
|
||||
Compare the failed result against the shot's actual identity/spatial anchors.
|
||||
Classify the failure: wrong input or API mapping, reference drift, overloaded
|
||||
action, timing, audio, or a render/export failure. Fix transport/configuration
|
||||
errors before spending another generation attempt.
|
||||
|
||||
For drift, quote the relevant anchors and remove unrelated reference roles. For
|
||||
overloaded action, simplify or split the shot while preserving the requested total
|
||||
duration and handoffs. For a scene change, use the new scene reference and avoid
|
||||
conditioning on a previous scene when that would contradict the new setting.
|
||||
|
||||
Use the retry/resource limit from the shared portable workflow. A persistent
|
||||
failure should lead to a concrete choice (simpler action, a supplied reference,
|
||||
an authorized alternate model, or an explicitly incomplete shot), not automatic
|
||||
provider switching or an endless loop. Never present a placeholder as a finished
|
||||
clip or hide an incorrect scene in final assembly.
|
||||
|
||||
For optional pencil previews, reduce annotation or panel count. Use the text
|
||||
storyboard if visualization is not essential; do not rerender correct video
|
||||
solely because a human-review sketch changed.
|
||||
@@ -1,21 +0,0 @@
|
||||
# Model and format selection
|
||||
|
||||
Honor an existing model choice. Otherwise prefer MiniMax H3 when available, and
|
||||
inspect the running backend for partitions, reference modes, audio, resolutions,
|
||||
duration limits, and long-video options. Do not assume hosted and local services
|
||||
have identical limits, quality, prices, or parameter names.
|
||||
|
||||
If another model is requested, use its real documented input contract. A provider
|
||||
switch needs the user's authorization; never silently use a different service
|
||||
because a preset names it as a fallback. For an authorized mixed-model workflow,
|
||||
record model and settings per shot and check cross-shot style differences.
|
||||
|
||||
Keep delivery resolution separate from native generation resolution. Record actual
|
||||
width/height, fps, frame count, and any crop, retiming, or upscaling. Do not claim
|
||||
native 2K when the video was generated at 768p and restored later.
|
||||
|
||||
Compile shot action, camera, identity, scene, and audio into the chosen model's
|
||||
prompt format. For H3 use the bundled writing guides. Strip production labels and
|
||||
preview grids. Prompt for designed silhouettes, elastic action, material and
|
||||
lighting details when these belong to the user's 3D style; do not add text/UI just
|
||||
because a model can generate them.
|
||||
@@ -1,19 +0,0 @@
|
||||
# Assembly and review
|
||||
|
||||
Use current clip revisions in shot-table order. Preserve useful dialogue and SFX;
|
||||
mix a continuous requested score underneath them. Inspect edits at sound bridges
|
||||
and dialogue boundaries, avoiding cut vowels and accidental audio gaps. Only add
|
||||
subtitles, music, or retiming consistent with the brief.
|
||||
|
||||
Review identity, landmarks, screen direction, light/time of day, prop state,
|
||||
emotional payoff, intelligibility, SFX timing, and music balance. Look for leaked
|
||||
storyboard panels, sketch lines, arrows, labels, timecodes, or watermarks. Verify
|
||||
the complete file's duration, dimensions, fps, frame count, audio, and decode.
|
||||
|
||||
Keep planning documents, references, raw clips, and final edits in distinguishable
|
||||
folders. Record the exact assets used in the final edit. After a character, scene,
|
||||
prompt, clip, or score is revised, update its downstream dependents and archive the
|
||||
previous result; do not silently combine stale and current versions.
|
||||
|
||||
Report observed defects and what was repaired, including external editing or
|
||||
upscaling. A successful API response proves completion, not visual correctness.
|
||||
@@ -1,30 +0,0 @@
|
||||
# Six-column shot table
|
||||
|
||||
| Shot ID & Duration | Continuity Handoff | Reference Anchors (Spatial + Identity) | Hook Type | Shot Description (Per-Second Directives) | Audio & Dialogue Track |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
|
||||
Use stable IDs such as S01. Record incoming and outgoing image state, prop
|
||||
position, eyeline, motion, and emotional state in Continuity Handoff.
|
||||
|
||||
Reference Anchors contains named landmarks and screen-relative positions; each
|
||||
character's screen position, facing and pose; recently exited characters' locations;
|
||||
lighting baseline and modifiers; and exact character/scene asset paths or IDs.
|
||||
An explicit scene/time change can replace these anchors; an accidental flip cannot.
|
||||
|
||||
Hook Type names the shot's purpose: setup, reveal, reversal, joke, suspense,
|
||||
tenderness, chase, or callback. Do not force a joke or cut into a quiet scene.
|
||||
|
||||
For each second or meaningful sub-second beat, cover action/pose/expression,
|
||||
camera, spatial/prop state, sound, and the handoff to the next interval. Use compact
|
||||
phrases rather than inventing movement in intentionally still moments. Include
|
||||
anticipation, squash/stretch, overshoot, and follow-through where appropriate.
|
||||
|
||||
Keep the complete dialogue/narration script in Audio & Dialogue Track, with
|
||||
speaker, timing, delivery, and SFX. Distinguish off-screen narration from speaking
|
||||
characters; a visible listener should not lip-sync the narrator's words.
|
||||
|
||||
Before rendering, check full time coverage, supported generation durations, a
|
||||
readable number of active characters, matching scene/identity references, and
|
||||
coherent row-to-row handoffs. Mark intentional cuts/time jumps. Fix contradictory
|
||||
states rather than using a generic continuity assurance. A visual shot may span
|
||||
multiple generation windows when the verified backend supports continuation.
|
||||
@@ -1,29 +0,0 @@
|
||||
# Text storyboard and optional visualization
|
||||
|
||||
Maintain one `storyboard.md` with a section per shot. Each section contains:
|
||||
|
||||
1. Shot ID, title, duration, and narrative purpose.
|
||||
2. Exact scene and character reference IDs.
|
||||
3. Spatial anchors: landmarks, character positions/facing, off-screen characters,
|
||||
key/fill/rim lighting, and important props.
|
||||
4. Incoming and outgoing continuity state.
|
||||
5. Time-ordered beats: pose/expression, camera, audio, and anchor changes.
|
||||
|
||||
Use per-second or finer intervals only where meaningful; cover the complete shot.
|
||||
For narrated moments, specify whether a visible character speaks or only reacts.
|
||||
Optional ASCII staging diagrams can clarify screen direction but are not images
|
||||
to supply as rendering references.
|
||||
|
||||
For heavy revision of a shot, extract its section to a named file and link it from
|
||||
the main document. Choose one authoritative version; reintegrate or update the
|
||||
link after revision so later generation cannot read stale content.
|
||||
|
||||
Only generate pencil storyboards when requested or already authorized as previews.
|
||||
Use equal-sized time-ordered panels, with timecode, pose/expression, camera arrow,
|
||||
and audio/anchor notes. Suggested layouts: three panels in a strip, four in a 2×2
|
||||
grid, six in 2×3. More complex scenes may need fewer representative key poses.
|
||||
Keep shot/character/scene labels readable for human review, but never feed these
|
||||
annotated grids as clean final frames. The text timeline remains authoritative.
|
||||
|
||||
If preview panels fail, simplify annotation or rely on text within the agreed
|
||||
scope. An optional visualization failure need not block a usable text storyboard.
|
||||
@@ -1,53 +0,0 @@
|
||||
# MiniMax H3 skill provenance
|
||||
|
||||
Upstream: https://github.com/MiniMax-AI/MiniMax-H3
|
||||
|
||||
Source snapshot: `d21241f0a4b3acbb34c97dae47fa417b7065e438`, retrieved 2026-09-19.
|
||||
|
||||
Source directory:
|
||||
https://github.com/MiniMax-AI/MiniMax-H3/tree/d21241f0a4b3acbb34c97dae47fa417b7065e438/skills
|
||||
|
||||
The nine upstream skill names are retained. Each local skill corresponds to the
|
||||
same-named upstream `skills/<name>/` directory. Credit for the original creative
|
||||
workflows and templates belongs to MiniMax-AI and the upstream contributors.
|
||||
|
||||
## Files and adaptation
|
||||
|
||||
- `h3-prompt-writing/references/base-format.md` and `ref-format.md` are new,
|
||||
condensed descriptions of the public prompt interface. They preserve necessary
|
||||
field names and notation but do not copy the upstream guides or their examples.
|
||||
The entrypoint adds a portable execution reference and clarifies clip/window
|
||||
duration for verified long-video backends.
|
||||
- The eight style-specific `SKILL.md` files are rewritten portable adaptations of
|
||||
the upstream workflows. Their reference documents retain the relevant creative
|
||||
methods while removing host-specific execution assumptions.
|
||||
- The two `co-op-game-intro-generator/references/` templates retain the upstream
|
||||
creative structure with an added portable contract and explicit user overrides.
|
||||
Their instructions are normalized to English, and the video template expresses
|
||||
visible UI copy as localized variables with English defaults. It also removes
|
||||
an unrequested gender restriction and a fixed yellow/black environment
|
||||
transformation.
|
||||
- The five 3D animation references are condensed adaptations of their upstream
|
||||
counterparts: shot table, storyboards, model selection, fallback, and review.
|
||||
- Product, papercraft, collage, and music references extract reusable direction
|
||||
from their corresponding upstream skill. The shared portable workflow, catalog,
|
||||
and UI metadata are additions for this repository.
|
||||
|
||||
No upstream `LICENSE`, `COPYING`, or `NOTICE` file was present in the Git tree at
|
||||
the pinned snapshot. Its README links to the MiniMax H3 Community License on the
|
||||
model release. Because that license has redistribution conditions, this package
|
||||
does not include the upstream prompt guides verbatim. This provenance record
|
||||
preserves attribution and does not assert that this repository's code license
|
||||
relicenses upstream material.
|
||||
|
||||
## Runtime changes
|
||||
|
||||
Canvas documents become local files and asset manifests; choice-card UI becomes
|
||||
the host's normal clarification mechanism when a decision is actually missing.
|
||||
Available generation/editing tools replace proprietary dispatch calls. Previous
|
||||
user choices and authorization persist across steps. Providers, prices, resolution
|
||||
support, and long-video features are checked rather than hardcoded as guarantees.
|
||||
|
||||
User intent overrides style presets, including instrumental/no-text work, selected
|
||||
language, final morph shape, no-cut sequences, and custom player designs. Native
|
||||
generation and external processing remain separately identified in delivery.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Portable MiniMax H3 Skills
|
||||
|
||||
This directory contains one general prompt-writing skill and eight creative
|
||||
skills adapted from
|
||||
[MiniMax-AI/MiniMax-H3](https://github.com/MiniMax-AI/MiniMax-H3/tree/main/skills).
|
||||
See [H3-SOURCES.md](H3-SOURCES.md) for the source revision, attribution, and
|
||||
adaptation scope.
|
||||
|
||||
## Skill catalog
|
||||
|
||||
| Skill | Intended use |
|
||||
| --- | --- |
|
||||
| [h3-prompt-writing](h3-prompt-writing/SKILL.md) | Structured prompts for T2VA, I2VA, FL2VA, L2VA, and Ref2VA |
|
||||
| [minimalist-product-ad-generator](minimalist-product-ad-generator/SKILL.md) | Minimal product ads, isolated product references, copy, and pacing |
|
||||
| [3d-animation-short-generator](3d-animation-short-generator/SKILL.md) | 3D narrative shorts, character and environment design, shot tables, storyboards, and editing |
|
||||
| [papercraft-stop-motion-explainer](papercraft-stop-motion-explainer/SKILL.md) | Layered paper art, puppets, mechanisms, and miniature educational scenes |
|
||||
| [brand-promo-video-generator](brand-promo-video-generator/SKILL.md) | Brand promos, verified assets and claims, product interaction, and calls to action |
|
||||
| [music-video-subtitle-generator](music-video-subtitle-generator/SKILL.md) | Music videos, instrumental visual narratives, beat planning, and optional lyric typography |
|
||||
| [co-op-game-intro-generator](co-op-game-intro-generator/SKILL.md) | Two-player game menus, confirmation images, and animated world entry |
|
||||
| [paper-collage-explainer-generator](paper-collage-explainer-generator/SKILL.md) | Halftone photo collage, incremental paper assembly, and visual metaphors |
|
||||
| [handdrawn-live-video-generator](handdrawn-live-video-generator/SKILL.md) | Live action with rough luminous drawings, physical contact, continuous morphing, and delayed pursuit |
|
||||
|
||||
## Usage
|
||||
|
||||
Codex discovers repository skills from `.agents/skills/`. Clients that scan
|
||||
skills only when a session starts may need a new session. Claude Code reads the
|
||||
same files through the same-named relative symlinks in `.claude/skills/`. Other
|
||||
agents that support Markdown skills can load the relevant `SKILL.md` directly
|
||||
and follow its links to supporting references.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
Use $music-video-subtitle-generator to design a 30-second two-dimensional
|
||||
fluid-motion graphic for this instrumental track. Do not add lyrics or
|
||||
subtitles. Preserve leftward motion and initially return only a storyboard and
|
||||
three H3 prompts.
|
||||
```
|
||||
|
||||
```text
|
||||
Use $co-op-game-intro-generator to create a two-player game intro. The players
|
||||
are Lin and Mei, the style is layered paper sculpture, and the title is
|
||||
Together. Create the confirmation image first and wait for my approval before
|
||||
generating video.
|
||||
```
|
||||
|
||||
## MV smoke test
|
||||
|
||||
Test `music-video-subtitle-generator` together with `h3-prompt-writing` using a
|
||||
user-owned song with timestamped lyrics and an original adult-character
|
||||
reference image:
|
||||
|
||||
```text
|
||||
Use $music-video-subtitle-generator and $h3-prompt-writing to turn this song
|
||||
into a historical-romance music video about a woman and a man. Intercut
|
||||
narrative acting with the woman's singing performance, do not display lyric
|
||||
text, split the work according to the backend's verified clip duration, and
|
||||
preserve the complete original song in the final master.
|
||||
```
|
||||
|
||||
Verify that the global timeline matches the source audio duration, lyrics remain
|
||||
verbatim, and each generation window records its incoming state, outgoing state,
|
||||
and local timing. Ref2VA prompts must preserve the six-section field order and
|
||||
resolve every reference label to a real input. The assembled video must decode
|
||||
completely, and its final audio must come from the supplied source rather than a
|
||||
model reconstruction.
|
||||
|
||||
When moving the collection to another repository or a user-level skill folder,
|
||||
copy all nine skill directories and preserve their sibling layout. The eight
|
||||
creative skills share `h3-prompt-writing/references/portable-workflow.md` and the
|
||||
H3 format guides, so copying only one `SKILL.md` is insufficient. Keep each
|
||||
skill's `references/` and `agents/` directories with it. `agents/openai.yaml`
|
||||
provides optional Codex UI metadata and does not restrict the workflow to an
|
||||
OpenAI-based agent or require a dedicated plugin.
|
||||
|
||||
## Execution boundaries
|
||||
|
||||
Prompt writing, storyboarding, and planning require only an agent that can read
|
||||
and write files. Actual image, video, speech, music, and editing work uses tools
|
||||
or services available in the current environment. When a capability is missing,
|
||||
deliver the complete usable production package and identify the unfinished media
|
||||
step.
|
||||
|
||||
The user's style, language, assets, audio, model selection, and existing
|
||||
authorization take priority. An MV template does not add lyrics to an
|
||||
instrumental track, and a clean motion-graphics request does not inherit film
|
||||
grain. A prompt-only request does not start generation. When the user requests
|
||||
approval of a confirmation image before rendering, preserve that checkpoint.
|
||||
|
||||
Treat vLLM-Omni long-video extensions as capabilities of the checked-out branch
|
||||
and active backend. Do not present a development branch's continuation parameters
|
||||
as a universal H3 API. Preserve real requests and raw outputs, distinguish model
|
||||
results from post-production and upscaling, and inspect actual media properties.
|
||||
|
||||
These skills contain no model weights and do not automatically start a service
|
||||
or invoke a paid generation endpoint.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
name: brand-promo-video-generator
|
||||
description: Plan, prompt, or produce brand, product, website, app, and service promotional shorts using traceable assets, accurate claims, timed beats, and clear calls to action. Use for launch and campaign videos, not invented product evidence or long narrative films.
|
||||
metadata:
|
||||
compatibility: File-based workflow with sibling h3-prompt-writing and available research, media generation, inspection, and editing tools.
|
||||
---
|
||||
|
||||
# Brand Promo Video Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
Use supplied brand assets, source links, and existing session decisions. No
|
||||
proprietary tool allowlist or canvas is required. A completed promo requires real
|
||||
generation/editing capability; otherwise deliver an explicitly incomplete production
|
||||
package with finished prompts and a concrete generation handoff.
|
||||
|
||||
## Establish facts and assets
|
||||
|
||||
Inspect product images, UI screenshots, logo exports, typography, colors, and
|
||||
campaign material. Resolve duration, ratio, audience, campaign focus, exact claims,
|
||||
CTA, copy language, narration, and required deliverable from context. Ask for
|
||||
missing identity-bearing assets when they are essential; do not fabricate logos,
|
||||
product UI, screenshots, results, or performance claims.
|
||||
|
||||
Build a compact truth sheet from user-supplied originals and official sources.
|
||||
Record asset ID, role, local path, source URL/note, and verification status. Do not
|
||||
confuse a retrieved search thumbnail with an original brand asset. Distinguish
|
||||
verified facts from clearly labeled concept copy. Preserve the requested language;
|
||||
chat language alone does not justify translating supplied campaign text.
|
||||
|
||||
## Design a product-specific story
|
||||
|
||||
Choose a spine that exposes real cause and effect:
|
||||
|
||||
- Software: user intent → actual interaction → useful output → evidence → CTA.
|
||||
- Physical product: reveal → interaction → mechanism/detail → use → benefit → CTA.
|
||||
- Service: situation → process → evidence → outcome → promise.
|
||||
- Image-led brand: authentic imagery → repeated visual motif → emotional payoff.
|
||||
|
||||
Plan time/frame ranges, asset IDs, one main action, product evidence, exact copy,
|
||||
readable hold, color state, camera, transition, and rhythm. Use the actual output
|
||||
fps for frame timing. Five to eight major beats can fit a 15-second promo, but
|
||||
readability takes priority over a fixed beat count.
|
||||
|
||||
Let product edges, UI movement, light, cursor paths, or matched geometry motivate
|
||||
transitions. Use a few energy peaks and deliberate settling moments. Preserve logo
|
||||
clear space and readable screenshots. Avoid invented HUDs or metrics that look like
|
||||
product proof. Summarize the truth sheet and plan before rendering; continue under
|
||||
existing authorization unless the user requested approval at that stage.
|
||||
|
||||
## Produce and inspect
|
||||
|
||||
Use authentic still/UI plates for exact content. Generated material can supply
|
||||
conceptual scenes, atmosphere, and transition geometry with clear provenance.
|
||||
Compile actual H3 references and shots using the appropriate bundled guide.
|
||||
Check backend resolution, duration, and audio support before calling it.
|
||||
|
||||
Use native instrumental/SFX direction when appropriate; do not generate a second
|
||||
score automatically. Separate narration or music is useful when requested for
|
||||
control or remixing. Preserve original SFX during assembly when they matter, and
|
||||
duck music under speech. Do not add subtitles unless requested.
|
||||
|
||||
Check claims and exact product names against sources, inspect logo geometry and
|
||||
copy, verify real UI states, and review transitions, aspect ratio, duration, audio,
|
||||
and decoding. For wrong identity-bearing content, use the correct supplied/source
|
||||
asset rather than polishing an imitation. Deliver the video, prompts/timeline,
|
||||
asset manifest, and any remaining factual or visual limitations.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Brand Promo Videos"
|
||||
short_description: "Build brand promos from verified assets and claims"
|
||||
default_prompt: "Use $brand-promo-video-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,73 +0,0 @@
|
||||
---
|
||||
name: co-op-game-intro-generator
|
||||
description: Design or generate a two-player co-op game menu and opening animation from player names, title, style, and optional identity references. Use for character-led game intros, not playable game implementation or complex multi-page interfaces.
|
||||
metadata:
|
||||
compatibility: File-based workflow with sibling h3-prompt-writing and bundled image/video templates. Rendering uses available generation tools.
|
||||
---
|
||||
|
||||
# Co-op Game Intro Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
Before writing a confirmation-image prompt, read
|
||||
[h3-confirmation-image-template.md](references/h3-confirmation-image-template.md).
|
||||
Before writing the video prompt, read
|
||||
[h3-video-prompt-template.md](references/h3-video-prompt-template.md).
|
||||
Both templates are required package resources, not remote tools.
|
||||
|
||||
## Resolve player and style bindings
|
||||
|
||||
Reuse or collect the two player names, game title, visual style, desired duration,
|
||||
ratio, and optional identity references. The template defaults to a 15-second,
|
||||
16:9 menu-to-world sequence; adapt it to explicit user requirements. Do not assume
|
||||
the players' gender, body shape, outfits, or equipment from a generic example.
|
||||
|
||||
Use each portrait only for recognizable identity anchors: face silhouette, hair,
|
||||
glasses, proportions, and distinguishing traits. Reinterpret rendering, skin
|
||||
texture, lighting, and costume in the selected visual style. If a portrait or body
|
||||
comparison reference is absent, write a text design rather than inventing an asset
|
||||
label. Maintain player-to-name mapping across all references and frames.
|
||||
|
||||
## Confirmation image
|
||||
|
||||
Fill the template's style, palette, characters, title, and name variables. Keep
|
||||
the two characters central, player cards toward the upper left, menu on the right,
|
||||
and the continue action as the main interaction focus. The title establishes
|
||||
identity without competing with that focus. Buttons, icons, typography, and all
|
||||
player/equipment panels share one palette and style. Keep menu labels on one line.
|
||||
|
||||
The style changes material, shapes, fonts, and decoration; it should not randomly
|
||||
rearrange the menu hierarchy. User-requested layout or ratio changes take priority
|
||||
over template defaults. Preserve the exact supplied title/names and requested copy.
|
||||
|
||||
For image-only/prompt-only work, stop at that deliverable. For video production,
|
||||
generate and inspect the confirmation image if within scope. Honor a request to
|
||||
approve the image first; otherwise use the established end-to-end authorization
|
||||
and continue after reporting the selected image and fixing material defects.
|
||||
|
||||
## Video event plan
|
||||
|
||||
Fill the video template from the current image and player bindings. Its baseline
|
||||
sequence is menu → first player's light equipment → second player's heavy equipment
|
||||
→ shared confirmation → world loading → both enter the world. Equipment, world,
|
||||
body contrast, and timing are editable design choices; preserve requested identity
|
||||
and style instead of forcing a cyberpunk example onto another setting.
|
||||
|
||||
Bind UI reference, each actual portrait, and any real body-comparison image to
|
||||
distinct roles. Compile this creative template into the H3 mode guide's fields;
|
||||
the mixed-language template is not itself an API request. Preserve literal visible
|
||||
UI text in its required language, and remove all unfilled variables before dispatch.
|
||||
Do not treat a composition reference as an exact first-frame anchor unless the
|
||||
selected backend actually supports that role.
|
||||
|
||||
## Review and repair
|
||||
|
||||
Check both names, exact menu strings, layout, player identity, side assignment,
|
||||
equipment distinction, and the transition into the requested world. Match UI
|
||||
hover/click, mechanical locks, loading, and footsteps to the visible events.
|
||||
For identity swaps, strengthen name/position/reference bindings; for weak style,
|
||||
revise palette, material, character, and typography together. Reduce excessive
|
||||
UI text if necessary without silently changing required player names.
|
||||
|
||||
Deliver the requested prompts/assets or the inspected intro video, with reference
|
||||
roles and any text/timing limitations. Missing templates block template execution;
|
||||
missing generation capability still permits a complete prompt handoff.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Co-op Game Intros"
|
||||
short_description: "Design two-player menus and world-entry animation"
|
||||
default_prompt: "Use $co-op-game-intro-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
-397
@@ -1,397 +0,0 @@
|
||||
# H3 Confirmation Image Template
|
||||
|
||||
## Portable adaptation
|
||||
|
||||
This is a creative template, not a tool call. User requirements override its
|
||||
default ratio, layout, poses, and wording. Fill every relevant variable from the
|
||||
brief; set `{character_count}` to two for this skill. When portraits are absent,
|
||||
replace portrait references with explicit character designs. Do not claim to
|
||||
reference an image that was not supplied. Use the image-generation capability
|
||||
available in the host environment; no specific image provider is required.
|
||||
|
||||
## Core Rule
|
||||
Generate a polished 16:9 landscape confirmation image for a cooperative game main menu.
|
||||
The image consists of two independent systems:
|
||||
• UI Framework (fixed)
|
||||
• Visual Style (dynamic)
|
||||
The UI Framework defines the layout, composition, information hierarchy, interaction logic, reading flow, menu structure, and readability. These elements must remain unchanged.
|
||||
{visual_style} only controls the artistic appearance, including rendering medium, materials, color language, lighting mood, environment style, character design, costume language, UI materials, icon language, typography appearance, decorative motifs, and overall atmosphere.
|
||||
Changing {visual_style} should feel like applying a different art direction to the same professionally designed game menu, never redesigning the interface itself.
|
||||
If any default description conflicts with {visual_style}, prioritize {visual_style} while preserving the UI framework.
|
||||
|
||||
---
|
||||
|
||||
## Design Goal
|
||||
Create a premium commercial console game main menu mockup in 16:9 landscape format. The image should combine the quality of a polished game promotional artwork with the clarity of a professional game UI. Characters and interface must feel naturally integrated, with strong visual hierarchy, excellent readability, clean composition, and a clear interaction focus.
|
||||
|
||||
---
|
||||
|
||||
## Overall Style
|
||||
Follow the user-selected visual style: {visual_style}.
|
||||
The selected style controls the rendering medium, illustration style, materials, palette language, environment, character appearance, outfit design, UI appearance, icon language, typography treatment, decorative elements, and atmosphere.
|
||||
All visual elements must belong to one unified artistic language.
|
||||
Do not mix incompatible visual styles within the same image.
|
||||
The UI framework, layout, hierarchy, composition, and interaction logic must remain consistent regardless of style.
|
||||
|
||||
---
|
||||
|
||||
## Color Palette
|
||||
Derive the entire color system from {visual_style}.
|
||||
Use:
|
||||
• {main_background_color} as the primary background color.
|
||||
• {ui_body_color} as the main UI color.
|
||||
• {text_color} as the primary text color.
|
||||
• {functional_accent_color} as the interaction highlight color.
|
||||
• Red only for warning, danger, or exit actions.
|
||||
Limit the overall palette to five primary colors.
|
||||
All buttons, player cards, icons, borders, highlights, decorative elements, typography, and visual effects must follow the same color system.
|
||||
Avoid introducing unrelated colors outside the selected palette.
|
||||
|
||||
---
|
||||
|
||||
## Composition
|
||||
Use a fixed 16:9 landscape layout.
|
||||
Place {character_count} playable characters in the center as the visual subject.
|
||||
The UI surrounds the characters without blocking them.
|
||||
Place the player information cards in the upper-left corner.
|
||||
Place the main menu vertically on the right.
|
||||
Place a decorative horizontal strip along the bottom, adapting naturally to {visual_style} (such as ribbon, warning strip, wooden plank, stone slab, energy bar, scroll, etc.).
|
||||
Keep a clear Z-shaped reading flow:
|
||||
Player Cards → Characters → Menu → Continue Button.
|
||||
The Continue button must always remain the primary visual focus.
|
||||
Maintain generous negative space and excellent readability.
|
||||
|
||||
---
|
||||
|
||||
## Background
|
||||
Generate a background naturally derived from {visual_style}.
|
||||
The background supports the interface rather than competing with it.
|
||||
Keep enough clean negative space behind all important UI elements.
|
||||
Use subtle textures, environmental details, and decorative motifs only where appropriate for the selected style.
|
||||
Avoid excessive visual noise, dense decoration, heavy dirt, scratches, paint splatter, or distracting textures.
|
||||
The overall background should feel polished, premium, modern, and suitable for a commercial game menu.
|
||||
|
||||
---
|
||||
|
||||
## Character Style
|
||||
All characters strictly follow {visual_style}.
|
||||
The selected style determines:
|
||||
• rendering medium
|
||||
• body proportion
|
||||
• facial stylization
|
||||
• costume language
|
||||
• material treatment
|
||||
• lighting response
|
||||
• animation style
|
||||
• overall character quality
|
||||
Only the character identity anchors and pose remain fixed. Facial rendering, eye design, nose simplification, mouth shape, skin treatment, expression style, and head proportion must be fully reinterpreted in {visual_style}.
|
||||
No default character style may override {visual_style}.
|
||||
|
||||
---
|
||||
|
||||
## Character A / PLAYER 1
|
||||
### Identity
|
||||
Use character reference image 1 only for identity mapping.
|
||||
The reference image donates identity anchors only: recognizable face silhouette, hairstyle, glasses if present, relative facial proportions, and distinctive personal traits.
|
||||
It does not donate photographic realism, real skin texture, real-world lighting, camera quality, or the original image style.
|
||||
Do not lose the character's identity anchors, but the face must be redesigned into {visual_style}.
|
||||
All facial rendering, eye design, nose simplification, mouth shape, skin treatment, expression style, and head proportion must follow {visual_style}.
|
||||
Do not swap identity with other characters.
|
||||
|
||||
### Appearance
|
||||
Character rendering, expression, outfit, materials, and accessories are fully derived from {visual_style} while maintaining a recognizable playable-game-character silhouette.
|
||||
|
||||
### Action
|
||||
Character A sits cross-legged on the ground.
|
||||
Both hands support the body on the floor.
|
||||
The upper body leans slightly backward.
|
||||
Character A looks upward toward the camera.
|
||||
|
||||
---
|
||||
|
||||
## Character B / PLAYER 2
|
||||
### Identity
|
||||
Use character reference image 2 only for identity mapping.
|
||||
The reference image donates identity anchors only: recognizable face silhouette, hairstyle, glasses if present, relative facial proportions, and distinctive personal traits.
|
||||
It does not donate photographic realism, real skin texture, real-world lighting, camera quality, or the original image style.
|
||||
Do not lose the character's identity anchors, but the face must be redesigned into {visual_style}.
|
||||
All facial rendering, eye design, nose simplification, mouth shape, skin treatment, expression style, and head proportion must follow {visual_style}.
|
||||
Do not swap identity with other characters.
|
||||
|
||||
### Appearance
|
||||
Character rendering, expression, outfit, materials, and accessories are fully derived from {visual_style} while maintaining a recognizable playable-game-character silhouette.
|
||||
|
||||
### Action
|
||||
Character B sits cross-legged on the ground.
|
||||
Both hands rest naturally in front of the legs.
|
||||
The upper body leans slightly forward.
|
||||
Character B looks directly toward the camera.
|
||||
|
||||
---
|
||||
|
||||
## Lighting
|
||||
Lighting follows {visual_style} while preserving clear readability.
|
||||
Use a primary top light, secondary fill light, soft ambient bounce, contact shadows, gentle rim lighting, and high-quality global illumination where appropriate.
|
||||
Faces should remain clearly visible.
|
||||
Characters must integrate naturally into the environment while maintaining strong separation from the background.
|
||||
|
||||
---
|
||||
|
||||
## Game UI
|
||||
Create a professional commercial console game main menu.
|
||||
The menu structure, interaction hierarchy, spacing, composition, reading flow, and information architecture remain fixed.
|
||||
{visual_style} only changes the artistic appearance of the interface.
|
||||
The selected style may affect:
|
||||
• UI materials
|
||||
• surface textures
|
||||
• border treatment
|
||||
• corner shape
|
||||
• decorative motifs
|
||||
• color rendering
|
||||
• lighting treatment
|
||||
• icon language
|
||||
• typography appearance
|
||||
• interaction effects
|
||||
It must NOT change:
|
||||
• menu hierarchy
|
||||
• button order
|
||||
• button placement
|
||||
• player-card placement
|
||||
• composition
|
||||
• interaction logic
|
||||
• information architecture
|
||||
• readability
|
||||
The entire interface should feel like one professionally designed game UI system rather than separate elements assembled together.
|
||||
|
||||
---
|
||||
|
||||
## UI Visual Language
|
||||
Derive the complete UI appearance from {visual_style}.
|
||||
The selected style determines:
|
||||
• button materials
|
||||
• border language
|
||||
• panel materials
|
||||
• decorative details
|
||||
• edge treatment
|
||||
• visual weight
|
||||
• highlight style
|
||||
• interaction feedback
|
||||
• shadow treatment
|
||||
• texture language
|
||||
Possible materials include wood, stone, paper, leather, cloth, glass, crystal, hologram, neon, metal, ceramic, carved surfaces, painted surfaces, magical energy, pixel blocks, watercolor paper, ink brush, comic graphics or any other material naturally belonging to {visual_style}.
|
||||
Every UI element must share the same material language and artistic style.
|
||||
|
||||
---
|
||||
|
||||
## Button Layout Rules
|
||||
Place all menu buttons vertically on the right side.
|
||||
Maintain a consistent vertical rhythm.
|
||||
Every button uses the same design family.
|
||||
Button width automatically adapts to text length.
|
||||
Keep all menu titles on a single line.
|
||||
No automatic wrapping.
|
||||
No stacked text.
|
||||
Text remains horizontally centered.
|
||||
Keep consistent padding.
|
||||
Maintain equal spacing between buttons.
|
||||
Avoid visual clutter.
|
||||
|
||||
---
|
||||
|
||||
## Continue Button
|
||||
The Continue button is always the primary visual focus.
|
||||
Make it visually dominant using the emphasis method naturally belonging to {visual_style}.
|
||||
Possible emphasis methods include:
|
||||
• larger size
|
||||
• stronger contrast
|
||||
• brighter color
|
||||
• unique material
|
||||
• distinctive border
|
||||
• lighting
|
||||
• glow
|
||||
• engraving
|
||||
• magical energy
|
||||
• holographic effect
|
||||
• embossed frame
|
||||
• animated visual cue
|
||||
Do not force glow or neon if they conflict with {visual_style}.
|
||||
The Continue button should immediately attract the player's attention.
|
||||
|
||||
---
|
||||
|
||||
## Start New Game Button
|
||||
Place above Continue.
|
||||
Use the standard secondary button appearance.
|
||||
Keep the visual language consistent with the selected style.
|
||||
Add one small style-appropriate icon if suitable.
|
||||
Maintain single-line text.
|
||||
|
||||
---
|
||||
|
||||
## Settings Button
|
||||
Use the standard secondary button appearance.
|
||||
Use one settings icon naturally derived from {visual_style}.
|
||||
Maintain consistent proportions, spacing, and hierarchy.
|
||||
|
||||
---
|
||||
|
||||
## Exit Game Button
|
||||
Use the standard secondary button appearance.
|
||||
Use red only as the danger indicator.
|
||||
Red should remain an accent rather than the dominant color.
|
||||
Add one style-appropriate exit icon if suitable.
|
||||
|
||||
---
|
||||
|
||||
## Player Cards
|
||||
Create an upper-left player information system for {character_count} players.
|
||||
The appearance follows {visual_style}.
|
||||
Card materials, borders, corners, textures, shadows, decorations, and icons are automatically derived from the selected style.
|
||||
Each player card contains:
|
||||
• Player label
|
||||
• Player nickname
|
||||
• READY status
|
||||
Player names:
|
||||
PLAYER 1
|
||||
{player1_name}
|
||||
PLAYER 2
|
||||
{player2_name}
|
||||
READY always uses the functional accent color.
|
||||
The cards should remain highly readable and visually connected to the rest of the interface.
|
||||
|
||||
---
|
||||
|
||||
## Icon System
|
||||
Derive all icons from {visual_style}.
|
||||
Use one unified icon family throughout the entire interface.
|
||||
Possible icon languages include:
|
||||
• minimal outline
|
||||
• solid filled
|
||||
• geometric
|
||||
• comic
|
||||
• pixel
|
||||
• engraved
|
||||
• carved
|
||||
• painted
|
||||
• brush
|
||||
• rune
|
||||
• glyph
|
||||
• holographic
|
||||
• crystal
|
||||
• paper-cut
|
||||
• origami
|
||||
• embossed
|
||||
• low-poly
|
||||
• flat design
|
||||
• stylized 3D
|
||||
All icons must share:
|
||||
• the same artistic style
|
||||
• the same material language
|
||||
• the same perspective
|
||||
• the same lighting treatment
|
||||
• the same stroke language
|
||||
• the same color hierarchy
|
||||
Icons should remain simple, recognizable, and functional.
|
||||
Each interface block may contain at most one horizontal row of icons.
|
||||
No wrapping.
|
||||
No vertical stacking.
|
||||
Icons should support the interface rather than dominate it.
|
||||
|
||||
---
|
||||
|
||||
## Typography
|
||||
Typography hierarchy remains fixed.
|
||||
Typography appearance follows {visual_style}.
|
||||
The selected style controls:
|
||||
• font family
|
||||
• decorative treatment
|
||||
• texture
|
||||
• material
|
||||
• outline
|
||||
• lighting
|
||||
• shadow
|
||||
• wear pattern
|
||||
Maintain:
|
||||
• excellent readability
|
||||
• strong visual hierarchy
|
||||
• single-line menu buttons
|
||||
• consistent spacing
|
||||
• high contrast
|
||||
Button width should always adapt to text length.
|
||||
Never allow menu titles to wrap onto multiple lines.
|
||||
|
||||
---
|
||||
|
||||
## Game Title
|
||||
Display the game title:
|
||||
{game_title}
|
||||
The title should become the primary branding element of the menu.
|
||||
Its material, decorative style, typography, and rendering all follow {visual_style}.
|
||||
Avoid imitating existing commercial game logos.
|
||||
Create an original title treatment inspired by the selected style.
|
||||
Keep the title highly readable.
|
||||
Avoid random symbols or meaningless letters.
|
||||
|
||||
---
|
||||
|
||||
## Interaction States
|
||||
Represent the interface as an active game menu.
|
||||
Buttons may display appropriate interaction feedback naturally belonging to {visual_style}.
|
||||
Possible interaction feedback includes:
|
||||
• hover
|
||||
• pressed state
|
||||
• magical glow
|
||||
• energy pulse
|
||||
• holographic highlight
|
||||
• carved illumination
|
||||
• ink spread
|
||||
• brush bloom
|
||||
• metallic reflection
|
||||
• paper emboss
|
||||
• animated border
|
||||
• pixel flashing
|
||||
Only use interaction effects that naturally belong to the selected style.
|
||||
Avoid forcing modern UI effects into incompatible artistic styles.
|
||||
|
||||
---
|
||||
|
||||
## Rendering
|
||||
Generate a polished AAA-quality commercial game menu artwork.
|
||||
Characters, environment, UI, typography, icons, and decorative elements should feel like one cohesive production.
|
||||
Maintain premium rendering quality.
|
||||
Strong visual hierarchy.
|
||||
Professional commercial presentation.
|
||||
High readability.
|
||||
Balanced composition.
|
||||
Clean finish.
|
||||
|
||||
---
|
||||
|
||||
## Negative Constraints
|
||||
No split screen.
|
||||
No duplicated characters.
|
||||
No identity swap.
|
||||
No loss of identity anchors. No photorealistic face unless {visual_style} explicitly requires photorealism. No realistic photo face pasted onto a stylized body. No mismatched face style.
|
||||
No hairstyle changes.
|
||||
No unreadable UI.
|
||||
No distorted buttons.
|
||||
No overlapping interface.
|
||||
No random text.
|
||||
No random symbols.
|
||||
No watermark.
|
||||
No copied game logo.
|
||||
No branded interface.
|
||||
No inconsistent icon styles.
|
||||
No mixed UI materials.
|
||||
No excessive decoration.
|
||||
No overwhelming background.
|
||||
No visual clutter.
|
||||
No broken layout.
|
||||
No low readability.
|
||||
|
||||
---
|
||||
|
||||
## Medium Lock
|
||||
Render as a polished commercial game key art combined with a professional console game main menu mockup.
|
||||
The interface remains fixed.
|
||||
Only the artistic appearance changes according to {visual_style}.
|
||||
@@ -1,249 +0,0 @@
|
||||
# H3 Video Prompt Template
|
||||
|
||||
## Portable adaptation
|
||||
|
||||
This is a creative planning template. User requirements override its default
|
||||
15-second timeline, body proportions, equipment, environment, ratio, and palette.
|
||||
Retiming must preserve readable event order. Use only supplied reference assets:
|
||||
omit `{height_ref}` or identity-image bindings when absent and replace their role
|
||||
with text design constraints. Compile the filled plan into the H3 base or Ref2VA
|
||||
format before rendering; preserve exact visible UI strings in the user's language.
|
||||
|
||||
Use available tools and the actual backend's parameter schema. Audio directions
|
||||
describe desired content rather than asserting that a particular API supports it.
|
||||
|
||||
## Template completion
|
||||
|
||||
Resolve the palette variables `{background_color}`, `{ui_color}`, `{text_color}`,
|
||||
`{functional_accent_color}`, and `{power_accent_color}` from the selected style.
|
||||
Keep the five values distinct enough for readable UI contrast. Use the power
|
||||
accent for danger, exit, and warning states unless the user supplies a separate
|
||||
semantic-color requirement.
|
||||
|
||||
Fill visible copy in the user's requested language. The following English values
|
||||
are defaults only, not mandatory output:
|
||||
|
||||
- `{player1_label}` / `{player2_label}`: `PLAYER 1` / `PLAYER 2`
|
||||
- `{ready_label}`: `READY`
|
||||
- `{start_label}`, `{continue_label}`, `{settings_label}`, `{exit_label}`:
|
||||
`START NEW GAME`, `CONTINUE`, `SETTINGS`, `EXIT GAME`
|
||||
- `{player1_equipment_label}`: `RIGHT ARM EQUIPMENT`
|
||||
- `{player1_option_a}` / `{player1_option_b}`: `PHANTOM GRIP` / `CHRONOS CLAW`
|
||||
- `{player2_equipment_label}`: `ARMAMENT CUSTOMIZATION`
|
||||
- `{hand_label}`, `{forearm_label}`, `{elbow_label}`, `{upper_arm_label}`:
|
||||
`HAND`, `FOREARM`, `ELBOW`, `UPPER ARM`
|
||||
- `{confirm_label}` / `{loading_label}`: `CONFIRM CONFIG` / `LOADING`
|
||||
|
||||
Before compiling the rendering prompt, search the filled template for every
|
||||
remaining token enclosed in braces. Replace required values. For an absent optional
|
||||
reference, remove its binding and rewrite every dependent clause as a text-only
|
||||
constraint. Do not dispatch a prompt while any template token remains.
|
||||
|
||||
## Prompt principle
|
||||
Use the same method as the GPT confirmation-image prompt:
|
||||
|
||||
**Fixed video event framework + dynamic user-style fill + locked character identity + palette-linked UI system.**
|
||||
|
||||
The video prompt must not blindly preserve the source prompt's default style. It must preserve the timeline, UI events, player positions, equipment logic, and text structure, while dynamically rewriting visual treatment, palette, character rendering, lighting, UI surface, icons, and city-world style according to the user's selected style.
|
||||
|
||||
## Priority order
|
||||
1. User-selected style: {visual_style}
|
||||
2. Confirmed image / UI reference: {ui_ref}
|
||||
3. PLAYER 1 identity reference: {player1_ref}
|
||||
4. PLAYER 2 identity reference: {player2_ref}
|
||||
5. Height/body comparison reference: {height_ref}
|
||||
6. Fixed source video event framework
|
||||
|
||||
## Reference roles
|
||||
- {ui_ref}: confirmed first image. Use it to lock UI layout, menu hierarchy, color system, typography scale, button structure, character-game integration, and overall composition logic.
|
||||
- {player1_ref}: PLAYER 1 identity anchor. Lock exact face, hairstyle, glasses if present, facial proportions, body identity, and nickname mapping to {player1_name}.
|
||||
- {player2_ref}: PLAYER 2 identity anchor. Lock exact face, hairstyle, facial proportions, body identity, and nickname mapping to {player2_name}.
|
||||
- {height_ref}: body comparison anchor. Lock the visible contrast between the two players and prevent identical body proportions.
|
||||
|
||||
## Global style baseline
|
||||
The overall art direction must prioritize the user's selected {visual_style}.
|
||||
Preserve these fixed qualities: a game main-menu interface, premium game-trailer
|
||||
finish, deep integration between the UI and characters, modern commercial game
|
||||
UI design, strong visual impact, clean composition, and restrained decoration.
|
||||
Derive the remaining visual treatment from {visual_style}: character rendering,
|
||||
expressions, costume language, color system, lighting temperature, UI materials,
|
||||
button icons, typography texture, and the world revealed after loading.
|
||||
|
||||
## Palette system
|
||||
Derive the complete video palette from {visual_style}, with linked UI colors:
|
||||
|
||||
- Use {background_color} as the primary background and world color.
|
||||
- Use {ui_color} as the main UI color.
|
||||
- Use {text_color} as the text color.
|
||||
- Use {functional_accent_color} as PLAYER 1's functional accent color.
|
||||
- Use {power_accent_color} as PLAYER 2's power accent and for danger, exit,
|
||||
and warning states.
|
||||
- Limit the complete palette to five colors.
|
||||
- Use a vivid, modern, high-contrast color language consistent with
|
||||
{visual_style}.
|
||||
|
||||
Menus, equipment panels, player cards, buttons, the HUD, loading bars, icons,
|
||||
and text must share this color system. Do not introduce unrelated colors.
|
||||
|
||||
## Character identity and style lock
|
||||
PLAYER 1 follows the facial identity in {player1_ref}. Preserve the face,
|
||||
facial proportions, hairstyle, glasses when present, personal identity, and the
|
||||
mapping to the nickname {player1_name}. Adapt expression, rendering, clothing,
|
||||
and mechanical equipment to {visual_style}. PLAYER 1 always stays on the left,
|
||||
with a taller, slender, agile build. Equipment uses the functional accent color,
|
||||
and the mechanical claw is light, narrow, and flexible.
|
||||
|
||||
PLAYER 2 follows the facial identity in {player2_ref}. Preserve the face,
|
||||
facial proportions, hairstyle, personal identity, and the mapping to the
|
||||
nickname {player2_name}. Adapt expression, rendering, clothing, and mechanical
|
||||
equipment to {visual_style}. PLAYER 2 always stays on the right, with a shorter,
|
||||
broader, power-oriented build. Equipment uses {power_accent_color}, and the
|
||||
mechanical fist is broad, heavy, and visibly weighty.
|
||||
|
||||
Do not swap character identities, merge faces, swap nicknames, or converge the
|
||||
two body types.
|
||||
|
||||
## Fixed timeline framework
|
||||
|
||||
### [0-2 seconds] - Two-player main menu
|
||||
|
||||
Composition and camera: A high-angle extreme wide shot follows the composition
|
||||
logic of {ui_ref}. The camera tilts slightly downward and slowly pushes in.
|
||||
|
||||
Visuals: PLAYER 1 ({player1_name}) and PLAYER 2 ({player2_name}) sit side by
|
||||
side at the center. PLAYER 1 remains on the left and PLAYER 2 on the right. They
|
||||
look up toward the camera with only natural breathing, blinking, and subtle body
|
||||
movement.
|
||||
|
||||
UI: The upper-left player card displays `{player1_label}`, `{player1_name}`, and
|
||||
`{ready_label}` exactly as filled. The second card in the upper-center or
|
||||
upper-left two-card system displays `{player2_label}`, `{player2_name}`, and
|
||||
`{ready_label}` exactly as filled. The vertical menu on the right displays
|
||||
`{start_label}`, `{continue_label}`, `{settings_label}`, and `{exit_label}`
|
||||
exactly as filled. `{continue_label}` is the visual center and primary
|
||||
highlighted button.
|
||||
|
||||
Dynamic style fill: Adapt the menu background, ground texture, button shapes,
|
||||
icons, typography, borders, glow, and sticker treatment to {visual_style}, while
|
||||
preserving the layout and hierarchy in {ui_ref}.
|
||||
|
||||
Sound: Menu ambience, subtle UI-hover sounds, and a low electronic atmosphere
|
||||
before the click. Ignore this direction when generating silent video.
|
||||
|
||||
### [2-4 seconds] - PLAYER 1 right-arm configuration
|
||||
|
||||
Composition and camera: In a medium shot, the camera pushes smoothly from the
|
||||
main menu toward PLAYER 1's right arm. PLAYER 2 remains visible and stable in the
|
||||
background.
|
||||
|
||||
UI: The right-side menu contracts and slides away. A panel with functional-accent
|
||||
guide lines slides in from the left and displays `{player1_label}` and
|
||||
`{player1_equipment_label}` exactly as filled. The equipment list first
|
||||
highlights `{player1_option_a}`, then moves the selection to
|
||||
`{player1_option_b}`.
|
||||
|
||||
Action: PLAYER 1's right cuff opens automatically and a lightweight mechanism
|
||||
unfolds beneath the forearm. The fingers spread as long claw-like joints slide
|
||||
into position and lock one by one, briefly revealing fine wiring, miniature
|
||||
pistons, and metal connectors. Functional-accent LEDs illuminate sequentially
|
||||
when configuration completes.
|
||||
|
||||
Dynamic style fill: Adapt the mechanism, panel material, icons, lines, locking
|
||||
animation, and light effects to {visual_style}. Keep the result light, precise,
|
||||
and flexible, proportioned to PLAYER 1 without changing the face, hairstyle, or
|
||||
main clothing design.
|
||||
|
||||
Sound: Precise mechanical unfolding, light UI switching tones, and small locking
|
||||
clicks.
|
||||
|
||||
### [4-7 seconds] - PLAYER 2 heavy-arm configuration
|
||||
|
||||
Composition and camera: A medium shot tracks smoothly between the players and
|
||||
arcs toward PLAYER 2's left side. PLAYER 1 remains in the background, quietly
|
||||
examining the configured mechanical hand.
|
||||
|
||||
UI: A {power_accent_color} panel slides in and displays `{player2_label}` and
|
||||
`{player2_equipment_label}` exactly as filled. Its grid contains `{hand_label}`,
|
||||
`{forearm_label}`, `{elbow_label}`, and `{upper_arm_label}`, with the selection
|
||||
moving quickly but legibly through all four.
|
||||
|
||||
Action: PLAYER 2's left sleeve opens in sections. Heavy forearm plates spring
|
||||
outward, the previous components detach, and new armor slides into place along
|
||||
guide rails. A thick mechanical bearing replaces the elbow joint, and the broad
|
||||
mechanical hand reassembles and locks. Thick wiring, hydraulic pistons, and a
|
||||
dark metal skeleton appear briefly during the replacement. A restrained warm
|
||||
indicator illuminates as each component locks.
|
||||
|
||||
Dynamic style fill: Adapt the heavy arm, UI panel, component icons, materials,
|
||||
and light effects to {visual_style}. Keep it broad, weighty, and powerful, in
|
||||
clear contrast with PLAYER 1's lightweight claw.
|
||||
|
||||
Sound: Low motor movement, heavy mechanical engagement, and weighty locking
|
||||
feedback.
|
||||
|
||||
### [7-8.5 seconds] - Shared configuration confirmation
|
||||
|
||||
Composition and camera: Pull back smoothly to a medium two-player composition,
|
||||
with PLAYER 1 on the left and PLAYER 2 on the right.
|
||||
|
||||
UI: The two equipment panels converge at center to form a shared button that
|
||||
displays `{confirm_label}` exactly as filled. Adapt its border, glow, icon, and
|
||||
sticker treatment to {visual_style}, while keeping the hierarchy and text legible.
|
||||
|
||||
Action: The cursor clicks the button. A functional-accent energy pulse travels
|
||||
through PLAYER 1's mechanical claw, while a {power_accent_color} pulse travels
|
||||
through PLAYER 2's fist. All UI panels contract inward and disappear. Both
|
||||
players uncross their legs and adjust their posture: PLAYER 1 lifts one knee
|
||||
lightly and articulates the claw fingers in sequence; PLAYER 2 plants one foot
|
||||
firmly and slowly closes the heavy fist.
|
||||
|
||||
Sound: Confirmation tone, two-color energy pulses, and UI contraction.
|
||||
|
||||
### [8.5-10 seconds] - Shared world loading
|
||||
|
||||
Composition and camera: A wide shot reveals a shared loading bar at the bottom.
|
||||
|
||||
UI: The loading bar displays `{loading_label}` exactly as filled and fills
|
||||
rapidly from 0% to 100%. Its left half uses {functional_accent_color} and its
|
||||
right half uses {power_accent_color}. Adapt the HUD and loading-bar shape,
|
||||
border, texture, and typography to {visual_style}, while keeping them legible.
|
||||
|
||||
Environment transformation: Continuously transform the confirmed
|
||||
{visual_style} menu background into a game world in the same style. Menu strips,
|
||||
color blocks, and patterns become corresponding roads, structures, and world
|
||||
elements. Preserve the confirmed palette rather than forcing a yellow-and-black
|
||||
or cyberpunk city.
|
||||
|
||||
Key constraint: The transformation remains continuous and natural, with no hard
|
||||
cut, smoke occlusion, or identity change.
|
||||
|
||||
Sound: Rising loading tone and an ambience transition from menu to game world.
|
||||
|
||||
### [10-15 seconds] - Two-player world entry
|
||||
|
||||
Composition and camera: Move from an extreme wide shot into a third-person
|
||||
tracking view. At 100% loading, both characters stand simultaneously. The camera
|
||||
descends smoothly and arcs behind them into a stable two-player cooperative view.
|
||||
|
||||
World style: Generate the complete world from {visual_style}. Preserve the game
|
||||
opening structure: dense buildings, roads, signal lights, signs, crowds or other
|
||||
moving background elements, passing vehicles, overhead wires, industrial pipes,
|
||||
and a distant skyline. Materials, architecture, signage, lighting, and motion
|
||||
must follow {visual_style}. Do not let the source template's cyberpunk default
|
||||
override the user unless the selected style is cyberpunk.
|
||||
|
||||
Character relationship: Clearly show the players from behind and preserve their
|
||||
body contrast. PLAYER 1 is on the left, tall and slender, with the lightweight
|
||||
claw hanging naturally. PLAYER 2 is on the right, shorter and broader, with the
|
||||
heavy fist slightly raised. PLAYER 1 steps forward first, PLAYER 2 follows, and
|
||||
they enter the street side by side.
|
||||
|
||||
HUD: The HUD fades in. A mini-map appears in the upper right. Separate status
|
||||
bars labeled `{player1_name}` and `{player2_name}` appear in the lower left.
|
||||
The first uses {functional_accent_color}; the second uses {power_accent_color}.
|
||||
A shared objective marker appears in the street ahead.
|
||||
|
||||
Sound: City ambience, distant vehicles, footsteps, and a HUD fade-in cue.
|
||||
|
||||
## Negative constraints
|
||||
No extra player, duplicated character, identity or username swapping, merged bodies, unintended face/hair changes, missing player, unrequested split screen or cuts, random camera shake, floating body parts, gruesome dismemberment, unwanted weapons, style-incompatible neon, unreadable UI, random letters, misspelled usernames, extra menu options, copied game logo, or watermark. Preserve the user's chosen character designs and palette.
|
||||
@@ -1,42 +0,0 @@
|
||||
---
|
||||
name: h3-prompt-writing
|
||||
description: Write MiniMax H3 video generation prompts for T2VA, I2VA, FL2VA, L2VA, and Ref2VA. Use when rewriting multimodal requests into H3 prompt structures, composing integrated_multimodal_description, overall_soundscape, and non_diegetic_music, aligning keyframes, or defining reference labels for images, videos, and audio.
|
||||
metadata:
|
||||
compatibility: Portable to any agent that can read local files — no external API calls, MiniMax Hub tools, or proprietary runtime required. The agents/openai.yaml file only adds optional ChatGPT/Codex UI metadata; it does not restrict the skill to OpenAI agents.
|
||||
---
|
||||
|
||||
# H3 Prompt Writing
|
||||
|
||||
## Workflow
|
||||
|
||||
0. Read [portable-workflow.md](references/portable-workflow.md) for scope, available tools, durable artifacts, and generation handoff. For prompt-only work, no generation tools are required.
|
||||
1. Identify the input mode: T2VA, I2VA, FL2VA, L2VA, or full-reference Ref2VA.
|
||||
2. For base text/keyframe modes, read `references/base-format.md` and follow its final prompt structure.
|
||||
3. For full-reference mode, read `references/ref-format.md` and follow its six-section rewrite format.
|
||||
4. Preserve the exact field names, section order, labels, and timing notation from the selected guide.
|
||||
|
||||
## Base Modes
|
||||
|
||||
- T2VA: build the full audiovisual timeline from text.
|
||||
- I2VA: start from the first frame and develop forward from it.
|
||||
- FL2VA: describe the continuous path between the first and last frames.
|
||||
- L2VA: infer a plausible opening and converge to the supplied last frame.
|
||||
|
||||
Use `integrated_multimodal_description`, `overall_soundscape`, and `non_diegetic_music` in the order shown in `references/base-format.md`.
|
||||
|
||||
## Full-Reference Mode
|
||||
|
||||
Ref2VA rewrites use `subject_definitions`, `summary`, `retention_analysis`, `detailed_description`, `overall_soundscape`, and `non_diegetic_music` in that order. Reference labels stay consistent across all sections.
|
||||
|
||||
Read `references/ref-format.md` for label rules and retention analysis.
|
||||
|
||||
## Output Rules
|
||||
|
||||
- Write rewrite sections in English; preserve dialogue, lyrics, and visible scene text in their original language.
|
||||
- Describe each shot by composition, subjects, environment, actions, camera, sound, and the exact point where referenced content appears.
|
||||
- Avoid plot summaries, unresolved reference labels, and timing that does not match the requested duration.
|
||||
## Tips for Better Results
|
||||
- Match the description to the effective duration of the current clip or generation window. The upstream guides describe 4–15-second clips; longer work requires a verified backend extension or an explicit assembly plan, as described in the portable workflow.
|
||||
- Keep reference labels consistent (e.g. `<Picture 1>`, `<Video 1>`, `<Audio 1>`) across every section.
|
||||
- Prefer concrete visual and audio details over abstract words like "cinematic" or "beautiful".
|
||||
- When using keyframes (I2VA / FL2VA / L2VA), clearly state how the first and/or last frame connects to the timeline.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "H3 Prompt Writing"
|
||||
short_description: "Write structured H3 prompts for each input mode"
|
||||
default_prompt: "Use $h3-prompt-writing to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,86 +0,0 @@
|
||||
# H3 base-mode prompt format
|
||||
|
||||
Use this reference for T2VA, I2VA, FL2VA, and L2VA prompt writing. It is a
|
||||
compact, repository-authored description of the required output contract, not a
|
||||
copy of an upstream prompt guide.
|
||||
|
||||
## Select the anchor mode
|
||||
|
||||
- T2VA has no image anchor. Build the audiovisual sequence from the brief.
|
||||
- I2VA binds `<Picture 1>` to the first frame at 0.00 seconds.
|
||||
- FL2VA binds `<Picture 1>` to the first frame and `<Picture 2>` to the final
|
||||
frame. Describe the observable path between them.
|
||||
- L2VA binds `<Picture 1>` to the final frame. Infer a compatible opening and
|
||||
converge to the supplied image.
|
||||
|
||||
For an image-anchored mode, put a plain-English alignment statement before the
|
||||
structured fields. State each picture label, its shot, and its exact target time.
|
||||
The ending time must match the effective clip duration, formatted to two decimal
|
||||
places. Do not invent a picture label when no image was supplied.
|
||||
|
||||
## Emit three fields in order
|
||||
|
||||
```text
|
||||
integrated_multimodal_description: [Shot 1] ...
|
||||
|
||||
overall_soundscape: ...
|
||||
|
||||
non_diegetic_music: ...
|
||||
```
|
||||
|
||||
`integrated_multimodal_description` is the playback-ordered visual and diegetic
|
||||
audio plan. Establish the initial style, framing, subjects, environment, and
|
||||
spatial relationships, then describe actions and state changes. For keyframe
|
||||
modes, explicitly preserve or approach the relevant composition, identity,
|
||||
clothing, colors, objects, and geometry.
|
||||
|
||||
`overall_soundscape` summarizes ambience, physical effects, and non-verbal human
|
||||
sound. Do not repeat dialogue or music already described elsewhere. Use `N/A`
|
||||
only for explicitly silent output.
|
||||
|
||||
`non_diegetic_music` describes music heard by the audience but not the characters.
|
||||
Specify audible properties such as instrumentation, tempo, rhythm, dynamics, and
|
||||
ending behavior. Put source-visible music, singing, radio, or instrument playing
|
||||
in the main description. Use `N/A` when no score is wanted.
|
||||
|
||||
## Timeline notation
|
||||
|
||||
- Start the opening with `[Shot 1]` and no timestamp.
|
||||
- Mark each later cut with a sequential label and a strictly increasing time,
|
||||
for example `[Shot 2] At 00:03.500, ...`.
|
||||
- Use a new shot for an actual edit or meaningful viewpoint/time change. Keep a
|
||||
continuous action or morph in one shot and describe its internal timing.
|
||||
- Describe camera motion as an action: type, direction, useful amplitude, and
|
||||
speed. Do not append a disconnected list of camera keywords.
|
||||
- Keep every cut and event within the requested clip duration.
|
||||
|
||||
## Speech, text, and sound
|
||||
|
||||
Give each speaking or singing source a stable ID such as `(S1)` or `(S2)` and
|
||||
reuse it across shots. Put only the language tag and literal words inside the
|
||||
dialogue wrapper:
|
||||
|
||||
```text
|
||||
The courier (S1) says softly: <d>[English] The gate is open.</d>
|
||||
```
|
||||
|
||||
Preserve supplied dialogue, lyrics, punctuation, and visible text verbatim. State
|
||||
when a voice is off-screen and keep the visible character's lips closed when that
|
||||
is required. If a line spans a cut, describe the audio continuity on both sides
|
||||
of the cut. Place visible signs, labels, and interface copy in double quotes.
|
||||
|
||||
Tie diegetic sounds to the event that produces them. Avoid describing the same
|
||||
sound as both diegetic and non-diegetic.
|
||||
|
||||
## Anchor-specific checks
|
||||
|
||||
- I2VA: begin from the supplied frame rather than merely borrowing its style.
|
||||
- FL2VA: describe intermediate motion; do not provide two unrelated still-image
|
||||
descriptions. Reach Picture 2 at the actual end time.
|
||||
- L2VA: keep the final picture assigned to the last shot, not automatically to
|
||||
Shot 1, and make the preceding action physically plausible.
|
||||
- All modes: reference labels must map to supplied assets and remain stable.
|
||||
|
||||
Before delivery, verify the field order, shot numbering, timestamps, reference
|
||||
labels, dialogue fidelity, requested language, duration, and absence of unresolved
|
||||
template variables.
|
||||
@@ -1,121 +0,0 @@
|
||||
# Portable H3 production workflow
|
||||
|
||||
This is the runtime contract for the nine H3 skills in this repository. Read it
|
||||
alongside the selected creative skill. The bundle uses local files and available
|
||||
tools; no proprietary canvas, choice-card widget, planner, or executor is required.
|
||||
|
||||
## Scope and decisions
|
||||
|
||||
- Resolve the requested deliverable first: prompt, storyboard, assets, or finished
|
||||
video. A prompt-only request ends with the prompt; it does not start generation.
|
||||
- Reuse the user's duration, language, style, model, audio policy, supplied assets,
|
||||
and existing authorization. State reasonable defaults for optional gaps. Ask
|
||||
only about missing inputs or choices that materially affect the requested work.
|
||||
- Creative checkpoints are reviewable artifacts, not mandatory repeated approvals.
|
||||
Honor an explicit request to approve a preview before rendering. When the user
|
||||
has authorized the complete workflow or delegated creative choices, show the
|
||||
plan and continue within that scope without requiring a special UI control.
|
||||
- A skill never grants permission to publish, send messages, spend outside an
|
||||
authorized budget, or change models against the user's choice. User constraints
|
||||
override a preset's style, language, shot count, timing, or audio defaults.
|
||||
|
||||
## Tools and durable artifacts
|
||||
|
||||
1. Inspect the actual available tools, existing scripts, endpoint documentation,
|
||||
and supplied media. Do not invent a tool name, model parameter, or returned path.
|
||||
2. Write plans and prompts to the user-selected project folder. If none is given,
|
||||
use a new named folder under `outputs/`, outside the skill package. Keep useful
|
||||
artifacts such as `brief.md`, `storyboard.md`, `prompts/`, `references/`,
|
||||
`clips/`, and `final/`; create only those needed for this deliverable.
|
||||
3. Record each reference's role and path: identity, environment, style, typography,
|
||||
first/last frame, or audio. Tie generated clips to the prompt revision, input
|
||||
assets, model, settings, and output paths. Update downstream references when an
|
||||
asset changes; retain previous outputs instead of overwriting user files.
|
||||
4. Use available image generation/editing tools for requested raster assets. Use
|
||||
an available video endpoint or local service for clips, and available audio
|
||||
tooling for requested standalone speech/music. For assembly and inspection,
|
||||
use an installed editor or tools such as FFmpeg, ffprobe, and OpenCV. Follow the
|
||||
host's tool policies, including its preferred image generation mechanism.
|
||||
5. Independent tool jobs may run concurrently when resources permit; this skill
|
||||
does not require separate agents or delegated generation.
|
||||
6. If a required capability is missing, finish the useful in-scope preparation:
|
||||
complete prompts, asset-role manifest, timeline, and verified request example
|
||||
where possible. State the missing capability and unfinished media step. Do not
|
||||
silently substitute another model, claim a render happened, or call a written
|
||||
plan a completed video. Resume generation when the capability becomes available.
|
||||
|
||||
## Compile the creative plan into H3 prompts
|
||||
|
||||
Read [base-format.md](base-format.md) for T2VA / I2VA / FL2VA / L2VA, or
|
||||
[ref-format.md](ref-format.md) for Ref2VA. These are writing modes, not guaranteed API
|
||||
task names. Check the selected backend's input mapping before sending a request.
|
||||
|
||||
- Base prompts use `integrated_multimodal_description`, `overall_soundscape`,
|
||||
`non_diegetic_music`, in that order, with the appropriate keyframe preamble.
|
||||
- Ref2VA uses `subject_definitions`, `summary`, `retention_analysis`,
|
||||
`detailed_description`, `overall_soundscape`, `non_diegetic_music`, in that order.
|
||||
- Every reference label must resolve to a supplied asset, in the same order used
|
||||
by the backend. Omit nonexistent optional references and their dependencies.
|
||||
- Keep the creative brief in the user's language. Structured H3 rendering prompts
|
||||
use English prose and exact field labels, while dialogue, lyrics, and visible
|
||||
text retain their requested language and wording. If the user explicitly wants
|
||||
only a same-language creative prompt, deliver that without silently replacing it
|
||||
with English. Compile a separate rendering prompt when generation is requested.
|
||||
- `[Shot 1]` has no initial timestamp. Later shot labels mark actual cuts. For a
|
||||
continuous morph, describe timed actions inside one shot rather than turning
|
||||
each form into a new shot. Express camera direction, speed, and useful amplitude.
|
||||
- Separate image style from character identity. A style reference must not inject
|
||||
its people, scenery, layout, typography, texture, or lighting into unrelated
|
||||
shots. For literal text fidelity, inspect the rendered result; a prompt is not
|
||||
proof that the exact lettering or beat synchronization was achieved.
|
||||
|
||||
## Longer videos and audio
|
||||
|
||||
Use the actual backend's supported duration, dimensions, frame grid, reference
|
||||
count, audio modes, and model partition. Inspect this checkout's H3 implementation
|
||||
or serving documentation before building requests. Do not assume that a feature
|
||||
branch's long-video extension exists on main or on a hosted service.
|
||||
|
||||
For a verified vLLM-Omni Ref2VA continuation backend, `long_video=true` with
|
||||
`long_video_mode=continuation` can accept one `continuation_prompts` entry per
|
||||
planned window only when the Ref2VA request is executed with its local text
|
||||
encoder. Step execution and stages that use an external text encoder reject
|
||||
`continuation_prompts`; omit the field and use the supported per-step inputs in
|
||||
those configurations. Compute the window count from the aligned total frames,
|
||||
window size, and overlap. Each prompt uses local window time; track global
|
||||
shot/music time separately. Repeated overlap is guidance, not additional output
|
||||
duration. Do not use this mode on a task/partition that does not support it.
|
||||
|
||||
Otherwise split into supported clips and assemble them. Same-scene clips may use
|
||||
tail/head keyframe continuity when supported. Scene changes should have explicit
|
||||
new scene references; a previous scene's latent tail can preserve the wrong
|
||||
setting. Choose cuts, morphing, and transitions from the brief rather than imposing
|
||||
hard cuts whenever a work exceeds 15 seconds. Generation windows and visual shots
|
||||
are different: one window may contain cuts, and one shot may span multiple windows.
|
||||
|
||||
For supplied music, use the selected excerpt as the master timeline unless the
|
||||
user requests replacement. For native audio, write the musical/SFX direction in
|
||||
the prompt and verify that the backend actually returns audio. Do not invent a
|
||||
generic `generate_audio` parameter. If available, locked-source audio latents guide
|
||||
generation but the audio-VAE reconstruction is not bit-identical to the source.
|
||||
Use explicit final audio stream-copy when an exact original track is required.
|
||||
Prompted BPM is a target, not measured timing. Avoid automatically adding lyrics,
|
||||
voiceover, subtitles, or multiple competing scores to an instrumental request.
|
||||
|
||||
## Verification and bounded recovery
|
||||
|
||||
Inspect reference images before using them. After rendering, inspect representative
|
||||
frames, transitions, opening/ending states, text, identities, and scene continuity.
|
||||
Check audio by listening when possible; report any limit on audio review. Check
|
||||
duration, dimensions, fps, frame count, audio presence, and decode validity with
|
||||
media tools. If preserving audio exactly, compare extracted audio stream hashes.
|
||||
|
||||
Correct a demonstrated failure with a targeted change to the affected prompt,
|
||||
reference, or edit. Keep raw generation alongside edited/upscaled versions and
|
||||
label the difference. Use an agreed retry/resource budget; absent one, allow one
|
||||
targeted rerender, then report persistent defects and concrete options rather than
|
||||
entering an unbounded generation loop. Do not switch providers automatically.
|
||||
|
||||
Deliver actual artifact paths, the prompts/settings needed to repeat the work,
|
||||
and material visual/audio limitations. Distinguish requested behavior, observed
|
||||
model output, and post-production repairs.
|
||||
@@ -1,105 +0,0 @@
|
||||
# H3 full-reference prompt format
|
||||
|
||||
Use this reference for Ref2VA prompt writing. It is a compact,
|
||||
repository-authored description of the output contract, not a copy of an
|
||||
upstream prompt guide.
|
||||
|
||||
## Emit six fields in order
|
||||
|
||||
```text
|
||||
subject_definitions: ...
|
||||
|
||||
summary: ...
|
||||
|
||||
retention_analysis: ...
|
||||
|
||||
detailed_description: [Shot 1] ...
|
||||
|
||||
overall_soundscape: ...
|
||||
|
||||
non_diegetic_music: ...
|
||||
```
|
||||
|
||||
Write the structural prose in English. Preserve the requested language and exact
|
||||
wording for dialogue, lyrics, and visible text.
|
||||
|
||||
## Reference labels
|
||||
|
||||
Assign labels only to supplied content and keep every label stable across all six
|
||||
fields:
|
||||
|
||||
- `<Subject N>` identifies reusable visible content such as a person, object,
|
||||
environment, costume, interface, action, or visual treatment.
|
||||
- `<Picture N>` identifies an image used as a concrete frame, composition, or
|
||||
storyboard anchor. If an image only defines a subject, cite it in that subject's
|
||||
definition instead of inventing a separate target-frame role.
|
||||
- `<Video N>` identifies a source video being edited or continued, or a video
|
||||
whose temporal structure is referenced.
|
||||
- `<Audio N>` identifies a supplied audio signal that is copied or referenced.
|
||||
Audio and video label numbers are independent even when they came from one file.
|
||||
|
||||
In `subject_definitions`, give each tracked item a separate line. State its source,
|
||||
role, and the features that later sections must preserve or transform. Do not
|
||||
create labels for assets or roles that do not exist.
|
||||
|
||||
## Summary and retention
|
||||
|
||||
`summary` is one short paragraph describing the target and the principal reference
|
||||
relationships. Use only labels already defined. Distinguish these operations:
|
||||
|
||||
- a concrete image frame anchor;
|
||||
- generation guided by identity, style, action, camera, or audio characteristics;
|
||||
- direct editing of a source video;
|
||||
- continuation from a source video;
|
||||
- copying an audio signal;
|
||||
- referencing audio properties without copying the signal.
|
||||
|
||||
`retention_analysis` has one line per label and states where it is used and what
|
||||
changes. Use consistent machine-readable relationship values:
|
||||
|
||||
- Visual: `fully_preserved`, `partially_preserved`, `attribute_transfer`, or
|
||||
`weak_reference`.
|
||||
- Audio: `fully_copy`, `partially_copy`, `reference`, or `weak_reference`.
|
||||
|
||||
Explain the concrete retained and changed traits after the relationship value.
|
||||
Do not claim full preservation when identity, geometry, timing, or signal content
|
||||
is intentionally altered. Do not count newly requested target action as an
|
||||
unintended loss of reference fidelity.
|
||||
|
||||
## Detailed description
|
||||
|
||||
`detailed_description` is the playback-ordered rendering plan. For each shot,
|
||||
cover the current composition, subject appearance and position, environment,
|
||||
lighting, actions and state changes, camera behavior, diegetic sound, and the
|
||||
point at which each reference becomes relevant.
|
||||
|
||||
- Start with `[Shot 1]` and no timestamp.
|
||||
- Give later cuts sequential labels with strictly increasing times, such as
|
||||
`[Shot 2] At 00:03.500, ...`.
|
||||
- Place relevant reference labels next to the described subject, frame, motion,
|
||||
or sound; do not collect detached labels at the end.
|
||||
- Use stable `(S1)`, `(S2)`, and subsequent IDs for vocal sources. Put literal
|
||||
speech or lyrics inside `<d>[Language] ...</d>`.
|
||||
- Preserve supplied visible text verbatim and place it in double quotes.
|
||||
- Describe continuous transformations inside one shot unless an actual cut is
|
||||
intended. State camera direction, speed, and amplitude when useful.
|
||||
|
||||
For video editing, explicitly identify what stays from the source timeline and
|
||||
what is replaced. For continuation, distinguish the inherited opening state from
|
||||
new content. For style reference, transfer visual properties without accidentally
|
||||
copying people, scenery, layout, or typography.
|
||||
|
||||
## Audio fields
|
||||
|
||||
`overall_soundscape` summarizes ambience, physical action sounds, and non-verbal
|
||||
human sounds. `non_diegetic_music` describes score audible only to the audience.
|
||||
Keep dialogue and diegetic music in `detailed_description`. Use `N/A` only when
|
||||
the corresponding layer is intentionally absent.
|
||||
|
||||
If audio is copied, state whether the whole signal or only a range/layer is used.
|
||||
If it is referenced, describe the specific property—such as voice timbre, tempo,
|
||||
rhythm, or texture—without claiming signal reuse.
|
||||
|
||||
Before delivery, verify the six-field order, one-to-one label definitions,
|
||||
retention coverage, shot timing, dialogue fidelity, requested language, and
|
||||
absence of unresolved template variables.
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
name: handdrawn-live-video-generator
|
||||
description: Write or generate a continuous live-action scene with a rough luminous hand-drawn entity that contacts real objects, morphs, and escapes a slightly delayed handheld camera. Use for gentle surreal mixed-media shorts, not flat-vector-only animation or horror clips.
|
||||
metadata:
|
||||
compatibility: Portable prompt writing with sibling h3-prompt-writing; actual generation requires an available video backend and any referenced media.
|
||||
---
|
||||
|
||||
# Handdrawn Live-Action Fusion
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
Keep the user's requested scope and language. A creative-prompt request delivers
|
||||
the prompt, without generating media. For a requested finished video, prepare
|
||||
the prompt and proceed under existing authorization using available tools.
|
||||
|
||||
## Creative contract
|
||||
|
||||
The default is a 15-second, 16:9 continuous scene in an everyday real space. A
|
||||
flat luminous drawn entity visibly contacts a real hand/object early, transforms
|
||||
as one traceable being, escapes through connected space, and the handheld camera
|
||||
reacts slightly late. Duration and ratio follow the user's explicit choices.
|
||||
|
||||
Use rough crayon, chalk, colored pencil, pastel, or brush strokes with uneven fill,
|
||||
frayed edges, line jitter, and frame-by-frame redraw. Keep the live-action setting
|
||||
physically grounded and the drawn presence visibly planar. Uniform neon tubes,
|
||||
plush characters, polished CG, and clean vector strokes would change this style.
|
||||
|
||||
Choose a fresh setting, entity, palette, contact mechanism, transformation chain,
|
||||
escape route, camera reaction, and emotional ending within the user's constraints.
|
||||
Do not ban an otherwise requested motif simply because an upstream example used
|
||||
it. The tone is warm, playful, and gently surprising rather than threatening.
|
||||
|
||||
## Timed action structure
|
||||
|
||||
For a 15-second brief, use these intervals as a starting structure:
|
||||
|
||||
- 0–3s: establish obvious real/drawn contact, such as landing on a palm or slipping
|
||||
around a finger. The real participant reacts.
|
||||
- 3–6s: begin a continuous morph while retaining a color smear, line, curve, or tail
|
||||
from the previous shape. Give it a playful action.
|
||||
- 6–10s: let it escape into an adjacent part of the same space. The camera pans,
|
||||
tilts, or moves only after the entity starts leaving its view.
|
||||
- 10–13s: add another readable interaction or surprise, keeping the same entity.
|
||||
- 13–15s: extend an earlier drawn motif across the space into a gentle larger
|
||||
transformation, then resolve with a small emotional/comic detail.
|
||||
|
||||
Retiming changes the intervals proportionally or reorganizes beats; do not leave
|
||||
15-second timestamps in a shorter prompt. Preserve a user's explicit final form
|
||||
instead of forcing the spatial finale when it conflicts with their ending.
|
||||
|
||||
## Prompt delivery and rendering
|
||||
|
||||
For a same-language creative prompt, begin with duration, ratio, real space, and
|
||||
the live-action/drawing fusion. Then describe phone-camera texture, the timed
|
||||
action intervals, drawn material, camera lag, specific exclusions, and ambient
|
||||
sound. Avoid unrelated headings or an essay when the user asked for copy-ready text.
|
||||
|
||||
For actual H3 generation, create a separate structured rendering prompt using the
|
||||
appropriate base/Ref2VA guide: English prose with literal dialogue/visible text in
|
||||
the requested language. Preserve the creative intent and actual media bindings.
|
||||
Do not send unsupported model settings or promise exact first-frame conditioning
|
||||
merely because an image is present.
|
||||
|
||||
## Review
|
||||
|
||||
Check clear early physical contact, one continuous entity, connected geography,
|
||||
retained shape motifs, delayed camera reaction, rough planar drawing, and a
|
||||
non-threatening ending. Avoid jump scares, sudden blackouts, menacing anatomy, and
|
||||
unrequested extra characters. Deliver the actual prompt/video and state any
|
||||
remaining contact, continuity, texture, or camera-timing defects.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Handdrawn Live-Action Fusion"
|
||||
short_description: "Direct drawn morphs and real-world interactions"
|
||||
default_prompt: "Use $handdrawn-live-video-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
name: minimalist-product-ad-generator
|
||||
description: Plan, prompt, or produce minimalist product advertising shorts from product images, preserving variants, material, color, concise typography, and rhythmic camera motion. Use for product launches and ecommerce films, not presenter-led ads or complex UI demonstrations.
|
||||
metadata:
|
||||
compatibility: File-based agent workflow. Requires the sibling h3-prompt-writing package; rendering uses available image, video, audio, and editing tools.
|
||||
---
|
||||
|
||||
# Minimalist Product Ad Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
Use [product direction](references/product-direction.md) when planning anchors,
|
||||
copy, and timed beats. Respect the requested output: prompt-only work needs no
|
||||
generation tool, while a finished film includes generation, inspection, and assembly.
|
||||
|
||||
## Resolve the product brief
|
||||
|
||||
Inspect supplied product photos, packaging, renders, and detail shots. Record the
|
||||
product category, main variant, actual body/accent colors, material finish, usable
|
||||
mechanisms, and requested copy. Preserve identity and product facts; visual polish
|
||||
does not authorize recoloring the product or inventing a feature.
|
||||
|
||||
Reuse the specified duration, ratio, language, and model. With no constraints,
|
||||
suggest a 10-second H3 film and one of white studio, dark rim light, brand color
|
||||
field, or light lifestyle staging. Ask for a missing product reference when exact
|
||||
identity is required; distinguish a speculative concept from an existing product.
|
||||
If the user delegates direction, record the adopted choices and proceed.
|
||||
|
||||
## Design the film
|
||||
|
||||
1. Choose a product-specific spine: launch reveal, mechanism interaction, or color
|
||||
family. Describe concrete actions such as a lid opening or a crown turning.
|
||||
2. Write short sensory/benefit copy grounded in supplied facts. Preserve the user's
|
||||
exact approved wording and language. For newly requested English copy, 3–5 words
|
||||
usually fit one readable line; do not force English on other-language campaigns.
|
||||
3. Plan three independent anchors: hero silhouette, material/mechanism detail, and
|
||||
final product/copy composition. Generate only the assets needed or requested.
|
||||
Do not combine them into a panel sheet that the video model might reproduce.
|
||||
4. Keep product color, light direction, ratio, and finish consistent across anchors.
|
||||
Original photos establish identity; generated anchors must pass that identity
|
||||
check before they can replace them as rendering references.
|
||||
5. Build the beat table from the reference. Include setup, action, impact, and a
|
||||
stable closing composition; vary shot distance only when it reveals something.
|
||||
|
||||
## Prompt and render
|
||||
|
||||
Compile each beat into the appropriate H3 fields using the sibling writing guide.
|
||||
Bind every anchor to its actual role. Include exact visible copy, entry/exit times,
|
||||
colors, and same-line text motion in the rendering prompt, not only in the plan.
|
||||
Keep typography in intentional negative space without covering the product. A
|
||||
copy-reference image does not mean its line must persist throughout the whole film.
|
||||
|
||||
For a short ad, prefer one coherent full-frame film when the backend supports it.
|
||||
Keep variants ordered: main variant first, supporting variants later, full family
|
||||
only when useful at the close. Avoid product walls, grids, or an anchor-sheet ending.
|
||||
|
||||
Use native audio when appropriate to the brief. A restrained electronic score may
|
||||
use crisp plucks, airy pads, kick/sub accents and tactile wooden percussion. Set
|
||||
tempo and energy from the user's direction. Generate a separate score only if
|
||||
requested or needed for a disclosed audio repair, using an available tool.
|
||||
|
||||
## Review and deliver
|
||||
|
||||
Verify product geometry, variant colors, functional action, final copy, aspect
|
||||
ratio, and closing readability. Listen for excessive bass, dropouts, and unfinished
|
||||
endings. If editing to supplied music, select a suitable continuous excerpt and
|
||||
align visual impacts to observed beats; do not claim synchronization from text alone.
|
||||
|
||||
For bad typography, first reduce simultaneous text or correct its reference. Exact
|
||||
lettering may need a disclosed overlay when consistent with the user's goal; keep
|
||||
the raw output. Deliver the film, prompt/beat plan, used copy and music excerpt,
|
||||
and any remaining product/text fidelity issues.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Minimalist Product Ads"
|
||||
short_description: "Direct product anchors, copy, motion, and audio"
|
||||
default_prompt: "Use $minimalist-product-ad-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,52 +0,0 @@
|
||||
# Product anchors, typography, and beats
|
||||
|
||||
## Narrative choices
|
||||
|
||||
- Launch: negative space → hero product → material detail → real mechanism action
|
||||
→ supporting variants → stable product and copy.
|
||||
- Feature touch: stillness → interaction trigger → visible mechanism → detail
|
||||
magnification → useful result → closing.
|
||||
- Color family: main variant alone → supporting colors enter → ordered relation
|
||||
→ unified material finish → full set and copy.
|
||||
|
||||
## Independent anchor briefs
|
||||
|
||||
| Anchor | What it locks | Check |
|
||||
| --- | --- | --- |
|
||||
| Hero | Main variant, recognizable silhouette, attractive opening view | Match supplied body color and product structure |
|
||||
| Detail | Material, reflection, texture, visible mechanism | No invented ports, controls, or functionality |
|
||||
| Closing | Product position, negative space, typography treatment | Real copy, readable contrast, no panel/grid layout |
|
||||
|
||||
Each is a standalone full-frame image with the project ratio. Reference roles
|
||||
protect appearance and layout; they do not replace the storyboard or require three
|
||||
sequential static-image segments. Simplify the number of anchors if the backend
|
||||
cannot bind them all, and update the prompt labels accordingly.
|
||||
|
||||
## Typography
|
||||
|
||||
Use one readable line at a time, with a typeface appropriate to the brand. A clean
|
||||
semibold sans-serif is a useful default, not a requirement to copy a specific brand.
|
||||
For a two-part color treatment, use dark text on a light field or light text on a
|
||||
dark field, then emphasize the second part with a specific real product color.
|
||||
Check contrast rather than blindly coloring text like the product.
|
||||
|
||||
The first phrase may enter gently; the second follows while the first shifts
|
||||
slightly within the same line. Do not accidentally turn this into stacked lines.
|
||||
Use actual copy in the prompt and spell out its timing. Avoid arrows, separators,
|
||||
or placeholder marks that the model may reproduce. For a 10-second ad, one middle
|
||||
copy moment and one stable closing line are often enough. Omit copy when requested.
|
||||
|
||||
## Beat table
|
||||
|
||||
| Time range | Shot purpose | Visual lead and camera | Variant state | Exact copy | Text color/motion | Handoff | Rhythm |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
|
||||
Use one main action per beat. For 5/10/15-second films, roughly 3–4/5–7/6–9 beats
|
||||
are planning suggestions, not required cuts. Record the ending product position
|
||||
and motion vector so the next beat can inherit them. Ensure the closing line has
|
||||
time to be read rather than using every second for motion.
|
||||
|
||||
If replacing music, keep useful SFX unless replacement of the whole track was
|
||||
requested. Use a continuous excerpt with a clear entry, stable middle, and suitable
|
||||
ending. A short tail fade may resolve a click; do not hide an incomplete phrase
|
||||
behind a long fade. Confirm the entire result remains audible.
|
||||
@@ -1,74 +0,0 @@
|
||||
---
|
||||
name: music-video-subtitle-generator
|
||||
description: Plan, write, audit, or produce music videos with coordinated music, imagery, camera motion, optional performance, and lyric typography. Use for beat-aware MVs and instrumental visual narratives, not ordinary subtitle cleanup or unrelated video edits.
|
||||
metadata:
|
||||
compatibility: File-based workflow with sibling h3-prompt-writing; audio analysis, generation, and final assembly use available tools and backends.
|
||||
---
|
||||
|
||||
# Music Video and Lyric Typography
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md)
|
||||
and, when building a timeline, [music timeline](references/music-timeline.md).
|
||||
Deliver the requested scope: audit, prompt, storyboard, or complete video.
|
||||
|
||||
## Lock the musical and visual intent
|
||||
|
||||
Identify duration, ratio, output resolution, music source/excerpt, visual style,
|
||||
character/scene references, performance mode, and whether text is wanted. Treat a
|
||||
supplied song as the master track unless the user requests replacement. Preserve
|
||||
provided lyrics exactly; do not translate or rewrite them unasked. For instrumental
|
||||
work, do not invent singing, lyrics, a performer, or typography. Generate original
|
||||
lyrics only when requested or clearly within an authorized songwriting brief.
|
||||
|
||||
Measure supplied audio duration and inspect/listen for beat, phrase, breath, drop,
|
||||
and energy changes using available tools. Label estimated or planned timing when
|
||||
measurement is unavailable. If the requested excerpt exceeds the audio, resolve
|
||||
the discrepancy without silently stretching or looping it.
|
||||
|
||||
## Reference roles
|
||||
|
||||
- Character references control identity, silhouette, wardrobe, and presence.
|
||||
- Scene references control environment, time of day, spatial anchors, and lighting.
|
||||
- Typography references control letter treatment, layout, and motion, not people
|
||||
or scenery visible in those references.
|
||||
- Music references control the requested track or sound style; distinguish copying
|
||||
a supplied recording from generating similar instrumentation.
|
||||
|
||||
Do not apply a preset's grain, darkness, hard cuts, or constant close-ups to an
|
||||
incompatible brief. A bright nighttime scene remains night. A continuous flat
|
||||
vector morph keeps smooth color fields and uses elastic shape motion for accents.
|
||||
Use hard cuts, slow holds, or dissolves only when appropriate to the user's intent.
|
||||
|
||||
## Compose the timeline and H3 prompts
|
||||
|
||||
Map visual actions, camera moves, scene changes, and optional text to the music's
|
||||
global time. Build a causal visual progression instead of defaulting to one person
|
||||
singing throughout. Allow room to read a shape, gesture, or lyric before changing it.
|
||||
|
||||
For longer work, choose a supported continuation or clip-assembly approach through
|
||||
the portable workflow. Keep visual shot boundaries distinct from generation
|
||||
windows. Local window prompts must account for overlap without replaying the full
|
||||
previous action. A new scene may require a new reference rather than inheritance
|
||||
of the old scene. Record entry/exit states for each planned segment.
|
||||
|
||||
Compile the timeline into H3's base or Ref2VA fields, preserving dialogue, lyrics,
|
||||
and visible text verbatim. Include concrete instrumentation and rhythmic accents,
|
||||
but treat requested BPM and beat alignment as targets until checked in the output.
|
||||
|
||||
## Typography and finishing
|
||||
|
||||
When requested, give each shot one main readable text event. Keep words away from
|
||||
eyes and critical lip motion. Match sung text to the active phrase, and distinguish
|
||||
spatial typography from subtitle bars. Reduce text complexity if the model cannot
|
||||
render it, or use a disclosed overlay consistent with the requested deliverable.
|
||||
|
||||
For supplied audio, assemble clips to one global master track, maintaining phrase
|
||||
and lip continuity; avoid cuts inside a vowel. For native audio, inspect actual
|
||||
continuity across generation windows. Preserve useful SFX and avoid layered duplicate
|
||||
music. Retiming, grain, grading, interpolation, and upscaling are deliberate editing
|
||||
choices, not automatic finishing requirements.
|
||||
|
||||
Verify transformation/story order, identities, scene changes, requested text,
|
||||
musical continuity, timing, and full media decode. Label raw and processed versions.
|
||||
Deliver the final file and current prompt/timeline files, with measured properties
|
||||
and remaining limitations rather than claims of perfect synchronization.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Music Video Direction"
|
||||
short_description: "Plan music-driven scenes and optional lyric text"
|
||||
default_prompt: "Use $music-video-subtitle-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,40 +0,0 @@
|
||||
# Global music and local video timing
|
||||
|
||||
Use this planning skeleton, omitting vocal/text fields when not requested:
|
||||
|
||||
```text
|
||||
Global: music/excerpt, duration, ratio, character/style/scene locks, cut policy.
|
||||
|
||||
Segment S01 — global start/end; local generation start/end
|
||||
Incoming state:
|
||||
Music cue: measured beat/phrase timestamp, or clearly labeled planned accent
|
||||
Vocal line and speaker (if present):
|
||||
Visible typography (if requested):
|
||||
Visual action and camera:
|
||||
Outgoing state and transition:
|
||||
References with exact roles:
|
||||
```
|
||||
|
||||
Keep one global music clock. For measured tempo B, the nominal quarter-note spacing
|
||||
is 60/B seconds, but real tracks may have pickups, swing, tempo changes, or inaccurate
|
||||
automatic detections. Validate important cut points against the waveform/listening.
|
||||
An onset detector also finds SFX and melodic attacks; it does not prove drum beats.
|
||||
|
||||
Choose a phrase-complete excerpt when possible. Track breaths, lyric pauses, and
|
||||
vowel continuity for a performance MV. For an instrumental morph, accent the
|
||||
physical preparation, transformation impact, and release rather than introducing
|
||||
unrequested cuts. The final shape or story payoff needs a readable ending.
|
||||
|
||||
For continuation, maintain a table of absolute output frames and local window
|
||||
times. Use the verified backend's native alignment and overlap behavior. Opening
|
||||
overlap continues the inherited state; new actions belong after it. Do not sum
|
||||
overlap twice or describe every window as a fresh opening shot.
|
||||
|
||||
For separately generated clips, use a previous tail frame only when the next shot
|
||||
continues the same scene and the backend supports that keyframe role. For an
|
||||
intentional cut, retain appropriate identity references and select the new scene.
|
||||
Preserve screen direction or a useful match element when it helps the transition.
|
||||
|
||||
When text is requested, track exact string, language, entry/exit time, screen
|
||||
region, and motion. Typography style cards must not donate their depicted characters
|
||||
or settings. Preserve user text; do not silently shorten lyrics to fit a template.
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
name: paper-collage-explainer-generator
|
||||
description: Translate narration, ideas, or knowledge points into editorial halftone paper-collage metaphors, still prompts, storyboards, or stop-motion clips. Use for photographic cutout collage and tactile assembly, distinct from paper-puppet dioramas or generic cartoons.
|
||||
metadata:
|
||||
compatibility: File-based workflow with sibling h3-prompt-writing; media creation uses available image/video tools and an editor when needed.
|
||||
---
|
||||
|
||||
# Paper Collage Explainer Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md)
|
||||
and [collage design](references/collage-design.md) when constructing scenes.
|
||||
Honor prompt-only and asset-only requests as well as finished video requests.
|
||||
|
||||
## Meaning and media policy
|
||||
|
||||
Extract each source line's meaning, emotion, action verb, visual metaphor, and
|
||||
three to six readable object groups. Preserve the user's topic and wording as
|
||||
context; do not put narration on screen merely because it is available.
|
||||
|
||||
Default to tactile paper SFX, with no BGM, voiceover, or subtitles unless requested.
|
||||
An explainer does not inherently require speech. Existing user requests for those
|
||||
layers already establish the choice; do not ask again. A short 16:9 clip of roughly
|
||||
four seconds is a useful default for one assembly beat, subject to backend limits.
|
||||
|
||||
## Plan the image and motion together
|
||||
|
||||
Create a concise plan of metaphor, palette, object groups, assembly order, duration,
|
||||
and sound cues. For a series, keep paper texture, halftone treatment, keylines,
|
||||
shadows, and rhythmic language consistent; vary the metaphor and color field only
|
||||
where the meaning benefits. A user's no-text or no-music requirement applies to
|
||||
every downstream asset, prompt, clip, and edit.
|
||||
|
||||
Design each still as the completed final composition. Use black-and-white
|
||||
photographic halftone cutouts, selective colored cardstock, warm cream keylines,
|
||||
subtle fiber/torn edges, and soft physical layer shadows on a bold color field.
|
||||
Keep layered depth and negative space; avoid excessive age, dirt, wrinkles, or
|
||||
automatic brown/kraft backgrounds. Exact editable layers require a compositing
|
||||
workflow, not a claim that a generated raster video is a layered project.
|
||||
|
||||
Generate final-composition stills when requested/needed and inspect them. Follow
|
||||
any requested approval gate; otherwise continue within the authorized workflow.
|
||||
|
||||
## Animate the assembly
|
||||
|
||||
Start from a clean paper field matching the planned final color. Base pieces enter,
|
||||
then the main metaphor, then supporting objects. Each piece slides or pops in,
|
||||
bounces slightly, presses flat, pauses, and locks. End with a brief hold on the
|
||||
completed composition. Use stop-motion assembly rather than global fades, smooth
|
||||
digital drifting, fast spinning, or liquid morphs unless explicitly requested.
|
||||
|
||||
Use the supplied final still as a last-frame anchor only when the backend supports
|
||||
it. Otherwise identify it honestly as composition/style guidance; do not describe
|
||||
a reference image as a guaranteed final-frame constraint. Compile the actual H3
|
||||
mode with the sibling guide and bind only real inputs.
|
||||
|
||||
Map short paper slides, pops, taps, rustles, and snaps to motion. Preserve those
|
||||
SFX during assembly. When the user adds music or narration later, mix with the
|
||||
useful SFX unless they request full replacement.
|
||||
|
||||
## Review and deliver
|
||||
|
||||
Check the metaphor without relying on text, object count/readability, matching
|
||||
opening color, visible piece-by-piece construction, final-frame resemblance,
|
||||
controlled paper texture, and no unwanted UI/letters/music/voiceover. If the
|
||||
assembly is weak, simplify the groups and make their arrival order concrete.
|
||||
|
||||
For multiple clips, preserve the planned order and SFX. Deliver prompts, current
|
||||
stills, and actual clips/assembly as requested. Explain any missing generation
|
||||
capability or remaining drift; keep raw output separate from edited fixes.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Paper Collage Explainers"
|
||||
short_description: "Animate editorial paper metaphors with tactile SFX"
|
||||
default_prompt: "Use $paper-collage-explainer-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,33 +0,0 @@
|
||||
# Editorial collage design
|
||||
|
||||
For each source idea record meaning, emotion, one active verb, a concrete visual
|
||||
metaphor, three to six object groups, and a tactile sound cue. For example, a
|
||||
collaboration concept can become separate cutout hands fitting paper routes
|
||||
together; it need not become a screen full of the word "collaboration".
|
||||
|
||||
Style signature: flat bold color field, black-and-white halftone photographic
|
||||
cutouts, selective colored cardstock accents, warm cream keylines, soft physical
|
||||
paper shadows, fine uncoated fiber, refined irregular torn edges, layered seams.
|
||||
Control texture so the video feels handmade without becoming dirty or distressed.
|
||||
|
||||
Palette choices may support meaning: orange/red for urgency, mustard for tools or
|
||||
warning, green for reflection, purple for memory, teal for collaboration, rose for
|
||||
theatrical contrast. These are suggestions, not universal semantic rules. Use the
|
||||
user's palette and avoid an unrelated kraft-paper opening.
|
||||
|
||||
Still specification: source meaning → metaphor → ratio → background/accent colors
|
||||
→ object roles → foreground/midground/background → final composition → exclusions.
|
||||
|
||||
Motion specification: matching empty color field → base paper structures → main
|
||||
subject → secondary groups → light bounce/press-flat/pause → completed composition.
|
||||
Explicitly describe material boundaries and which piece moves at each beat. A
|
||||
static camera is usually enough to reveal assembly; add movement only when useful.
|
||||
|
||||
Sound specification: synchronize restrained slides, pops, taps, rustles, and snaps.
|
||||
Set `non_diegetic_music: N/A` for the default SFX-only H3 prompt. Do not use silence
|
||||
as a synonym for no background music. Dialogue and subtitles are optional layers.
|
||||
|
||||
Review one established clip as a style/cadence reference for subsequent clips.
|
||||
Change the specific failure: fewer groups for clutter, more physical arrival
|
||||
steps for digital drifting, less aging for dirty paper, stronger final state for
|
||||
composition drift. Do not apply an unrelated aesthetic preset to fix one defect.
|
||||
@@ -1,68 +0,0 @@
|
||||
---
|
||||
name: papercraft-stop-motion-explainer
|
||||
description: Turn knowledge topics into layered papercraft stop-motion explainers, including visual metaphors, miniature sets, prompts, storyboards, motion and sound plans, or rendered video. Use for tactile paper dioramas and mechanisms, not ordinary vector cartoons or photographic collage.
|
||||
metadata:
|
||||
compatibility: Portable file-based workflow with sibling h3-prompt-writing; actual media generation uses tools available in the host environment.
|
||||
---
|
||||
|
||||
# Papercraft Stop-Motion Explainer
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
For scene/prompt planning read [paper staging](references/paper-staging.md).
|
||||
Route a single image prompt, image series, short animation prompt, or storyboard
|
||||
request directly to that artifact; do not force a full film workflow.
|
||||
|
||||
## Knowledge and direction
|
||||
|
||||
Identify the exact topic, source facts, audience, desired learning outcome, and
|
||||
output type. Preserve technical terms and causal meaning. A visual metaphor helps
|
||||
explain a mechanism; it must not turn into a false scientific claim.
|
||||
|
||||
Reuse requested duration and ratio. If unspecified, a 30-second, 16:9 general-audience
|
||||
concept is a reasonable starting point. Offer distinct directions only when useful:
|
||||
pop-up-book journey, paper laboratory, layered cross-section, miniature nature
|
||||
theater, or mechanism board. Each should identify one learning outcome, a paper
|
||||
mechanism that conveys it, and the signature visual moment.
|
||||
|
||||
## Build the paper world
|
||||
|
||||
Design characters only if they help the explanation. Specify paper construction,
|
||||
silhouette, joint/hinge design, replaceable facial parts, gestures, and repeatable
|
||||
identity cues. Keep the central knowledge object readable.
|
||||
|
||||
Plan foreground occluders, midground mechanism, background scenery, and a distant
|
||||
paper plane when the shot benefits from depth. Record each layer's material,
|
||||
thickness, cut/fold edges, shadow, movement, and educational role. Use real paper
|
||||
mechanics such as pull-tabs, sliders, rotating discs, page folds, and pinned joints.
|
||||
Motion is stepped with small pauses and rebounds, not a smooth CG melt.
|
||||
|
||||
Create an asset list by function: characters, scientific objects, information cards,
|
||||
environment, and moving mechanisms. For a complete production, prepare one to three
|
||||
preview briefs or images when useful and authorized. Respect any preview approval
|
||||
requested by the user; otherwise continue under the existing end-to-end scope.
|
||||
|
||||
## Prompts and timeline
|
||||
|
||||
Write a compact storyboard with columns: time, knowledge beat, visual action,
|
||||
paper movement, camera/transition, sound cue. Each shot explains one knowledge
|
||||
step, with a readable pause for a necessary label. Keep ratio, material, light
|
||||
direction, identity, and domain vocabulary consistent across image-series prompts.
|
||||
|
||||
For a short image-to-video prompt, begin from the actual image and animate a few
|
||||
paper mechanisms with restrained push-in or lateral parallax. Preserve thickness,
|
||||
cut edges, texture, and layer shadows. Compile final rendering prompts using the
|
||||
bundled H3 mode guide, with truthful references and local clip timing.
|
||||
|
||||
## Sound, generation, and review
|
||||
|
||||
Map paper flips, cardboard slides, joint taps, soft pops, and rustles to actual
|
||||
motion. Match any requested music to the topic and cultural context rather than
|
||||
automatically using the same whimsical score. Keep speech intelligible. If narration
|
||||
is requested, preserve its approved wording, measure its duration, and adjust the
|
||||
timeline or script within the user's scope; do not silently speed up the voice.
|
||||
|
||||
Use available generation/editing tools, then inspect both educational accuracy
|
||||
and paper construction: readable mechanism, separate layers/shadows, hinged or
|
||||
stepped motion, plausible camera movement, and a clear ending takeaway. Check
|
||||
narration tails, clicks, audio gaps, and music balance. Deliver requested artifacts
|
||||
and state missing rendering capabilities or unresolved scientific/visual issues.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Papercraft Explainers"
|
||||
short_description: "Explain ideas with layered paper mechanisms"
|
||||
default_prompt: "Use $papercraft-stop-motion-explainer to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,48 +0,0 @@
|
||||
# Paper staging and prompt building blocks
|
||||
|
||||
## Materials and depth
|
||||
|
||||
Use layered matte cardboard, visible thickness, fibers, folds, seams, tabs, brads,
|
||||
cut edges, and physical shadows. Separate foreground, mechanism, background, and
|
||||
far background with spacing. Roughly 4–7 layers often suffice; remove layers that
|
||||
hide the explanation. Place reading-heavy labels on stable planes. Use miniature
|
||||
macro photography and moderate depth of field without blurring necessary facts.
|
||||
|
||||
| Plane | Useful content | Motion | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| Foreground | Paper leaves, frame, curtain, props | Small parallax or hinged reveal | Depth without concealing the mechanism |
|
||||
| Midground | Knowledge model, host, arrow/cards | Pull-tab, slider, joint, rotating disc | Main cause and effect |
|
||||
| Background | Diorama set or schematic layers | Slow paper rail shift | Context and hierarchy |
|
||||
| Far plane | Paper sky/board | Restrained change | Stable visual support |
|
||||
|
||||
## Prompt ingredients
|
||||
|
||||
Image: topic and core object → miniature diorama → layered paper construction →
|
||||
specific material edges → physical light/shadows → clear depth → necessary labels.
|
||||
One frame should communicate one main idea, not the entire lesson at once.
|
||||
|
||||
Image series: repeat the style/identity anchors, then state what changes and why
|
||||
that state explains the next knowledge step. Keep the same ratio and light direction.
|
||||
|
||||
Short animation: preserve the supplied image, make a small paper mechanism move,
|
||||
include tiny pauses/rebounds, and select one restrained camera move. Small puppet
|
||||
gestures and background rails can provide life without overwhelming the model.
|
||||
|
||||
## Camera and transitions
|
||||
|
||||
Use slow push-in, lateral parallax, static demonstration, macro detail, or a slight
|
||||
top-down cross-section view. A full orbit requires a set built for that view.
|
||||
Transitions may use a page flip, pop-up unfold, pull-tab, paper cloud, circular
|
||||
mask, cut-paper door, layer separation, confetti, or sticker reveal. Avoid liquid
|
||||
morphing, neon glitches, and glass shatter unless explicitly requested as contrast.
|
||||
|
||||
## Sound and exclusions
|
||||
|
||||
Use a few motion-specific accents: paper flip, scissors, cardboard slide, joint
|
||||
tap, tape peel, or box opening. A score, when appropriate, can use light percussion,
|
||||
plucked instruments, or topic-specific instrumentation. Do not infer narration
|
||||
merely because the video teaches something.
|
||||
|
||||
Keep constraints concise and relevant: no glossy plastic, smooth CG morphs, metal
|
||||
surfaces replacing paper, missing thickness/shadows, or frantic camera movement.
|
||||
Include them in supported prompt fields; do not invent a negative-prompt API field.
|
||||
@@ -1,69 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Build and push the ROCm CI image with a registry-backed BuildKit cache. The
|
||||
# AMD builders are ephemeral, so a local Docker cache cannot survive between
|
||||
# jobs. Keeping the cache beside the runtime image also reuses the registry
|
||||
# credentials already configured on the amd-cpu agents.
|
||||
|
||||
REGISTRY="${ROCM_CI_IMAGE_REGISTRY:-rocm/vllm-omni}"
|
||||
DOCKERFILE="docker/Dockerfile.rocm"
|
||||
BUILDER_NAME="vllm-omni-rocm-builder"
|
||||
ROCM_ARCH="${PYTORCH_ROCM_ARCH:-gfx942}"
|
||||
|
||||
if [[ -z "${BUILDKITE_COMMIT:-}" ]]; then
|
||||
echo "ERROR: BUILDKITE_COMMIT is not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
dependency_files=(
|
||||
"$DOCKERFILE"
|
||||
"${DOCKERFILE}.dockerignore"
|
||||
pyproject.toml
|
||||
setup.py
|
||||
tools/install_torchcodec_rocm.sh
|
||||
)
|
||||
while IFS= read -r dependency_file; do
|
||||
dependency_files+=("$dependency_file")
|
||||
done < <(find requirements -maxdepth 1 -type f -name '*.txt' -print | LC_ALL=C sort)
|
||||
|
||||
for dependency_file in "${dependency_files[@]}"; do
|
||||
if [[ ! -f "$dependency_file" ]]; then
|
||||
echo "ERROR: cache-key input does not exist: $dependency_file" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
CACHE_KEY=$(
|
||||
{
|
||||
printf 'rocm-arch=%s\n' "$ROCM_ARCH"
|
||||
for dependency_file in "${dependency_files[@]}"; do
|
||||
printf 'path=%s\n' "$dependency_file"
|
||||
sha256sum "$dependency_file"
|
||||
done
|
||||
} | sha256sum | cut -c1-16
|
||||
)
|
||||
CACHE_REF="${REGISTRY}:rocm-deps-cache-${CACHE_KEY}"
|
||||
IMAGE_REF="${REGISTRY}:${BUILDKITE_COMMIT}"
|
||||
|
||||
echo "ROCm dependency cache: ${CACHE_REF}"
|
||||
echo "ROCm CI image: ${IMAGE_REF}"
|
||||
|
||||
# The docker-container driver supports exporting the full cache to a registry;
|
||||
# the default docker driver on CI does not.
|
||||
if ! docker buildx inspect "$BUILDER_NAME" >/dev/null 2>&1; then
|
||||
docker buildx create --name "$BUILDER_NAME" --driver docker-container
|
||||
fi
|
||||
docker buildx use "$BUILDER_NAME"
|
||||
docker buildx inspect --bootstrap
|
||||
|
||||
docker buildx build --push --pull --progress=plain \
|
||||
--target test \
|
||||
--build-arg "ARG_PYTORCH_ROCM_ARCH=${ROCM_ARCH}" \
|
||||
--cache-from "type=registry,ref=${CACHE_REF}" \
|
||||
--cache-to "type=registry,ref=${CACHE_REF},mode=max,compression=zstd,ignore-error=true" \
|
||||
--file "$DOCKERFILE" \
|
||||
--tag "$IMAGE_REF" .
|
||||
@@ -9,26 +9,15 @@ steps:
|
||||
steps:
|
||||
- label: "Simple · Model Executor Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
parallelism: 3
|
||||
parallelism: 2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
|
||||
# that is not supported on CPU
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu and not omni' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Simple · Omni Processor Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- "pytest -sv tests/model_executor/models/test_omni_processing.py -m 'core_model and cpu and omni'"
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Simple · Diffusion Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
@@ -36,38 +25,11 @@ steps:
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 45
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
|
||||
# that is not supported on CPU
|
||||
# Keep multi-GPU tests out of this one-GPU pool even when their module
|
||||
# also carries the cpu marker for unit tests.
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu and not (cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)' --ignore=tests/diffusion/cache/test_teacache_extractors.py --ignore=tests/diffusion/models/pi05/test_pi05_units.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
# This module takes about 22 minutes on MI300 and was consistently assigned to shard 4,
|
||||
# pushing that shard to the 45-minute timeout. Keep its module-scoped fixture in one lane.
|
||||
- label: "Simple · Pi0.5 CPU Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 45
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- "pytest -sv tests/diffusion/models/pi05/test_pi05_units.py -m 'core_model and cpu'"
|
||||
|
||||
- label: "Diffusion · AuK VAE GPU Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 35m
|
||||
pytest -sv tests/diffusion/models/auk/test_auk_vae_fastpath.py
|
||||
-m 'core_model and rocm and cards_1' --run-level core_model
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu' --ignore=tests/diffusion/cache/test_teacache_extractors.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Simple · Engine&Entrypoints Test"
|
||||
agent_pool: mi300_1
|
||||
@@ -157,11 +119,6 @@ steps:
|
||||
grade: Blocking
|
||||
commands:
|
||||
- pytest -s -v tests/diffusion/distributed/test_tensor_parallel.py
|
||||
- >-
|
||||
timeout 20m pytest -s -v
|
||||
tests/diffusion/attention/test_ulysses_uaa.py::test_ulysses_uaa_2d_mask_layout_matches_baseline
|
||||
-m 'core_model and cards_2'
|
||||
--run-level "core_model"
|
||||
|
||||
- label: "Diffusion Model Test"
|
||||
agent_pool: mi300_1
|
||||
@@ -169,9 +126,7 @@ steps:
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
commands:
|
||||
# A cold Z-Image run spends about six minutes loading weights before
|
||||
# the first-request AITER compilation, which can itself exceed 23m.
|
||||
- timeout 55m pytest -s -v tests/e2e/offline_inference/test_z_image.py
|
||||
- timeout 30m pytest -s -v tests/e2e/offline_inference/test_z_image.py
|
||||
|
||||
- label: "Diffusion GPU Worker Test"
|
||||
agent_pool: mi300_2
|
||||
|
||||
@@ -64,21 +64,3 @@ steps:
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=1
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni_expansion.py::test_text_to_text_audio_001 -m "full_model and rocm and MI325 and cards_2" --run-level "full_model"
|
||||
|
||||
- group: ":speaker: CosyVoice3 MI300 Nightly"
|
||||
steps:
|
||||
# Exercise all sync/async and Chinese/English cases with the platform's
|
||||
# default attention backend. On ROCm this retains end-to-end AITER
|
||||
# coverage without putting its cold JIT build on the READY critical path.
|
||||
- label: "CosyVoice3-TTS E2E Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 90
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: 134
|
||||
limit: 1
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py -m "slow" --run-level "core_model"
|
||||
@@ -9,65 +9,26 @@ steps:
|
||||
steps:
|
||||
- label: "Simple · Model Executor Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
parallelism: 3
|
||||
parallelism: 2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
|
||||
# that is not supported on CPU
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu and not omni' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Simple · Omni Processor Test"
|
||||
- label: "Simple · Diffusion Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- "pytest -sv tests/model_executor/models/test_omni_processing.py -m 'core_model and cpu and omni'"
|
||||
|
||||
- label: "Simple · Diffusion Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
parallelism: 4
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 45
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
|
||||
# that is not supported on CPU
|
||||
# Keep multi-GPU tests out of this one-GPU pool even when their module
|
||||
# also carries the cpu marker for unit tests.
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu and not (cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)' --ignore=tests/diffusion/cache/test_teacache_extractors.py --ignore=tests/diffusion/models/pi05/test_pi05_units.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
# This module takes about 22 minutes on MI300 and was consistently assigned to shard 4,
|
||||
# pushing that shard to the 45-minute timeout. Keep its module-scoped fixture in one lane.
|
||||
- label: "Simple · Pi0.5 CPU Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 45
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- "pytest -sv tests/diffusion/models/pi05/test_pi05_units.py -m 'core_model and cpu'"
|
||||
|
||||
- label: "Diffusion · AuK VAE GPU Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 35m
|
||||
pytest -sv tests/diffusion/models/auk/test_auk_vae_fastpath.py
|
||||
-m 'core_model and rocm and cards_1' --run-level core_model
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu' --ignore=tests/diffusion/cache/test_teacache_extractors.py"
|
||||
|
||||
- label: "Simple · Engine&Entrypoints Test"
|
||||
agent_pool: mi300_1
|
||||
@@ -108,27 +69,6 @@ steps:
|
||||
|
||||
- group: ":card_index_dividers: Diffusion Test"
|
||||
steps:
|
||||
# Keep real FlashAttention execution out of the CPU-marked Simple lane.
|
||||
# AITER may need to build several kernels on a cold worker, so run one
|
||||
# pytest worker with job-local build directories and a bounded timeout.
|
||||
- label: "ROCm · FlashAttention/AITER GPU Coverage"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=1
|
||||
- export AITER_JIT_DIR="/tmp/vllm-omni-aiter-$$BUILDKITE_JOB_ID"
|
||||
- export TORCH_EXTENSIONS_DIR="/tmp/vllm-omni-torch-extensions-$$BUILDKITE_JOB_ID"
|
||||
- mkdir -p "$$AITER_JIT_DIR" "$$TORCH_EXTENSIONS_DIR"
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 55m
|
||||
pytest -s -v -n 1
|
||||
tests/diffusion/attention/
|
||||
-m 'core_model and rocm and MI325 and cards_1'
|
||||
--run-level "core_model"
|
||||
|
||||
- label: "Diffusion · Batch Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
@@ -167,11 +107,6 @@ steps:
|
||||
- export VLLM_LOGGING_LEVEL=DEBUG
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- timeout 20m pytest -s -v tests/diffusion/distributed/test_sequence_parallel.py -m core_model
|
||||
- >-
|
||||
timeout 20m pytest -s -v
|
||||
tests/diffusion/attention/test_ulysses_uaa.py::test_ulysses_uaa_2d_mask_layout_matches_baseline
|
||||
-m 'core_model and cards_2'
|
||||
--run-level "core_model"
|
||||
- >-
|
||||
timeout 20m pytest -s -v
|
||||
tests/diffusion/models/ltx2/test_ltx2_transformer_ulysses.py
|
||||
@@ -196,7 +131,6 @@ steps:
|
||||
# registered multi-GPU marker from this one-GPU job.
|
||||
- >-
|
||||
timeout 15m pytest -sv tests/diffusion/models/
|
||||
--ignore=tests/diffusion/models/auk/test_auk_vae_fastpath.py
|
||||
-m 'core_model and cuda and not
|
||||
(cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)'
|
||||
--run-level "core_model"
|
||||
@@ -360,23 +294,13 @@ steps:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- timeout 30m pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py -m "core_model" --run-level "core_model"
|
||||
|
||||
# Keep a fast model/server integration check in the blocking lane. The
|
||||
# full default-backend suite remains in the nightly pipeline because a
|
||||
# cold ROCm AITER MHA build can add more than 25 minutes to this job.
|
||||
- label: "CosyVoice3-TTS E2E Smoke (SDPA)"
|
||||
- label: "CosyVoice3-TTS E2E Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
# A ROCm hardware hang aborts the interpreter with 134. Let Buildkite
|
||||
# create one fresh pod instead of retrying on the poisoned GPU process.
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: 134
|
||||
limit: 1
|
||||
commands:
|
||||
- export DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA
|
||||
- pytest -s -v tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py::test_voice_clone_zh_002 -m "slow" --run-level "core_model"
|
||||
- pytest -s -v tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py -m "slow" --run-level "core_model"
|
||||
|
||||
# TODO: Bagel test on ROCm is very unstable. @tjtanaa
|
||||
# Need to debug before reneable numerical changes across large PRs
|
||||
|
||||
@@ -3,8 +3,7 @@
|
||||
Last synced: 2026-08-04
|
||||
Modifications: Use the shared MI300 queues and Omni command-passing contract
|
||||
#}
|
||||
{% set docker_registry_amd = "rocm/vllm-omni" %}
|
||||
{% set docker_image_amd = docker_registry_amd ~ ":$BUILDKITE_COMMIT" %}
|
||||
{% set docker_image_amd = "rocm/vllm-omni:$BUILDKITE_COMMIT" %}
|
||||
{% set default_working_dir = "/app/vllm-omni" %}
|
||||
|
||||
{# MI300 runs tests natively in Kubernetes; Docker-in-Docker is unavailable. #}
|
||||
@@ -75,12 +74,11 @@ steps:
|
||||
depends_on: ~
|
||||
soft_fail: false
|
||||
commands:
|
||||
- "bash .buildkite/amd/scripts/build-ci-image.sh"
|
||||
- "docker build -f docker/Dockerfile.rocm -t {{ docker_image_amd }} --target test --build-arg ARG_PYTORCH_ROCM_ARCH=gfx942 --progress plain ."
|
||||
- "docker push {{ docker_image_amd }}"
|
||||
key: "amd-build"
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
PYTORCH_ROCM_ARCH: "gfx942"
|
||||
ROCM_CI_IMAGE_REGISTRY: "{{ docker_registry_amd }}"
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: -1 # Agent was lost
|
||||
@@ -158,14 +156,6 @@ steps:
|
||||
{% for path in step.artifact_paths %}
|
||||
- "{{ path }}"
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if step.retry %}
|
||||
retry:
|
||||
automatic:
|
||||
{% for retry_rule in step.retry.automatic %}
|
||||
- exit_status: {{ retry_rule.exit_status }}
|
||||
limit: {{ retry_rule.limit }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if step.timeout_in_minutes %}
|
||||
timeout_in_minutes: {{ step.timeout_in_minutes }}
|
||||
@@ -234,14 +224,6 @@ steps:
|
||||
{% for path in step.artifact_paths %}
|
||||
- "{{ path }}"
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if step.retry %}
|
||||
retry:
|
||||
automatic:
|
||||
{% for retry_rule in step.retry.automatic %}
|
||||
- exit_status: {{ retry_rule.exit_status }}
|
||||
limit: {{ retry_rule.limit }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if step.timeout_in_minutes %}
|
||||
timeout_in_minutes: {{ step.timeout_in_minutes }}
|
||||
|
||||
@@ -16,30 +16,13 @@ x-qwen3-tts: &qwen3_tts
|
||||
- vllm_omni/worker/gpu_model_runner.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/fused_sampling.py
|
||||
- vllm_omni/model_executor/models/common/short_kv_attention.py
|
||||
- vllm_omni/model_executor/models/common/talker_first_audio.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/common/alias_free_activation.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
|
||||
- vllm_omni/deploy/qwen3_tts.yaml
|
||||
- vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2_single_gpu.yaml
|
||||
- vllm_omni/entrypoints/openai/serving_speech.py
|
||||
- vllm_omni/entrypoints/openai/tts_adapters/
|
||||
- vllm_omni/engine/stage_pool.py
|
||||
- vllm_omni/worker_v2/
|
||||
- vllm_omni/core/sched/
|
||||
- vllm_omni/distributed/omni_connectors/
|
||||
- vllm_omni/config/stage_config.py
|
||||
- vllm_omni/config/omni_config.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
- vllm_omni/engine/stage_engine_core_proc.py
|
||||
- vllm_omni/engine/cuda_mps.py
|
||||
- vllm_omni/engine/stage_runtime.py
|
||||
- vllm_omni/engine/stage_init_utils.py
|
||||
- vllm_omni/deploy/qwen3_tts_mrv2.yaml
|
||||
- vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2.yaml
|
||||
- vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2_b4.yaml
|
||||
|
||||
x-minicpmo-4-5: &minicpmo_4_5
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
@@ -61,7 +44,6 @@ x-minicpmo-4-5-duplex: &minicpmo_4_5_duplex
|
||||
- vllm_omni/entrypoints/duplex/
|
||||
- vllm_omni/entrypoints/duplex_omni.py
|
||||
- vllm_omni/entrypoints/async_omni_base.py
|
||||
- vllm_omni/protocol/
|
||||
- vllm_omni/clients/
|
||||
- vllm_omni/outputs/duplex.py
|
||||
- vllm_omni/model_executor/duplex_sampling.py
|
||||
@@ -88,16 +70,11 @@ x-qwen3-omni: &qwen3_omni
|
||||
- vllm_omni/entrypoints/openai/serving_chat.py
|
||||
- vllm_omni/model_executor/models/qwen3_omni/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/fused_sampling.py
|
||||
- vllm_omni/model_executor/models/common/short_kv_attention.py
|
||||
- vllm_omni/model_executor/models/common/talker_first_audio.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_thinker.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_omni.py
|
||||
- vllm_omni/deploy/qwen3_omni_moe.yaml
|
||||
- vllm_omni/deploy/qwen3_omni_moe_mrv2.yaml
|
||||
- vllm_omni/deploy/qwen3_omni_moe_mrv2_h200.yaml
|
||||
- vllm_omni/deploy/qwen3_omni_moe_thinking.yaml
|
||||
|
||||
x-wan22: &wan22
|
||||
@@ -122,7 +99,6 @@ x-hunyuan-image3: &hunyuan_image3
|
||||
|
||||
x-minimax-h3: &minimax_h3
|
||||
- vllm_omni/diffusion/models/minimax_h3/
|
||||
- vllm_omni/diffusion/layers/indexed_modulation.py
|
||||
|
||||
x-hunyuan-video: &hunyuan_video
|
||||
- vllm_omni/diffusion/models/hunyuan_video/
|
||||
@@ -200,10 +176,6 @@ source_file_dependencies:
|
||||
|
||||
# --- models ---
|
||||
# --- omni / tts jobs ---
|
||||
entrypoints_function:
|
||||
- tests/entrypoints/
|
||||
- vllm_omni/entrypoints/
|
||||
|
||||
omni_qwen3_omni_function:
|
||||
- *qwen3_omni
|
||||
- tests/e2e/offline_inference/test_qwen3_omni.py
|
||||
@@ -252,14 +224,6 @@ source_file_dependencies:
|
||||
- *minicpmo_4_5
|
||||
- tests/e2e/accuracy/minicpmo_4_5/
|
||||
|
||||
omni_duplex_eval_ci:
|
||||
- *minicpmo_4_5
|
||||
- *minicpmo_4_5_duplex
|
||||
- tests/e2e/accuracy/omni_duplex_eval/**
|
||||
- tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py
|
||||
- vllm_omni/benchmarks/duplex/**
|
||||
- vllm_omni/entrypoints/cli/benchmark/omni_duplex_eval.py
|
||||
|
||||
omni_minicpmo_4_5_perf:
|
||||
- *minicpmo_4_5
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
@@ -279,8 +243,6 @@ source_file_dependencies:
|
||||
- *minicpmo_4_5_duplex
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_duplex.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_window.py
|
||||
- tests/e2e/online_serving/helpers/minicpmo_window_e2e.py
|
||||
- tests/e2e/online_serving/test_duplex_client_live.py
|
||||
- tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py
|
||||
- tests/e2e/online_serving/helpers/minicpmo_realtime_duplex_scenarios.py
|
||||
@@ -297,7 +259,6 @@ source_file_dependencies:
|
||||
- tests/entrypoints/test_duplex_omni.py
|
||||
- tests/entrypoints/openai_api/test_duplex_api_server.py
|
||||
- tests/model_executor/models/minicpmo_4_5/duplex/
|
||||
- tests/protocol/
|
||||
|
||||
omni_minicpmo_4_5_duplex_perf:
|
||||
- *minicpmo_4_5
|
||||
@@ -343,7 +304,7 @@ source_file_dependencies:
|
||||
|
||||
diffusion_wan22_perf:
|
||||
- *wan22
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/scripts/run_diffusion_benchmark.py
|
||||
- tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json
|
||||
|
||||
diffusion_wan22_reliability:
|
||||
@@ -369,7 +330,7 @@ source_file_dependencies:
|
||||
|
||||
diffusion_cosmos3_perf:
|
||||
- *cosmos3
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/scripts/run_diffusion_benchmark.py
|
||||
- tests/dfx/perf/tests/test_cosmos3_vllm_omni.json
|
||||
|
||||
diffusion_qwen_image_function:
|
||||
@@ -414,8 +375,6 @@ source_file_dependencies:
|
||||
diffusion_hunyuan_image3_accuracy:
|
||||
- *hunyuan_image3
|
||||
- tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py
|
||||
- tests/e2e/accuracy/test_hunyuan_image3_lora.py
|
||||
- vllm_omni/diffusion/lora/
|
||||
|
||||
diffusion_hunyuan_image3_perf:
|
||||
- *hunyuan_image3
|
||||
@@ -430,7 +389,6 @@ source_file_dependencies:
|
||||
|
||||
diffusion_minimax_h3_function:
|
||||
- *minimax_h3
|
||||
- tests/helpers/assertions.py
|
||||
- tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py
|
||||
- tests/e2e/online_serving/minimax_h3/
|
||||
- tests/diffusion/models/minimax_h3/
|
||||
@@ -445,7 +403,7 @@ source_file_dependencies:
|
||||
|
||||
diffusion_minimax_h3_perf:
|
||||
- *minimax_h3
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/scripts/run_diffusion_benchmark.py
|
||||
- tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json
|
||||
|
||||
diffusion_hunyuan_video_function:
|
||||
@@ -459,7 +417,7 @@ source_file_dependencies:
|
||||
|
||||
diffusion_hunyuan_video_perf:
|
||||
- *hunyuan_video
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/scripts/run_diffusion_benchmark.py
|
||||
- tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json
|
||||
- tests/dfx/perf/tests/test_hunyuanvideo15_i2v_vllm_omni.json
|
||||
|
||||
@@ -482,15 +440,9 @@ source_file_dependencies:
|
||||
- *lingbot_video
|
||||
- *lingbot_world
|
||||
- tests/e2e/online_serving/test_lingbot_video.py
|
||||
- tests/e2e/online_serving/test_lingbot_video_expansion.py
|
||||
- tests/e2e/online_serving/test_lingbot_video_moe.py
|
||||
- tests/e2e/offline_inference/test_lingbot_world_v2.py
|
||||
|
||||
diffusion_lingbot_perf:
|
||||
- *lingbot_video
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_lingbot_video_vllm_omni.json
|
||||
|
||||
|
||||
# Invalid-param jobs: model business code only, plus the reliability scripts.
|
||||
reliability_invalid_param_h100:
|
||||
|
||||
@@ -18,7 +18,7 @@ steps:
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Simple · Diffusion Test"
|
||||
timeout_in_minutes: 30
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu'"
|
||||
mirror_hardwares: l4_1
|
||||
@@ -60,7 +60,11 @@ steps:
|
||||
depends_on: upload-merge-pipeline
|
||||
steps:
|
||||
- label: "Entrypoint Test with H100"
|
||||
timeout_in_minutes: 60
|
||||
# TEMPORARY 60 -> 90: engine startup/sleep-wake is ~13-15 min slower per
|
||||
# run on the vLLM 0.26 image (model loading ~2x, init ~2x; see builds
|
||||
# 2915 vs 2916), which pushed this suite past 60 min. Revert once the
|
||||
# 0.26 startup regression is understood and fixed.
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
# Turn-based server VAD over /v1/realtime is deferred to RFC #7181 PR 5
|
||||
# (see the P0 thread on #7413); the test stays in the tree and runs again there.
|
||||
@@ -69,25 +73,10 @@ steps:
|
||||
|
||||
- label: "Entrypoint Test with L4"
|
||||
timeout_in_minutes: 60
|
||||
# Workaround https://github.com/vllm-project/vllm-omni/issues/8016: shutdown()
|
||||
# leaves inline UniProc DiffusionWorker VRAM mapped in the pytest parent.
|
||||
# This job does not collect the ready leftover
|
||||
# (test_diffusion_generate_request_id, core_model); isolate the
|
||||
# advanced_model DiT handshake instead. Revert to a single sweep after
|
||||
# the product shutdown fix.
|
||||
commands:
|
||||
- |
|
||||
set -eu
|
||||
pytest -s -v tests/entrypoints/test_omni_sleep_mode.py::TestOmniDiffusionSleepMode::test_diffusion_sleep_handshake -m "advanced_model and cuda and L4 and cards_1" --run-level "advanced_model"
|
||||
pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and L4 and cards_1" --run-level "advanced_model" --deselect tests/entrypoints/test_omni_sleep_mode.py::TestOmniDiffusionSleepMode::test_diffusion_sleep_handshake
|
||||
- pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and L4" --run-level "advanced_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Entrypoint Test with L4 · 2-GPU"
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and L4 and cards_2" --run-level "advanced_model"
|
||||
mirror_hardwares: l4_2
|
||||
|
||||
- group: ":card_index_dividers: E2E Test"
|
||||
depends_on: upload-merge-pipeline
|
||||
steps:
|
||||
@@ -124,30 +113,6 @@ steps:
|
||||
- pytest -s -v tests/e2e/offline_inference/test_qwen3_omni.py tests/e2e/online_serving/test_qwen3_omni.py tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Omni · Qwen3-Omni Async-Chunk SHM Leak"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_omni_shm_leak.py
|
||||
- tests/helpers/runtime.py
|
||||
- tests/helpers/stage_config.py
|
||||
- vllm_omni/core/sched/omni_ar_scheduler.py
|
||||
- vllm_omni/distributed/omni_connectors/connectors/shm_connector.py
|
||||
- vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
- vllm_omni/engine/stage_engine_core_proc.py
|
||||
- vllm_omni/engine/stage_pool.py
|
||||
- vllm_omni/entrypoints/openai/serving_chat.py
|
||||
- vllm_omni/model_executor/models/qwen3_omni/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_thinker.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_omni.py
|
||||
- vllm_omni/deploy/qwen3_omni_moe.yaml
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni_shm_leak.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_function
|
||||
timeout_in_minutes: 50
|
||||
@@ -165,7 +130,7 @@ steps:
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_function
|
||||
timeout_in_minutes: 50
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_minicpmo_4_5_window.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Qwen Image Test"
|
||||
@@ -198,16 +163,6 @@ steps:
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · LingBot-Video Test"
|
||||
source_file_dependencies: diffusion_lingbot_function
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_lingbot_video.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · MiniMax-H3 L3 DLO DP2 and FastH3"
|
||||
source_file_dependencies: diffusion_minimax_h3_function
|
||||
# Keep four server lifecycles in one Buildkite job, but never in one
|
||||
@@ -232,14 +187,3 @@ steps:
|
||||
timeout 35m pytest -s -v tests/e2e/online_serving/minimax_h3/test_minimax_h3_turbo_lora.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
timeout 35m pytest -s -v tests/e2e/online_serving/minimax_h3/test_minimax_h3_fasth3_lora.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · MiniMax-H3 Latent Upscale and Refine"
|
||||
source_file_dependencies: diffusion_minimax_h3_function
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- |
|
||||
set -eu
|
||||
VLLM_TEST_MINIMAX_H3_LATENT_UPSCALER=$(python3 -c 'from huggingface_hub import hf_hub_download; print(hf_hub_download(repo_id="LBH-123-AI/Minimax_h3_latent_Upscaler", filename="minimax_h3_latent_upscaler_3d_conv_v1/minimax_h3_latent_upscaler_3d_conv_v1_bf16.safetensors"))')
|
||||
export VLLM_TEST_MINIMAX_H3_LATENT_UPSCALER
|
||||
timeout 35m pytest -s -v tests/e2e/online_serving/minimax_h3/test_minimax_h3.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_2
|
||||
@@ -21,7 +21,7 @@ steps:
|
||||
- omni_minicpmo_4_5_function
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- pytest -sv tests/e2e --run-level "full_model" -m "full_model and H100 and B200 and omni and cards_1" --ignore=tests/e2e/accuracy --ignore=tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py --ignore=tests/e2e/online_serving/test_minicpmo_4_5_window.py
|
||||
- pytest -sv tests/e2e --run-level "full_model" -m "full_model and H100 and B200 and omni and cards_1" --ignore=tests/e2e/accuracy --ignore=tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py
|
||||
|
||||
- label: ":full_moon: Omni · Function Test with H100 · 2-GPU"
|
||||
source_file_dependencies:
|
||||
@@ -33,10 +33,9 @@ steps:
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 Duplex Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_function
|
||||
timeout_in_minutes: 120
|
||||
timeout_in_minutes: 50
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py -m "full_model and cuda and H100 and B200 and omni and cards_1" --run-level "full_model"
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_window.py -m "full_model and cuda and H100 and B200 and omni and cards_1" --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Omni · Doc Test with L4 · 4-GPU"
|
||||
source_file_dependencies: omni_qwen2_5_omni_doc
|
||||
@@ -54,78 +53,80 @@ steps:
|
||||
- label: ":full_moon: Omni · Accuracy Test"
|
||||
source_file_dependencies: omni_qwen3_omni_accuracy
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/e2e/accuracy/qwen3_omni/results/qwen_omni_acc/*.json
|
||||
commands:
|
||||
- export SEED_TTS_WER_EVAL=1
|
||||
- export SEED_TTS_EVAL_DEVICE=cuda:1
|
||||
- pytest -s -v tests/e2e/accuracy/qwen3_omni/test_qwen3_omni.py -m "full_model and H100 and B200 and cards_2" --run-level full_model
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/e2e/accuracy/qwen3_omni/test_qwen3_omni.py -m "full_model and H100 and B200 and cards_2" --run-level full_model
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/qwen3_omni/results/qwen_omni_acc/*.json"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Accuracy Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_accuracy
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/e2e/accuracy/minicpmo_4_5/results/*.json
|
||||
commands:
|
||||
- export SEED_TTS_WER_EVAL=1
|
||||
- export SEED_TTS_EVAL_DEVICE=cuda:1
|
||||
- pytest -s -v tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py -m "full_model and H100 and B200 and cuda and (cards_1 or cards_2)" --run-level full_model
|
||||
|
||||
- label: ":full_moon: Omni · Omni-DuplexEval CI Guard"
|
||||
source_file_dependencies: omni_duplex_eval_ci
|
||||
timeout_in_minutes: 240
|
||||
env:
|
||||
VLLM_DUPLEX_EVAL_JUDGE_URL: "http://127.0.0.1:8001"
|
||||
commands:
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/e2e/accuracy/omni_duplex_eval/test_omni_duplex_eval_ci.py \
|
||||
-m "full_model and H100 and B200 and cuda and cards_2" \
|
||||
--run-level full_model \
|
||||
--basetemp /tmp/omni_duplex_ci
|
||||
pytest -s -v tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py -m "full_model and H100 and B200 and cuda and (cards_1 or cards_2)" --run-level full_model
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/omni_duplex_eval/results/*.json"
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/minicpmo_4_5/results/*.json"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · No Async Chunk"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: nightly-omni-performance-no-async-chunk
|
||||
timeout_in_minutes: 300
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json -m "H100 and B200 and cards_2"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json -m "H100 and B200 and cards_2"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · Async Chunk"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: nightly-omni-performance-async-chunk
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json -m "H100 and B200 and full_model and cards_2"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json -m "H100 and B200 and full_model and cards_2"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Perf Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_perf
|
||||
key: nightly-omni-performance-minicpmo-4-5
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5.json -m "H100 and B200 and cards_1"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5.json -m "H100 and B200 and cards_1"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Duplex Seed-TTS Perf Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_perf
|
||||
key: nightly-omni-performance-minicpmo-4-5-duplex-seed-tts
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json -m "H100 and B200 and cards_1"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json -m "H100 and B200 and cards_1"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: Omni · Multi-Replica Startup Test with 4x H100"
|
||||
source_file_dependencies: omni_qwen3_omni_function
|
||||
@@ -133,20 +134,6 @@ steps:
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni_multi_replicas.py -m "full_model and H100 and B200 and cards_4" --run-level "core_model"
|
||||
|
||||
- group: ":card_index_dividers: Entrypoints Test"
|
||||
key: nightly-entrypoints-test-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test"
|
||||
steps:
|
||||
|
||||
- label: ":full_moon: Entrypoint Test with H100"
|
||||
source_file_dependencies: entrypoints_function
|
||||
# Merge collects advanced_model (5 duplex interruptions). Nightly
|
||||
# collects full_model (20 interruptions + BAGEL sleep).
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/entrypoints/ -m "full_model and cuda and H100 and B200 and cards_2" --run-level "full_model"
|
||||
|
||||
- group: ":card_index_dividers: TTS Model Test"
|
||||
key: nightly-tts-test-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
@@ -166,23 +153,29 @@ steps:
|
||||
source_file_dependencies: tts_qwen3_tts_perf
|
||||
key: nightly-tts-performance-single
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json -m "H100 and B200 and cards_1"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json -m "H100 and B200 and cards_1"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: TTS · Perf Test · 2-GPU"
|
||||
source_file_dependencies: tts_qwen3_tts_perf
|
||||
key: nightly-tts-performance-2gpu
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json -m "H100 and B200 and cards_2"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json -m "H100 and B200 and cards_2"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
|
||||
- group: ":card_index_dividers: Tiny Model Tests [Multi-GPU]"
|
||||
key: nightly-tiny-model-test-group
|
||||
@@ -267,33 +260,39 @@ steps:
|
||||
commands:
|
||||
- export HUNYUAN_IMAGE3_MODEL="tencent/HunyuanImage-3.0-Instruct"
|
||||
- export HUNYUAN_IMAGE3_DEVICES="0,1,2,3"
|
||||
- pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py tests/e2e/accuracy/test_hunyuan_image3_lora.py -m "full_model and H100 and B200 and cards_4" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py -m "full_model and H100 and B200 and cards_4" --run-level full_model
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image · Single-GPU"
|
||||
source_file_dependencies: diffusion_qwen_image_perf
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image-single
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/diffusion_result_*.json
|
||||
- tests/dfx/perf/results/logs/*.log
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export CACHE_DIT_VERSION=1.5.0
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json -m "H100 and B200 and cards_1"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json -m "H100 and B200 and cards_1"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image · 4-GPU"
|
||||
source_file_dependencies: diffusion_qwen_image_perf
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image-4gpu
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/diffusion_result_*.json
|
||||
- tests/dfx/perf/results/logs/*.log
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export CACHE_DIT_VERSION=1.5.0
|
||||
# Do not pin DIFFUSION_ATTENTION_BACKEND: H100 auto-selects FLASH_ATTN;
|
||||
# B200 rejects explicit FLASH_ATTN without FA4 and uses CUDNN/TRTLLM.
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json -m "H100 and B200 and cards_4"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json -m "H100 and B200 and cards_4"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
|
||||
- group: ":card_index_dividers: Diffusion X2V Model Test"
|
||||
key: nightly-diffusion-x2v-group
|
||||
@@ -339,7 +338,7 @@ steps:
|
||||
commands:
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/e2e/online_serving/test_lingbot_video_expansion.py
|
||||
tests/e2e/online_serving/test_lingbot_video.py
|
||||
tests/e2e/online_serving/test_lingbot_video_moe.py
|
||||
tests/e2e/offline_inference/test_lingbot_world_v2.py
|
||||
-m "full_model and diffusion and H100 and B200 and cards_1"
|
||||
@@ -389,11 +388,15 @@ steps:
|
||||
source_file_dependencies: diffusion_wan22_perf
|
||||
key: nightly-diffusion-x2v-performance-single
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json -m "H100 and B200 and cards_1"
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json -m "H100 and B200 and cards_1"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test · 2-GPU"
|
||||
source_file_dependencies:
|
||||
@@ -401,16 +404,16 @@ steps:
|
||||
- diffusion_cosmos3_perf
|
||||
key: nightly-diffusion-x2v-performance-2gpu
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json -m "H100 and B200 and cards_2"
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json -m "H100 and B200 and cards_2"
|
||||
EXIT1=$$?
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json -m "H100 and B200 and cards_2"
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json -m "H100 and B200 and cards_2"
|
||||
EXIT2=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
if [ $$EXIT1 -ne 0 ]; then exit $$EXIT1; fi
|
||||
exit $$EXIT2
|
||||
|
||||
@@ -418,16 +421,16 @@ steps:
|
||||
source_file_dependencies: diffusion_hunyuan_video_perf
|
||||
key: nightly-diffusion-x2v-performance-hunyuanvideo15
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json -m "H100 and B200 and (cards_1 or cards_2)"
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json -m "H100 and B200 and (cards_1 or cards_2)"
|
||||
EXIT1=$$?
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_i2v_vllm_omni.json -m "H100 and B200 and (cards_1 or cards_2)"
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_i2v_vllm_omni.json -m "H100 and B200 and (cards_1 or cards_2)"
|
||||
EXIT2=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
if [ $$EXIT1 -ne 0 ]; then exit $$EXIT1; fi
|
||||
exit $$EXIT2
|
||||
|
||||
@@ -435,21 +438,15 @@ steps:
|
||||
source_file_dependencies: diffusion_minimax_h3_perf
|
||||
key: nightly-diffusion-x2v-performance-minimax-h3
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json -m "H100 and B200 and cards_4"
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test · LingBot"
|
||||
source_file_dependencies: diffusion_lingbot_perf
|
||||
key: nightly-diffusion-x2v-performance-lingbot
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_lingbot_video_vllm_omni.json -m "H100 and B200 and cards_1"
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json -m "H100 and B200 and cards_4"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
|
||||
- group: ":card_index_dividers: Diffusion Test"
|
||||
key: nightly-diffusion-test-group
|
||||
@@ -525,9 +522,6 @@ steps:
|
||||
- nightly-diffusion-x2v-performance-single
|
||||
- nightly-diffusion-x2v-performance-2gpu
|
||||
if: build.env("NIGHTLY") == "1" && build.env("EMAIL_DISTRIBUTION") == "1"
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.xlsx
|
||||
- tests/dfx/perf/results/*.html
|
||||
commands:
|
||||
- pip install openpyxl
|
||||
- export DEFAULT_INPUT_DIR=tests/dfx/perf/results
|
||||
@@ -543,5 +537,7 @@ steps:
|
||||
- python tools/nightly/generate_nightly_perf_excel.py
|
||||
- python tools/nightly/generate_nightly_perf_html.py
|
||||
- python tools/nightly/send_nightly_email.py --report-file "tests/dfx/perf/results/*.xlsx, tests/dfx/perf/results/*.html"
|
||||
- buildkite-agent artifact upload "tests/dfx/perf/results/*.xlsx"
|
||||
- buildkite-agent artifact upload "tests/dfx/perf/results/*.html"
|
||||
agents:
|
||||
queue: "cpu_queue_premerge"
|
||||
@@ -16,27 +16,8 @@ steps:
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu'"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Simple · CosyVoice3 Packed Flow Test"
|
||||
timeout_in_minutes: 15
|
||||
source_file_dependencies:
|
||||
- tests/model_executor/models/cosyvoice3/test_cosyvoice3_full_batch.py
|
||||
- vllm_omni/model_executor/models/cosyvoice3/
|
||||
commands:
|
||||
- "pytest -sv tests/model_executor/models/cosyvoice3/test_cosyvoice3_full_batch.py -k packed -m 'core_model and cuda' --run-level=core_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Simple · MRV2 FA3 Graph Bounds Test"
|
||||
timeout_in_minutes: 15
|
||||
source_file_dependencies:
|
||||
- tests/worker_v2/test_capture_query_bound_gpu.py
|
||||
- tests/worker_v2/test_decode_query_bound_gpu.py
|
||||
- vllm_omni/worker_v2/model_states/omni_model_state.py
|
||||
commands:
|
||||
- "pytest -sv tests/worker_v2/test_capture_query_bound_gpu.py tests/worker_v2/test_decode_query_bound_gpu.py -m 'core_model and cuda and H100' --run-level=core_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Simple · Diffusion Test"
|
||||
timeout_in_minutes: 30
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu'"
|
||||
mirror_hardwares: l4_1
|
||||
@@ -57,12 +38,6 @@ steps:
|
||||
depends_on: upload-ready-pipeline
|
||||
steps:
|
||||
|
||||
- label: "Diffusion · FlashAttention Test"
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- timeout 15m pytest -s -v tests/diffusion/attention/ -m 'core_model and cuda and L4 and cards_1' --run-level "core_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Diffusion · Offloader Test"
|
||||
commands:
|
||||
- timeout 40m pytest -s -v tests/diffusion/offloader -m "core_model and cuda" --run-level "core_model"
|
||||
@@ -88,7 +63,6 @@ steps:
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
commands:
|
||||
- timeout 20m pytest -sv tests/diffusion/distributed -m 'core_model and cuda and L4 and cards_2' --run-level "core_model"
|
||||
- timeout 10m pytest -sv tests/diffusion/attention/test_ulysses_uaa.py -m 'core_model and cuda and L4 and cards_2' --run-level "core_model"
|
||||
mirror_hardwares: l4_2
|
||||
|
||||
- label: "Diffusion · Distributed & Attention Test · 4-GPU"
|
||||
@@ -122,7 +96,7 @@ steps:
|
||||
commands:
|
||||
- |
|
||||
timeout 15m bash -c "
|
||||
pytest -sv tests/engine/ tests/model_executor/ tests/worker/test_batched_omni_output.py tests/worker_v2/ -m 'core_model and cuda' --run-level 'core_model'
|
||||
pytest -sv tests/engine/ tests/model_executor/ tests/worker/test_batched_omni_output.py -m 'core_model and cuda' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
@@ -137,18 +111,11 @@ steps:
|
||||
|
||||
- label: "Entrypoints Test"
|
||||
depends_on: upload-ready-pipeline
|
||||
# Workaround https://github.com/vllm-project/vllm-omni/issues/8016: shutdown()
|
||||
# leaves inline UniProc DiffusionWorker VRAM mapped in the pytest parent.
|
||||
# Isolate that leftover case in its own process so the driver drops the
|
||||
# mapping; the rest stays a directory sweep with --deselect. Revert to a
|
||||
# single sweep after the product shutdown fix.
|
||||
commands:
|
||||
- |
|
||||
timeout 30m bash -c "
|
||||
set -eu
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
pytest -sv tests/entrypoints/test_async_omni.py::test_diffusion_generate_request_id -m 'core_model and cuda' --run-level 'core_model'
|
||||
pytest -sv tests/entrypoints/ -m 'core_model and cuda' --run-level 'core_model' --deselect tests/entrypoints/test_async_omni.py::test_diffusion_generate_request_id
|
||||
pytest -sv tests/entrypoints/ -m 'core_model and cuda' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
@@ -215,16 +182,6 @@ steps:
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5_ready.json -m 'core_model and cuda'
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "TTS · Higgs v3 Full Response Test"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_higgs_audio_v3.py
|
||||
- vllm_omni/model_executor/models/higgs_audio_v3/
|
||||
- vllm_omni/model_executor/stage_input_processors/higgs_audio_v3.py
|
||||
- vllm_omni/deploy/higgs_multimodal_qwen3.yaml
|
||||
commands:
|
||||
- timeout 20m pytest -s -v tests/e2e/online_serving/test_higgs_audio_v3.py::test_full_response_v1_preserves_audio -m 'core_model and cuda' --run-level 'core_model'
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "TTS · Qwen3-TTS CustomVoice Test"
|
||||
source_file_dependencies: tts_qwen3_tts_function
|
||||
commands:
|
||||
@@ -241,7 +198,7 @@ steps:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_wan22_t2v.py -m 'core_model and cuda' --run-level 'core_model'
|
||||
pytest -s -v tests/e2e/online_serving/test_wan22_t2v.py -m 'core_model' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
@@ -285,13 +242,3 @@ steps:
|
||||
pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m 'core_model and diffusion' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · LingBot-Video Test"
|
||||
source_file_dependencies: diffusion_lingbot_function
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_lingbot_video.py -m 'core_model and cuda' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
+22
-133
@@ -12,8 +12,6 @@ steps:
|
||||
depends_on: upload-weekly-pipeline
|
||||
if: build.env("WEEKLY") == "1"
|
||||
timeout_in_minutes: 60
|
||||
artifact_paths:
|
||||
- coverage-core-model-cpu-*.xml.gz
|
||||
commands:
|
||||
- |
|
||||
set +e
|
||||
@@ -21,6 +19,7 @@ steps:
|
||||
pytest -sv tests/ -m 'core_model and cpu' --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml:$${REPORT}
|
||||
EXIT=$$?
|
||||
gzip -9 -f "$${REPORT}"
|
||||
buildkite-agent artifact upload "$${REPORT}.gz"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
@@ -80,32 +79,41 @@ steps:
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: weekly-omni-performance-vllm-text
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_vllm_text.json
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_vllm_text.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · Async Chunk · Random"
|
||||
key: weekly-omni-performance-async-chunk-random
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json -m "H100 and slow"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json -m "H100 and slow"
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · Multi-Replica"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: weekly-omni-performance-multi-replicas
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_3
|
||||
|
||||
- group: ":card_index_dividers: E2E Tests"
|
||||
@@ -152,122 +160,15 @@ steps:
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- label: "TTS · Moss · L4"
|
||||
timeout_in_minutes: 90
|
||||
- label: "TTS · L4"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_moss_tts_expansion.py \
|
||||
tests/e2e/offline_inference/test_moss_tts_nano_expansion.py \
|
||||
tests/e2e/offline_inference/test_moss_tts_realtime.py \
|
||||
tests/e2e/offline_inference/test_moss_tts_realtime_expansion.py \
|
||||
tests/e2e/online_serving/test_moss_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_moss_tts_nano_expansion.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · VoxCPM2 · Voxtral · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_voxcpm2_tts.py \
|
||||
tests/e2e/online_serving/test_voxcpm2_tts.py \
|
||||
tests/e2e/online_serving/test_voxcpm2_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_voxcpm2_tts_lora.py \
|
||||
tests/e2e/offline_inference/test_voxtral_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_voxtral_tts_expansion.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · Higgs · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_higgs_audio_v2_expansion.py \
|
||||
tests/e2e/online_serving/test_higgs_audio_v2_expansion.py \
|
||||
tests/e2e/online_serving/test_higgs_audio_v3.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · OmniVoice · Step-Audio · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_omnivoice_expansion.py \
|
||||
tests/e2e/online_serving/test_omnivoice_expansion.py \
|
||||
tests/e2e/online_serving/test_omnivoice_parity.py \
|
||||
tests/e2e/offline_inference/test_step_audio2_expansion.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · Ming · GLM · IndexTTS · Breeze · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_ming_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_ming_tts_expansion.py \
|
||||
tests/e2e/offline_inference/test_glm_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_glm_tts_expansion.py \
|
||||
tests/e2e/offline_inference/test_indextts2_expansion.py \
|
||||
tests/e2e/online_serving/test_indextts2_expansion.py \
|
||||
tests/e2e/online_serving/test_breeze_tts_2.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · Audex · CosyVoice · Qwen3-TTS · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_audex_tts.py \
|
||||
tests/e2e/offline_inference/test_audex_s2t.py \
|
||||
tests/e2e/offline_inference/test_audex_tta.py \
|
||||
tests/e2e/offline_inference/test_cosyvoice3_expansion.py \
|
||||
tests/e2e/online_serving/test_qwen3_tts_word_timestamps.py
|
||||
do
|
||||
for f in $(pytest --collect-only -q tests/e2e/ -m "slow and L4 and tts" --run-level "full_model" | grep '\.py::' | sed -E 's|::.*||' | sort -u); do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
@@ -331,17 +232,6 @@ steps:
|
||||
|
||||
# Split the old shared H100 Single-GPU sweep: one pytest process collected
|
||||
# every slow+cards_1 diffusion file and exceeded 150 minutes.
|
||||
- label: ":robot_face: Robot Policy OpenPI · π0.5 · H100 · Single-GPU"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- pip install --no-deps "openpi-client @ git+https://github.com/Physical-Intelligence/openpi.git@215abfb217dbac7d5f1273282331b9b1866c0479#subdirectory=packages/openpi-client"
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_pi05_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Qwen-Image · H100 · Single-GPU"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
@@ -534,7 +424,6 @@ steps:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_bagel_expansion.py
|
||||
tests/e2e/offline_inference/test_bagel_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
@@ -10,7 +10,7 @@ steps:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
# Buildkite will automatically replace this with the actual commit hash
|
||||
VLLM_IMAGE_TAG: "${BUILDKITE_COMMIT}"
|
||||
VLLM_VERSION: "v0.30.0"
|
||||
VLLM_VERSION: "v0.29.0"
|
||||
priority: 100
|
||||
timeout_in_minutes: 60
|
||||
soft_fail: false
|
||||
@@ -64,11 +64,6 @@ steps:
|
||||
tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py
|
||||
-m "npu and A3 and omni"
|
||||
--run-level full_model
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/e2e/online_serving/test_minicpmo_4_5_window.py
|
||||
-m "full_model and npu and A3 and omni"
|
||||
--run-level full_model
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Accuracy Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_accuracy
|
||||
timeout_in_minutes: 180
|
||||
@@ -90,25 +85,6 @@ steps:
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/minicpmo_4_5/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Omni · Omni-DuplexEval CI Guard"
|
||||
source_file_dependencies: omni_duplex_eval_ci
|
||||
timeout_in_minutes: 240
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
VLLM_DUPLEX_EVAL_JUDGE_URL: "http://127.0.0.1:8001"
|
||||
commands:
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/e2e/accuracy/omni_duplex_eval/test_omni_duplex_eval_ci.py \
|
||||
-m "full_model and npu and A3 and cards_2" \
|
||||
--run-level full_model \
|
||||
--basetemp /tmp/omni_duplex_ci
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/omni_duplex_eval/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Perf Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_perf
|
||||
key: nightly-omni-performance-minicpmo-4-5-npu
|
||||
@@ -252,42 +228,6 @@ steps:
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image · Single-NPU"
|
||||
source_file_dependencies: diffusion_qwen_image_perf
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image-single-npu
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "npu and A3 and cards_1" --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image · 4-NPU"
|
||||
source_file_dependencies: diffusion_qwen_image_perf
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image-4npu
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "npu and A3 and cards_4" --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
- group: ":card_index_dividers: Diffusion X2V Model Test"
|
||||
key: nightly-diffusion-x2v-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
@@ -295,36 +235,6 @@ steps:
|
||||
build.env("NIGHTLY") == "1" ||
|
||||
build.pull_request.labels includes "nightly-test"
|
||||
steps:
|
||||
- label: ":full_moon: Diffusion X2V · HunyuanVideo-1.5 Function Test"
|
||||
source_file_dependencies: diffusion_hunyuan_video_function
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a2b3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
HF_HUB_OFFLINE: "1"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m "npu and A2" --run-level "full_model"
|
||||
- label: ":full_moon: Diffusion X2V · HunyuanVideo-1.5 Perf Test"
|
||||
source_file_dependencies: diffusion_hunyuan_video_perf
|
||||
key: nightly-diffusion-x2v-performance-hunyuanvideo15
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
HF_HUB_OFFLINE: "1"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py -m "npu and A3" --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Diffusion X2V · Wan 2.2 Function Test"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
mirror_hardwares: a2b3_npu_4
|
||||
@@ -345,10 +255,11 @@ steps:
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py -m npu --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m npu --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
@@ -55,8 +55,8 @@ steps:
|
||||
-m 'core_model and npu and 310P'
|
||||
--run-level 'core_model'
|
||||
|
||||
- group: ":card_index_dividers: E2E Test"
|
||||
key: ready-e2e-group
|
||||
- group: ":card_index_dividers: Omni Model Test"
|
||||
key: ready-omni-group
|
||||
depends_on: upload-ready-pipeline
|
||||
steps:
|
||||
- label: "Omni · MiniCPM-o 4.5 Test"
|
||||
@@ -66,7 +66,6 @@ steps:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/offline_inference/test_minicpmo_4_5.py tests/e2e/online_serving/test_minicpmo_4_5.py -m 'core_model and npu and A3 and omni' --run-level 'core_model'
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Duplex Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
@@ -74,7 +73,6 @@ steps:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py -m 'core_model and npu and A3 and omni' --run-level 'core_model'
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Perf Test"
|
||||
timeout_in_minutes: 60
|
||||
mirror_hardwares: a3_npu_2
|
||||
@@ -85,15 +83,10 @@ steps:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5_ready.json -m 'core_model and npu and A3 and omni'
|
||||
|
||||
- label: "Omni · Qwen3-Omni Test"
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m 'core_model and npu and A3 and omni' --run-level 'core_model'
|
||||
timeout_in_minutes: 60
|
||||
|
||||
- group: ":card_index_dividers: TTS Model Test"
|
||||
key: ready-tts-group
|
||||
depends_on: upload-ready-pipeline
|
||||
steps:
|
||||
- label: ":sound: TTS · Qwen3-TTS CustomVoice Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
@@ -102,55 +95,3 @@ steps:
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py -m 'core_model and npu' --run-level 'core_model'
|
||||
|
||||
- label: "Diffusion · HunyuanVideo-1.5 Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m 'core_model and diffusion and npu' --run-level 'core_model'
|
||||
"
|
||||
timeout_in_minutes: 60
|
||||
|
||||
- label: "Diffusion · Qwen Image Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model and npu' --run-level 'core_model'
|
||||
"
|
||||
timeout_in_minutes: 60
|
||||
|
||||
- label: "Diffusion · Wan22 Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_wan22_t2v.py -m 'core_model and npu' --run-level 'core_model'
|
||||
"
|
||||
timeout_in_minutes: 60
|
||||
|
||||
- label: "Diffusion · Hunyuan Image Test"
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_hunyuan_image.py -m 'core_model and diffusion and npu' --run-level 'core_model'
|
||||
"
|
||||
timeout_in_minutes: 60
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/3d-animation-short-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/brand-promo-video-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/co-op-game-intro-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/h3-prompt-writing
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/handdrawn-live-video-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/minimalist-product-ad-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/music-video-subtitle-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/paper-collage-explainer-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/papercraft-stop-motion-explainer
|
||||
@@ -340,7 +340,7 @@ On supported GPU hosts, wrap a test or soak with the repository monitor:
|
||||
|
||||
```bash
|
||||
bash tests/dfx/stability/scripts/resource_monitor.sh run --backend gpu -- \
|
||||
pytest -s -v tests/dfx/stability/scripts/run_stability_<model>.py -m slow
|
||||
pytest -s -v tests/dfx/stability/scripts/test_stability_<model>.py -m slow
|
||||
```
|
||||
|
||||
Check the target revision's monitor backends. Do not claim CPU/NPU monitoring
|
||||
|
||||
@@ -60,14 +60,6 @@ self-check.
|
||||
`test-weekly.yml`), and copy-paste local plus CI-like `pytest` commands; see
|
||||
`references/test-routing.md` for level-to-command mapping
|
||||
|
||||
## MiniMax H3 Creative Skills
|
||||
|
||||
The [portable H3 collection](../../.agents/skills/README.md) contains the prompt
|
||||
writing skill and eight creative workflows for product ads, 3D shorts, paper art,
|
||||
brand promos, music videos, co-op intros, and hand-drawn/live-action fusion.
|
||||
Same-named directories here link to the canonical `.agents/skills/` copies, so
|
||||
Claude Code and Codex use the same instructions and templates.
|
||||
|
||||
## Maintenance Guidelines
|
||||
|
||||
- Keep skill names short and task-oriented.
|
||||
|
||||
@@ -67,11 +67,6 @@
|
||||
# File-level overrides within entrypoints/
|
||||
/vllm_omni/entrypoints/openai/errors.py @alex-jw-brooks @NickCao
|
||||
/vllm_omni/entrypoints/cli/benchmark/ @alex-jw-brooks @Bounty-hunter
|
||||
# Shared wire codec: protocol/realtime (OpenAI Realtime GA) and protocol/duplex
|
||||
# (vLLM-Omni's extension of it) — RFC #6592 P0a. Layer-neutral by design — the
|
||||
# engine depends on it — so it is owned by the serving-boundary document rather
|
||||
# than by any one consumer. — docs/design/module/entrypoints.md
|
||||
/vllm_omni/protocol/ @alex-jw-brooks @linyueqian @NickCao
|
||||
|
||||
# Error contracts — docs/design/module/error_contracts.md
|
||||
/vllm_omni/errors.py @alex-jw-brooks @NickCao
|
||||
|
||||
@@ -12,7 +12,4 @@ PLEASE FILL IN THE PR DESCRIPTION HERE.
|
||||
## Test Result
|
||||
|
||||
**BEFORE SUBMITTING:** read [CONTRIBUTING.md](https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md) and run the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) with the code agent for a self-check against project conventions.
|
||||
|
||||
> [!NOTE]
|
||||
> To avoid agent spam, vLLM-Omni limits contributors without write access to **5 open PRs**. If this blocks well-intentioned critical work, please contact a committer to request bypass list access.
|
||||
(anything written below this line will be removed by GitHub Actions)
|
||||
@@ -56,35 +56,6 @@ jobs:
|
||||
run: |
|
||||
bash scripts/build_wheel.sh --python python
|
||||
|
||||
- name: Validate runtime package data from source archive
|
||||
run: |
|
||||
set -euo pipefail
|
||||
archive_dir="$(mktemp -d)"
|
||||
install_dir="$(mktemp -d)"
|
||||
vcs_wheels=(dist/*.whl)
|
||||
if [[ ${#vcs_wheels[@]} -ne 1 || ! -f "${vcs_wheels[0]}" ]]; then
|
||||
echo "Expected exactly one VCS-built wheel" >&2
|
||||
exit 1
|
||||
fi
|
||||
git archive HEAD | tar -x -C "${archive_dir}"
|
||||
(
|
||||
cd "${archive_dir}"
|
||||
VLLM_OMNI_TARGET_DEVICE=cuda \
|
||||
VLLM_OMNI_VERSION_OVERRIDE=0.0.0 \
|
||||
python -m build --wheel --outdir archive-dist
|
||||
)
|
||||
wheels=("${archive_dir}"/archive-dist/*.whl)
|
||||
if [[ ${#wheels[@]} -ne 1 || ! -f "${wheels[0]}" ]]; then
|
||||
echo "Expected exactly one archive-built wheel" >&2
|
||||
exit 1
|
||||
fi
|
||||
python -m pip install --no-deps --target "${install_dir}" "${wheels[0]}"
|
||||
python tools/check_wheel_package_data.py \
|
||||
--source-root "${archive_dir}" \
|
||||
--wheel "${wheels[0]}" \
|
||||
--vcs-wheel "${vcs_wheels[0]}" \
|
||||
--installed-root "${install_dir}"
|
||||
|
||||
- name: Upload Wheel Artifact
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
|
||||
@@ -243,15 +243,9 @@ datasets/
|
||||
*.csv
|
||||
*.json
|
||||
!recipes/cosmos3/negative_prompt.json
|
||||
!benchmarks/mammoth_moda2/reference_results.json
|
||||
!tests/dfx/perf/tests/*.json
|
||||
!apps/ComfyUI-vLLM-Omni/example_workflows/*.json
|
||||
# Omni-DuplexEval CI config JSON (e.g. accuracy/omni_duplex_eval)
|
||||
!tests/e2e/accuracy/omni_duplex_eval/*.json
|
||||
*.jsonl
|
||||
# Shared Hunyuan IT2I input for accuracy tests and vllm bench.
|
||||
!tests/assets/hunyuan_image3/it2i.jsonl
|
||||
!tests/assets/livekit/client_vad_replay.jsonl
|
||||
*.parquet
|
||||
|
||||
# Output files
|
||||
|
||||
@@ -11,5 +11,3 @@ how to extend allowlists) are documented in the
|
||||
section of that guide.
|
||||
|
||||
Before submitting a PR, run the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) with the code agent for a self-check against project conventions.
|
||||
|
||||
vLLM-Omni limits contributors without write access to **5 open PRs**; see the [Pull Request Limit](docs/contributing/README.md#pull-request-limit) section of the contributing guide.
|
||||
@@ -16,7 +16,6 @@ Easy, fast, and cheap omni-modality model serving for everyone
|
||||
|
||||
*Latest News* 🔥
|
||||
|
||||
- [2026/09] We released [0.30.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.30.0), rebased onto vLLM 0.30.0, featuring a unified full-duplex serving framework around engine-owned sessions for [MiniCPM-o 4.5](recipes/OpenBMB/MiniCPM-o-4_5.md) and AURA, native cross-stage KV and multimodal payload transfer (Mooncake AR-to-DiT handoff and NIXL connectors), interactive world-model serving with [LingBot World](recipes/Robbyant/LingBot-World-2.0.md), and realtime [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) Turbo inference on NVIDIA Blackwell and Ascend 950.
|
||||
- [2026/08] We released [0.28.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.28.0), featuring production-ready [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) serving on GPU and NPU, a unified AR/DiT paged KV cache runtime, and enhanced realtime full-duplex serving for the [MiniCPM-o series](recipes/OpenBMB/MiniCPM-o-4_5.md).
|
||||
- [2026/08] [VeRL-Omni](https://github.com/verl-project/verl-omni) `v0.2.0` is released: faster diffusion RL powered by vLLM-Omni (request-level/step-wise batching with FA3), rebuilt Qwen3-Omni multimodal training (DPO & GSPO), plus LTX-2.3, Qwen-Image-Edit support and more. See the [release notes](https://github.com/verl-project/verl-omni/releases/tag/v0.2.0).
|
||||
- [2026/08] We released [0.26.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.26.0) - aligned with the vLLM 0.26 release line, featuring [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) joint video/audio generation, an experimental full-duplex realtime runtime for [MiniCPM-o 4.5](recipes/OpenBMB/MiniCPM-o-4_5.md), distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
|
||||
@@ -59,9 +58,9 @@ vLLM-Omni is flexible and easy to use with:
|
||||
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
|
||||
|
||||
- **Omni-modality models** (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
|
||||
- **TTS models** (e.g. Qwen3-TTS, Tencent AuK, Breeze-TTS-2, CosyVoice3)
|
||||
- **Diffusion models** — image, video, and audio generation (e.g. MiniMax H3, LingBot World, MAGI-2, LTX-2.5, Wan2.2)
|
||||
- **Robot-policy and action models** (e.g. π0.5, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
|
||||
- **TTS models** (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
|
||||
- **Diffusion models** — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
|
||||
- **Robot-policy and action models** (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
|
||||
|
||||
## Getting Started
|
||||
|
||||
|
||||
@@ -49,7 +49,6 @@ This extension offers the following nodes based on the output modalities (at **C
|
||||
|
||||
- **Generate Image** for text-to-image and image-to-image tasks
|
||||
- **Generate Video** for text-to-video, first-frame/image-to-video, and reference-conditioned video
|
||||
- **Latent Mask Editing** for MiniMax-H3 latent-mask editing (source media plus video/audio noise masks)
|
||||
- **FastH3 Deployment** for routing text-to-video requests to a MiniMax-H3 server with FastH3 fused at startup
|
||||
- **Multimodality Understanding** for multimodality-to-text and multimodality-to-audio tasks
|
||||
- **TTS** and **TTS Voice Clone** for TTS tasks
|
||||
@@ -68,7 +67,7 @@ Every node carries the vLLM-Omni mark in its title bar and is tinted by what it
|
||||
| Amber | AR / Diffusion / Multi-Stage Sampling Params | Sampling parameters that apply to any model |
|
||||
| Purple | Qwen TTS Params, Wan Video Params, MiniMax-H3 Video Params | Parameters that only one model family accepts |
|
||||
| Red | LoRA, FastH3 Deployment | Which weights the server is expected to have loaded |
|
||||
| Teal | Video References, Latent Mask Editing | Reference media |
|
||||
| Teal | Video References | Reference media |
|
||||
|
||||
Recolouring a node by hand (right click -> Colors) overrides its tint, and the choice is kept.
|
||||
|
||||
@@ -135,13 +134,11 @@ You can configure per-stage sampling parameters for multi-stage models.
|
||||
</p>
|
||||
|
||||
> [!TIP]
|
||||
> Connect **frame** for backward-compatible first-frame / image-to-video behavior. For MiniMax-H3 FL2VA,
|
||||
> connect **first_frame**, **last_frame**, or both to condition the start frame, end frame, or both.
|
||||
> Connect a **frame** image for first-frame / image-to-video (e.g., Wan I2V, MiniMax-H3 FL2VA).
|
||||
>
|
||||
> For reference-conditioned generation (MiniMax-H3 Ref2VA), connect a **Video References** node instead.
|
||||
>
|
||||
> Do not combine `frame` with `first_frame` or `last_frame`, and do not combine any frame input with
|
||||
> `references`. Task routing is automatic from which inputs you connect.
|
||||
> Do not use `frame` and `references` together. Task routing is automatic from which inputs you connect.
|
||||
|
||||
For MiniMax-H3 Ref2VA, **Video References** accepts up to 9 images (`image_1`–`image_9`),
|
||||
3 videos (`video_1`–`video_3`), and 3 audio clips (`audio_1`–`audio_3`), with at most
|
||||
@@ -264,18 +261,6 @@ Open the **vLLM-Omni FastH3 Text to Video** template, then:
|
||||
|
||||
The node records which server the workflow targets; it does not start one, nor switch adapters or attention backends on a running server.
|
||||
|
||||
#### Latent-mask editing (MiniMax-H3)
|
||||
|
||||
The [WF-05 template](example_workflows/vLLM-Omni%20MiniMax-H3%20Latent%20Mask%20Editing.json) contains inpainting, object removal, continuation, and extension examples in one graph. See the [workflow guide](docs/wf05-h3-latent-editing.md) for inputs, mask settings, dependencies, and preview limitations.
|
||||
|
||||
Connect a **Latent Mask Editing** node to **Generate Video → latent_edit** to edit a source clip instead of generating from scratch. It uploads the source media and serializes the video/audio noise masks the MiniMax H3 API accepts:
|
||||
|
||||
- `source_video` / `source_audio` — the media to edit.
|
||||
- `video_mask` — a ComfyUI mask image; `0` preserves a region, `1` regenerates it, fractional values blend. A 2D mask `[H, W]` is applied to every frame; a 3D mask `[T, H, W]` is treated as a temporal mask (one slice per frame) for continuation or extension. It is resized to the video latent grid.
|
||||
- `audio_mask` — a scalar in `[0, 1]`; `0` keeps the source audio, `1` regenerates it, fractional values blend.
|
||||
|
||||
A non-trivial mask requires its matching source, and a source without a mask is rejected. This node only forwards inputs to the server; the served model must declare latent-mask editing support (MiniMax-H3).
|
||||
|
||||
### TTS (e.g., Qwen TTS series)
|
||||
|
||||
(Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni TTS**)
|
||||
|
||||
@@ -21,9 +21,7 @@ from .comfyui_vllm_omni.nodes import (
|
||||
VLLMOmniGenerateImage,
|
||||
VLLMOmniGenerateMusic,
|
||||
VLLMOmniGenerateVideo,
|
||||
VLLMOmniLatentMaskEditing,
|
||||
VLLMOmniMiniMaxH3Params,
|
||||
VLLMOmniMiniMaxH3TemporalMask,
|
||||
VLLMOmniQwenTTSParams,
|
||||
VLLMOmniRemoteLoRA,
|
||||
VLLMOmniSamplingParamsList,
|
||||
@@ -36,7 +34,6 @@ from .comfyui_vllm_omni.nodes import (
|
||||
|
||||
# A dictionary that contains all nodes you want to export with their names
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"VLLMOmniMiniMaxH3TemporalMask": VLLMOmniMiniMaxH3TemporalMask,
|
||||
# === Generation ===
|
||||
"VLLMOmniGenerateImage": VLLMOmniGenerateImage,
|
||||
"VLLMOmniGenerateVideo": VLLMOmniGenerateVideo,
|
||||
@@ -45,7 +42,6 @@ NODE_CLASS_MAPPINGS = {
|
||||
"VLLMOmniGenerateMusic": VLLMOmniGenerateMusic,
|
||||
"VLLMOmniVoiceClone": VLLMOmniVoiceClone,
|
||||
"VLLMOmniVideoReferences": VLLMOmniVideoReferences,
|
||||
"VLLMOmniLatentMaskEditing": VLLMOmniLatentMaskEditing,
|
||||
# === Params ===
|
||||
"VLLMOmniARSampling": VLLMOmniARSampling,
|
||||
"VLLMOmniDiffusionSampling": VLLMOmniDiffusionSampling,
|
||||
@@ -59,7 +55,6 @@ NODE_CLASS_MAPPINGS = {
|
||||
|
||||
# A dictionary that contains the friendly/humanly readable titles for the nodes
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"VLLMOmniMiniMaxH3TemporalMask": "MiniMax-H3 Temporal Mask",
|
||||
# === Generation ===
|
||||
"VLLMOmniGenerateImage": "Generate Image",
|
||||
"VLLMOmniGenerateVideo": "Generate Video",
|
||||
@@ -68,7 +63,6 @@ NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"VLLMOmniGenerateMusic": "Generate Music",
|
||||
"VLLMOmniVoiceClone": "TTS Voice Cloning",
|
||||
"VLLMOmniVideoReferences": "Video References",
|
||||
"VLLMOmniLatentMaskEditing": "Latent Mask Editing",
|
||||
# === Params ===
|
||||
"VLLMOmniARSampling": "AR Sampling Params",
|
||||
"VLLMOmniDiffusionSampling": "Diffusion Sampling Params",
|
||||
|
||||
@@ -1,14 +1,12 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import math
|
||||
from typing import Literal
|
||||
|
||||
import torch
|
||||
from comfy_api.input import AudioInput, VideoInput
|
||||
|
||||
from .utils.api_client import VLLMOmniClient
|
||||
from .utils.latent_mask import _align_frame_count, _video_latent_t
|
||||
from .utils.logger import get_logger
|
||||
from .utils.models import lookup_model_spec
|
||||
from .utils.types import (
|
||||
@@ -19,7 +17,6 @@ from .utils.types import (
|
||||
AutoregressionSamplingParams,
|
||||
DiffusionSamplingParams,
|
||||
FastH3Deployment,
|
||||
LatentMaskEditing,
|
||||
MiniMaxH3ModelSpecificParams,
|
||||
QwenTTSModelSpecificParams,
|
||||
VideoReferences,
|
||||
@@ -206,14 +203,11 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
},
|
||||
"optional": {
|
||||
"frame": ("IMAGE",),
|
||||
"first_frame": ("IMAGE",),
|
||||
"last_frame": ("IMAGE",),
|
||||
"references": ("VIDEO_REFERENCES",),
|
||||
"sampling_params": ("SAMPLING_PARAMS",),
|
||||
"lora": ("REMOTE_LORA",),
|
||||
"model_params": ("VIDEO_PARAMS",),
|
||||
"fast_h3": ("FASTH3_DEPLOYMENT",),
|
||||
"latent_edit": ("LATENT_MASK_EDITING",),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -222,30 +216,12 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
FUNCTION = "generate"
|
||||
|
||||
@classmethod
|
||||
def VALIDATE_INPUTS(
|
||||
cls,
|
||||
url,
|
||||
model,
|
||||
frame=None,
|
||||
first_frame=None,
|
||||
last_frame=None,
|
||||
references=None,
|
||||
fast_h3=None,
|
||||
**_kwargs,
|
||||
) -> str | Literal[True]:
|
||||
def VALIDATE_INPUTS(cls, url, model, frame=None, references=None, **_kwargs) -> str | Literal[True]:
|
||||
base = super().VALIDATE_INPUTS(url, model)
|
||||
if base is not True:
|
||||
return base
|
||||
if frame is not None and references is not None:
|
||||
return "Provide only one of frame or references, not both."
|
||||
if frame is not None and (first_frame is not None or last_frame is not None):
|
||||
return "Provide either frame or first_frame/last_frame, not both."
|
||||
if references is not None and (first_frame is not None or last_frame is not None):
|
||||
return "Provide either first_frame/last_frame or references, not both."
|
||||
if fast_h3 is not None and any(value is not None for value in (frame, first_frame, last_frame, references)):
|
||||
return (
|
||||
"FastH3 Preview supports T2VA only; disconnect frame, first_frame, last_frame, and references inputs."
|
||||
)
|
||||
return True
|
||||
|
||||
async def generate(
|
||||
@@ -259,14 +235,11 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
duration: float,
|
||||
negative_prompt: str | None = None,
|
||||
frame: torch.Tensor | None = None,
|
||||
first_frame: torch.Tensor | None = None,
|
||||
last_frame: torch.Tensor | None = None,
|
||||
references: dict | None = None,
|
||||
sampling_params: dict | list[dict] | None = None,
|
||||
model_params: dict | None = None,
|
||||
lora: dict | None = None,
|
||||
fast_h3: dict | None = None,
|
||||
latent_edit: dict | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
if kwargs:
|
||||
@@ -279,11 +252,8 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
spec_model = model
|
||||
|
||||
if fast_h3 is not None:
|
||||
if any(value is not None for value in (frame, first_frame, last_frame, references)):
|
||||
raise ValueError(
|
||||
"FastH3 Preview supports T2VA only; disconnect frame, first_frame, "
|
||||
"last_frame, and references inputs."
|
||||
)
|
||||
if frame is not None or references is not None:
|
||||
raise ValueError("FastH3 Preview supports T2VA only; disconnect frame and references inputs.")
|
||||
if lora is not None:
|
||||
raise ValueError(
|
||||
"FastH3 is already fused into the selected server; disconnect the request-level LoRA input."
|
||||
@@ -344,8 +314,6 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
spec_model=spec_model,
|
||||
prompt=prompt,
|
||||
frame=frame, # frame present => fl2va / Wan I2V
|
||||
first_frame=first_frame,
|
||||
last_frame=last_frame,
|
||||
references=references,
|
||||
width=width,
|
||||
height=height,
|
||||
@@ -355,7 +323,6 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
sampling_params=sampling_params,
|
||||
lora=lora,
|
||||
model_params=model_params,
|
||||
latent_edit=latent_edit,
|
||||
)
|
||||
return (output,)
|
||||
|
||||
@@ -1057,89 +1024,3 @@ class VLLMOmniVideoReferences:
|
||||
if value is not None:
|
||||
refs[f"{kind}_{index}"] = value
|
||||
return (refs,)
|
||||
|
||||
|
||||
class VLLMOmniLatentMaskEditing:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {},
|
||||
"optional": {
|
||||
"source_video": ("VIDEO",),
|
||||
"source_audio": ("AUDIO",),
|
||||
"video_mask": ("MASK",),
|
||||
"audio_mask": ("FLOAT", {"default": -1.0, "min": -1.0, "max": 1.0, "step": 0.01}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("LATENT_MASK_EDITING",)
|
||||
RETURN_NAMES = ("latent_edit",)
|
||||
FUNCTION = "get_latent_edit"
|
||||
CATEGORY = "vLLM-Omni"
|
||||
|
||||
def get_latent_edit(
|
||||
self,
|
||||
source_video: VideoInput | None = None,
|
||||
source_audio: AudioInput | None = None,
|
||||
video_mask: torch.Tensor | None = None,
|
||||
audio_mask: float = -1.0,
|
||||
**kwargs,
|
||||
):
|
||||
if kwargs:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
edit = LatentMaskEditing()
|
||||
if source_video is not None:
|
||||
edit["source_video"] = source_video
|
||||
if source_audio is not None:
|
||||
edit["source_audio"] = source_audio
|
||||
if video_mask is not None:
|
||||
edit["video_mask"] = video_mask
|
||||
if audio_mask >= 0.0:
|
||||
edit["audio_mask"] = audio_mask
|
||||
return (edit,)
|
||||
|
||||
|
||||
class VLLMOmniMiniMaxH3TemporalMask:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"images": ("IMAGE",),
|
||||
"source_fps": ("FLOAT", {"default": 24.0, "min": 0.01}),
|
||||
"duration": ("FLOAT", {"default": 5.0, "min": 0.01, "max": 15.0}),
|
||||
"mode": (["continuation", "extension"],),
|
||||
"preserve_fraction": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MASK", "FLOAT", "IMAGE", "MASK")
|
||||
RETURN_NAMES = ("mask", "preview_fps", "preview_images", "preview_mask")
|
||||
FUNCTION = "build"
|
||||
CATEGORY = "vLLM-Omni"
|
||||
|
||||
def build(self, images, source_fps, duration, mode, preserve_fraction=0.5):
|
||||
if images.shape[0] <= 0 or not math.isfinite(source_fps) or source_fps <= 0:
|
||||
raise ValueError("Source must contain frames and have a positive finite FPS.")
|
||||
if not math.isfinite(duration) or duration <= 0 or not 0 <= preserve_fraction <= 1:
|
||||
raise ValueError("Invalid duration or preserve_fraction.")
|
||||
frames = _align_frame_count(max(1, round(duration * 24)))
|
||||
source_seconds = images.shape[0] / source_fps
|
||||
if mode == "extension":
|
||||
if frames / 24 <= source_seconds:
|
||||
raise ValueError("Extension output must be longer than the source; increase duration.")
|
||||
boundary = source_seconds
|
||||
elif mode == "continuation":
|
||||
boundary = min(source_seconds, frames / 24) * preserve_fraction
|
||||
else:
|
||||
raise ValueError(f"Unknown temporal mask mode: {mode}")
|
||||
available = min(frames, math.floor(boundary * 24 + 1e-8))
|
||||
prefix = 0 if available < 5 else 5 + 17 * ((available - 5) // 17)
|
||||
preserved = _video_latent_t(prefix) if prefix else 0
|
||||
total = _video_latent_t(frames)
|
||||
mask = torch.ones(total, 1, 1)
|
||||
mask[:preserved] = 0
|
||||
indices = torch.arange(frames, device=images.device)
|
||||
source_indices = (indices * (source_fps / 24)).floor().long().clamp(max=images.shape[0] - 1)
|
||||
preview_images = images.index_select(0, source_indices)
|
||||
preview_mask = mask.index_select(0, torch.arange(frames) * total // frames)
|
||||
return mask, 24.0, preview_images, preview_mask
|
||||
@@ -30,7 +30,6 @@ from .format import (
|
||||
video_to_base64,
|
||||
video_to_bytes,
|
||||
)
|
||||
from .latent_mask import scalar_mask_to_json, video_mask_to_grid_json
|
||||
from .logger import get_logger, pretty_printer
|
||||
from .models import lookup_model_spec
|
||||
from .types import (
|
||||
@@ -278,13 +277,10 @@ class VLLMOmniClient:
|
||||
fps: int,
|
||||
negative_prompt: str | None = None,
|
||||
frame: torch.Tensor | None = None,
|
||||
first_frame: torch.Tensor | None = None,
|
||||
last_frame: torch.Tensor | None = None,
|
||||
references: dict | None = None,
|
||||
sampling_params: dict | None = None,
|
||||
model_params: dict | None = None,
|
||||
lora: dict | None = None,
|
||||
latent_edit: dict | None = None,
|
||||
spec_model: str | None = None,
|
||||
**extra_params,
|
||||
) -> VideoInput:
|
||||
@@ -296,16 +292,6 @@ class VLLMOmniClient:
|
||||
"""
|
||||
if frame is not None and references is not None:
|
||||
raise ValueError("Provide only one of frame or references, not both.")
|
||||
if frame is not None and (first_frame is not None or last_frame is not None):
|
||||
raise ValueError("Provide either frame or first_frame/last_frame, not both.")
|
||||
if references is not None and (first_frame is not None or last_frame is not None):
|
||||
raise ValueError("Provide either first_frame/last_frame or references, not both.")
|
||||
|
||||
spec, matched_pattern = lookup_model_spec(spec_model or model)
|
||||
if (first_frame is not None or last_frame is not None) and (
|
||||
matched_pattern is None or "MiniMax-H3" not in matched_pattern
|
||||
):
|
||||
raise ValueError("first_frame and last_frame are supported only for MiniMax-H3; use frame for this model.")
|
||||
|
||||
# === regular payload fields ===
|
||||
form = aiohttp.FormData()
|
||||
@@ -325,24 +311,11 @@ class VLLMOmniClient:
|
||||
|
||||
# === multimodal inputs (first-last-frames, references, etc.) ===
|
||||
input_reference_image: torch.Tensor | None = None
|
||||
keyframe_images: list[tuple[str, torch.Tensor]] = []
|
||||
video_task: str | None = None
|
||||
|
||||
if frame is not None:
|
||||
input_reference_image = frame
|
||||
video_task = "fl2va"
|
||||
elif first_frame is not None or last_frame is not None:
|
||||
frame_indices: list[int] = []
|
||||
if first_frame is not None:
|
||||
keyframe_images.append(("first_frame.png", first_frame))
|
||||
frame_indices.append(0)
|
||||
if last_frame is not None:
|
||||
keyframe_images.append(("last_frame.png", last_frame))
|
||||
frame_indices.append(-1)
|
||||
if len(keyframe_images) == 1:
|
||||
input_reference_image = keyframe_images[0][1]
|
||||
extra_params["frame_indices"] = frame_indices
|
||||
video_task = "fl2va"
|
||||
elif references is not None:
|
||||
reference_formats = (
|
||||
("image", MAX_REFERENCE_IMAGES, "png", "image/png", image_tensor_to_png_bytes),
|
||||
@@ -380,7 +353,7 @@ class VLLMOmniClient:
|
||||
video_task = "t2va"
|
||||
|
||||
if input_reference_image is not None:
|
||||
image_filename = keyframe_images[0][0] if keyframe_images else "image.png"
|
||||
image_filename = "image.png" # Required for multipart form
|
||||
form.add_field(
|
||||
"input_reference",
|
||||
image_tensor_to_png_bytes(input_reference_image, image_filename),
|
||||
@@ -388,67 +361,12 @@ class VLLMOmniClient:
|
||||
content_type="image/png",
|
||||
)
|
||||
|
||||
# === latent-mask editing (MiniMax H3) ===
|
||||
if latent_edit is not None:
|
||||
source_video = latent_edit.get("source_video")
|
||||
source_audio = latent_edit.get("source_audio")
|
||||
video_mask = latent_edit.get("video_mask")
|
||||
audio_mask = latent_edit.get("audio_mask")
|
||||
|
||||
if video_mask is None and audio_mask is None:
|
||||
raise ValueError("Latent-mask editing requires at least one mask.")
|
||||
|
||||
video_mask_trivial = video_mask is None or bool((video_mask == 1.0).all().item())
|
||||
audio_mask_trivial = audio_mask is None or audio_mask == 1.0
|
||||
if not video_mask_trivial and source_video is None:
|
||||
raise ValueError("A non-trivial video mask requires a source video.")
|
||||
if not audio_mask_trivial and source_audio is None and source_video is None:
|
||||
raise ValueError("A non-trivial audio mask requires a source audio or a source video with audio.")
|
||||
|
||||
if source_video is not None:
|
||||
form.add_field(
|
||||
"source_video",
|
||||
video_to_bytes(source_video, "source.mp4"),
|
||||
filename="source.mp4",
|
||||
content_type="video/mp4",
|
||||
)
|
||||
if source_audio is not None:
|
||||
form.add_field(
|
||||
"source_audio",
|
||||
audio_to_bytes(source_audio, "source_audio.mp3"),
|
||||
filename="source_audio.mp3",
|
||||
content_type="audio/mpeg",
|
||||
)
|
||||
if video_mask is not None:
|
||||
mask_json = video_mask_to_grid_json(video_mask, width=width, height=height, num_frames=num_frames)
|
||||
form.add_field(
|
||||
"video_noise_mask",
|
||||
mask_json.encode("utf-8"),
|
||||
filename="video-mask.json",
|
||||
content_type="application/json",
|
||||
)
|
||||
if audio_mask is not None:
|
||||
form.add_field(
|
||||
"audio_noise_mask",
|
||||
scalar_mask_to_json(audio_mask).encode("utf-8"),
|
||||
filename="audio-mask.json",
|
||||
content_type="application/json",
|
||||
)
|
||||
|
||||
if len(keyframe_images) == 2:
|
||||
for image_filename, image in keyframe_images:
|
||||
form.add_field(
|
||||
"input_references",
|
||||
image_tensor_to_png_bytes(image, image_filename),
|
||||
filename=image_filename,
|
||||
content_type="image/png",
|
||||
)
|
||||
|
||||
# === model specific params. Either use a specialized builder, or add flattened fields as-is ===
|
||||
if model_params is not None:
|
||||
model_params = dict(model_params)
|
||||
model_params.pop("type", None)
|
||||
|
||||
spec, _ = lookup_model_spec(spec_model or model)
|
||||
params_builder = spec.get("params_builder") if spec else None
|
||||
if params_builder is not None:
|
||||
form_fields = params_builder(
|
||||
|
||||
@@ -1,51 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import json
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
def scalar_mask_to_json(value: float) -> str:
|
||||
if not 0.0 <= value <= 1.0:
|
||||
raise ValueError(f"mask value must be in [0, 1], got {value}")
|
||||
return str(value)
|
||||
|
||||
|
||||
def _align_frame_count(frame_count: int) -> int:
|
||||
if frame_count <= 0:
|
||||
return 1
|
||||
current = int(frame_count)
|
||||
while current % 17 != 5:
|
||||
current += 1
|
||||
return current
|
||||
|
||||
|
||||
def _video_latent_t(frame_count: int) -> int:
|
||||
if frame_count <= 5:
|
||||
return 2
|
||||
return ((int(frame_count) - 5) // 17) * 5 + 2
|
||||
|
||||
|
||||
def video_mask_to_grid(mask: torch.Tensor, *, width: int, height: int, num_frames: int) -> torch.Tensor:
|
||||
tv = _video_latent_t(_align_frame_count(num_frames))
|
||||
height = int(height) // 32 * 32
|
||||
width = int(width) // 32 * 32
|
||||
gh, gw = height // 16, width // 16
|
||||
|
||||
if mask.ndim == 2:
|
||||
mask = mask.unsqueeze(0)
|
||||
elif mask.ndim != 3:
|
||||
raise ValueError(f"expected a 2D or 3D mask tensor, got {mask.ndim}D")
|
||||
|
||||
grid = F.interpolate(mask.unsqueeze(1).float(), size=(gh, gw), mode="area").squeeze(1)
|
||||
if grid.shape[0] == 1:
|
||||
grid = grid.expand(tv, gh, gw)
|
||||
elif grid.shape[0] != tv:
|
||||
grid = F.interpolate(grid.unsqueeze(0).unsqueeze(0), size=(tv, gh, gw), mode="nearest").squeeze(0).squeeze(0)
|
||||
return grid
|
||||
|
||||
|
||||
def video_mask_to_grid_json(mask: torch.Tensor, *, width: int, height: int, num_frames: int) -> str:
|
||||
return json.dumps(video_mask_to_grid(mask, width=width, height=height, num_frames=num_frames).tolist())
|
||||
@@ -40,10 +40,6 @@ class VideoReferences(dict):
|
||||
pass
|
||||
|
||||
|
||||
class LatentMaskEditing(dict):
|
||||
pass
|
||||
|
||||
|
||||
class FastH3Deployment(dict):
|
||||
"""Descriptor for a server that fused FastH3 at startup."""
|
||||
|
||||
|
||||
@@ -1,17 +0,0 @@
|
||||
# WF-05: MiniMax-H3 latent-mask editing
|
||||
|
||||
Edit a source video with MiniMax-H3 using spatial or temporal masks. The workflow includes object removal, inpainting, continuation, and extension, with a mask preview for each case.
|
||||
|
||||
## Requirements
|
||||
|
||||
- ComfyUI with ComfyUI-vLLM-Omni installed.
|
||||
- An H3 service using the FL2VA checkpoint partition with latent-mask editing support. Follow the [H3 recipe](../../../recipes/MiniMaxAI/MiniMax-H3.md) for server setup.
|
||||
|
||||
## Use
|
||||
|
||||
Open **vLLM-Omni MiniMax-H3 Latent Mask Editing.json** from the example workflows and set the Generate Video URL/model for the case you want to run.
|
||||
|
||||
1. Upload your source video in Load Video.
|
||||
2. Adjust the case's mask. For removal and inpainting, set Mask Canvas to the video dimensions, Mask Rectangle to the region size, and Combine Masks x/y to its position. For continuation and extension, set MiniMax-H3 Temporal Mask's mode and duration; continuation also uses `preserve_fraction`. Match the duration in Generate Video, and use a duration longer than the source for extension. Mask values are `0` to preserve and `1` to regenerate.
|
||||
3. Adjust the prompt to describe the desired content. Set the output dimensions and duration in Generate Video. If editing audio, update the sound description and audio mask (`0` preserves, `1` regenerates the whole clip).
|
||||
4. Select the case's Mask Preview Save Video node and choose **Execute to selected output nodes** to preview the mask as a red overlay without inference, or select its Result Save Video node to generate the edited video. Running the entire workflow generates all four cases.
|
||||
-499
@@ -1,499 +0,0 @@
|
||||
{
|
||||
"id": "latent-mask-editing-7380",
|
||||
"revision": 0,
|
||||
"last_node_id": 7,
|
||||
"last_link_id": 5,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
40,
|
||||
20
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
210
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"title": "Latent-mask editing server prerequisite",
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"Requires a vLLM-Omni server that declares latent-mask editing support (MiniMax-H3, EDIT-01). The source clip and video/audio noise masks are uploaded to /v1/videos; 0 preserves, 1 regenerates, fractional blends."
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#000"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "LoadVideo",
|
||||
"pos": [
|
||||
40,
|
||||
280
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
120
|
||||
],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "file",
|
||||
"name": "file",
|
||||
"type": "COMBO",
|
||||
"widget": {
|
||||
"name": "file"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
1
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "LoadImageMask",
|
||||
"pos": [
|
||||
40,
|
||||
460
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "image",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "image"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "channel",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "channel"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "mask",
|
||||
"name": "mask",
|
||||
"type": "MASK",
|
||||
"links": [
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImageMask"
|
||||
},
|
||||
"widgets_values": [
|
||||
"",
|
||||
"red"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "VLLMOmniLatentMaskEditing",
|
||||
"pos": [
|
||||
560,
|
||||
260
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
300
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "source_video",
|
||||
"name": "source_video",
|
||||
"type": "VIDEO",
|
||||
"link": 1,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "source_audio",
|
||||
"name": "source_audio",
|
||||
"type": "AUDIO",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "video_mask",
|
||||
"name": "video_mask",
|
||||
"type": "MASK",
|
||||
"link": 2,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "audio_mask",
|
||||
"name": "audio_mask",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "audio_mask"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "latent_edit",
|
||||
"name": "latent_edit",
|
||||
"type": "LATENT_MASK_EDITING",
|
||||
"links": [
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniLatentMaskEditing"
|
||||
},
|
||||
"widgets_values": [
|
||||
0.5
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "VLLMOmniMiniMaxH3Params",
|
||||
"pos": [
|
||||
40,
|
||||
680
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
140
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "audio_flow_shift",
|
||||
"name": "audio_flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "audio_flow_shift"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "flow_shift",
|
||||
"name": "flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "flow_shift"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video params",
|
||||
"name": "video params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"links": [
|
||||
4
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniMiniMaxH3Params"
|
||||
},
|
||||
"widgets_values": [
|
||||
3.0,
|
||||
12.0
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "VLLMOmniGenerateVideo",
|
||||
"pos": [
|
||||
1080,
|
||||
200
|
||||
],
|
||||
"size": [
|
||||
540,
|
||||
680
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "frame",
|
||||
"name": "frame",
|
||||
"type": "IMAGE",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "references",
|
||||
"name": "references",
|
||||
"type": "VIDEO_REFERENCES",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "sampling_params",
|
||||
"name": "sampling_params",
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "lora",
|
||||
"name": "lora",
|
||||
"type": "REMOTE_LORA",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "model_params",
|
||||
"name": "model_params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"link": 4,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "fast_h3",
|
||||
"name": "fast_h3",
|
||||
"type": "FASTH3_DEPLOYMENT",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "latent_edit",
|
||||
"name": "latent_edit",
|
||||
"type": "LATENT_MASK_EDITING",
|
||||
"link": 3,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "url",
|
||||
"name": "url",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "url"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "model",
|
||||
"name": "model",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "model"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "prompt",
|
||||
"name": "prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "negative_prompt",
|
||||
"name": "negative_prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "negative_prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "width",
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "width"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "height",
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "height"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "fps",
|
||||
"name": "fps",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "fps"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "duration",
|
||||
"name": "duration",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "duration"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
5
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniGenerateVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"http://127.0.0.1:8000/v1",
|
||||
"MiniMaxAI/MiniMax-H3",
|
||||
"Partially restyle the clip and soundtrack",
|
||||
"",
|
||||
160,
|
||||
120,
|
||||
24,
|
||||
4.458
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "SaveVideo",
|
||||
"pos": [
|
||||
1700,
|
||||
300
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"link": 5,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "filename_prefix",
|
||||
"name": "filename_prefix",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "filename_prefix"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "format",
|
||||
"name": "format",
|
||||
"type": "COMFY_DYNAMICCOMBO_V3",
|
||||
"widget": {
|
||||
"name": "format"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": []
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SaveVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"vllm-omni/latent-edit",
|
||||
"mp4"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
2,
|
||||
0,
|
||||
4,
|
||||
0,
|
||||
"VIDEO"
|
||||
],
|
||||
[
|
||||
2,
|
||||
3,
|
||||
0,
|
||||
4,
|
||||
2,
|
||||
"MASK"
|
||||
],
|
||||
[
|
||||
3,
|
||||
4,
|
||||
0,
|
||||
6,
|
||||
6,
|
||||
"LATENT_MASK_EDITING"
|
||||
],
|
||||
[
|
||||
4,
|
||||
5,
|
||||
0,
|
||||
6,
|
||||
4,
|
||||
"VIDEO_PARAMS"
|
||||
],
|
||||
[
|
||||
5,
|
||||
6,
|
||||
0,
|
||||
7,
|
||||
0,
|
||||
"VIDEO"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {},
|
||||
"version": 0.4
|
||||
}
|
||||
-560
@@ -1,560 +0,0 @@
|
||||
{
|
||||
"last_node_id": 9,
|
||||
"last_link_id": 7,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 2,
|
||||
"type": "LoadVideo",
|
||||
"pos": [
|
||||
40,
|
||||
100
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
120
|
||||
],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "file",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "file"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
1
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "SolidMask",
|
||||
"pos": [
|
||||
40,
|
||||
300
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "value",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "value"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "width"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "height"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "mask",
|
||||
"type": "MASK",
|
||||
"links": [
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SolidMask"
|
||||
},
|
||||
"widgets_values": [
|
||||
0.0,
|
||||
160,
|
||||
120
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "SolidMask",
|
||||
"pos": [
|
||||
40,
|
||||
500
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "value",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "value"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "width"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "height"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "mask",
|
||||
"type": "MASK",
|
||||
"links": [
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SolidMask"
|
||||
},
|
||||
"widgets_values": [
|
||||
1.0,
|
||||
160,
|
||||
120
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "BatchMasksNode",
|
||||
"pos": [
|
||||
250,
|
||||
380
|
||||
],
|
||||
"size": [
|
||||
200,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "masks.mask0",
|
||||
"type": "MASK",
|
||||
"link": 2,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "masks.mask1",
|
||||
"type": "MASK",
|
||||
"link": 3,
|
||||
"slot_index": 1
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": [
|
||||
4
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "BatchMasksNode"
|
||||
},
|
||||
"widgets_values": []
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "VLLMOmniLatentMaskEditing",
|
||||
"pos": [
|
||||
450,
|
||||
180
|
||||
],
|
||||
"size": [
|
||||
400,
|
||||
120
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "source_video",
|
||||
"type": "VIDEO",
|
||||
"link": 1
|
||||
},
|
||||
{
|
||||
"name": "source_audio",
|
||||
"type": "AUDIO",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "video_mask",
|
||||
"type": "MASK",
|
||||
"link": 4
|
||||
},
|
||||
{
|
||||
"name": "audio_mask",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "audio_mask"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "latent_edit",
|
||||
"type": "LATENT_MASK_EDITING",
|
||||
"links": [
|
||||
5
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniLatentMaskEditing"
|
||||
},
|
||||
"widgets_values": [
|
||||
-1.0
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "VLLMOmniMiniMaxH3Params",
|
||||
"pos": [
|
||||
40,
|
||||
700
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "audio_flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "audio_flow_shift"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "flow_shift"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "video params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"links": [
|
||||
6
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniMiniMaxH3Params"
|
||||
},
|
||||
"widgets_values": [
|
||||
3.0,
|
||||
12.0
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "VLLMOmniGenerateVideo",
|
||||
"pos": [
|
||||
700,
|
||||
180
|
||||
],
|
||||
"size": [
|
||||
450,
|
||||
380
|
||||
],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "frame",
|
||||
"type": "IMAGE",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "references",
|
||||
"type": "VIDEO_REFERENCES",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "sampling_params",
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "lora",
|
||||
"type": "LORA",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "model_params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"link": 6
|
||||
},
|
||||
{
|
||||
"name": "fast_h3",
|
||||
"type": "FAST_H3",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "latent_edit",
|
||||
"type": "LATENT_MASK_EDITING",
|
||||
"link": 5
|
||||
},
|
||||
{
|
||||
"name": "url",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "url"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "model",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "model"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "negative_prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "negative_prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "width"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "height"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "fps",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "fps"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "duration",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "duration"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
7
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniGenerateVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"http://127.0.0.1:8000/v1",
|
||||
"MiniMaxAI/MiniMax-H3",
|
||||
"temporal mask: regenerate the ending",
|
||||
"",
|
||||
160,
|
||||
120,
|
||||
24,
|
||||
4.458
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "SaveVideo",
|
||||
"pos": [
|
||||
900,
|
||||
180
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
200
|
||||
],
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"link": 7
|
||||
},
|
||||
{
|
||||
"name": "filename_prefix",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "filename_prefix"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "format",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "format"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": []
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SaveVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"temporal_mask",
|
||||
"mp4"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
2,
|
||||
0,
|
||||
6,
|
||||
0,
|
||||
"VIDEO"
|
||||
],
|
||||
[
|
||||
2,
|
||||
3,
|
||||
0,
|
||||
5,
|
||||
0,
|
||||
"MASK"
|
||||
],
|
||||
[
|
||||
3,
|
||||
4,
|
||||
0,
|
||||
5,
|
||||
1,
|
||||
"MASK"
|
||||
],
|
||||
[
|
||||
4,
|
||||
5,
|
||||
0,
|
||||
6,
|
||||
2,
|
||||
"MASK"
|
||||
],
|
||||
[
|
||||
5,
|
||||
6,
|
||||
0,
|
||||
8,
|
||||
6,
|
||||
"LATENT_MASK_EDITING"
|
||||
],
|
||||
[
|
||||
6,
|
||||
7,
|
||||
0,
|
||||
8,
|
||||
4,
|
||||
"VIDEO_PARAMS"
|
||||
],
|
||||
[
|
||||
7,
|
||||
8,
|
||||
0,
|
||||
9,
|
||||
0,
|
||||
"VIDEO"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {},
|
||||
"version": 0.4
|
||||
}
|
||||
@@ -1,511 +0,0 @@
|
||||
{
|
||||
"id": "latent-mask-editing-7380",
|
||||
"revision": 0,
|
||||
"last_node_id": 7,
|
||||
"last_link_id": 5,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
40,
|
||||
20
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
210
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"title": "Latent-mask editing server prerequisite",
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"Requires a vLLM-Omni server that declares latent-mask editing support (MiniMax-H3, EDIT-01). The source clip and video/audio noise masks are uploaded to /v1/videos; 0 preserves, 1 regenerates, fractional blends."
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#000"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "LoadVideo",
|
||||
"pos": [
|
||||
40,
|
||||
280
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
120
|
||||
],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "file",
|
||||
"name": "file",
|
||||
"type": "COMBO",
|
||||
"widget": {
|
||||
"name": "file"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
1
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
""
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "SolidMask",
|
||||
"pos": [
|
||||
40,
|
||||
460
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "value",
|
||||
"name": "value",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "value"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "width",
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "width"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "height",
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "height"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "mask",
|
||||
"name": "mask",
|
||||
"type": "MASK",
|
||||
"links": [
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SolidMask"
|
||||
},
|
||||
"widgets_values": [
|
||||
1.0,
|
||||
160,
|
||||
120
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "VLLMOmniLatentMaskEditing",
|
||||
"pos": [
|
||||
560,
|
||||
260
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
300
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "source_video",
|
||||
"name": "source_video",
|
||||
"type": "VIDEO",
|
||||
"link": 1,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "source_audio",
|
||||
"name": "source_audio",
|
||||
"type": "AUDIO",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "video_mask",
|
||||
"name": "video_mask",
|
||||
"type": "MASK",
|
||||
"link": 2,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "audio_mask",
|
||||
"name": "audio_mask",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "audio_mask"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "latent_edit",
|
||||
"name": "latent_edit",
|
||||
"type": "LATENT_MASK_EDITING",
|
||||
"links": [
|
||||
3
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniLatentMaskEditing"
|
||||
},
|
||||
"widgets_values": [
|
||||
0.5
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "VLLMOmniMiniMaxH3Params",
|
||||
"pos": [
|
||||
40,
|
||||
680
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
140
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "audio_flow_shift",
|
||||
"name": "audio_flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "audio_flow_shift"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "flow_shift",
|
||||
"name": "flow_shift",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "flow_shift"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video params",
|
||||
"name": "video params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"links": [
|
||||
4
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniMiniMaxH3Params"
|
||||
},
|
||||
"widgets_values": [
|
||||
3.0,
|
||||
12.0
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "VLLMOmniGenerateVideo",
|
||||
"pos": [
|
||||
1080,
|
||||
200
|
||||
],
|
||||
"size": [
|
||||
540,
|
||||
680
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "frame",
|
||||
"name": "frame",
|
||||
"type": "IMAGE",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "references",
|
||||
"name": "references",
|
||||
"type": "VIDEO_REFERENCES",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "sampling_params",
|
||||
"name": "sampling_params",
|
||||
"type": "SAMPLING_PARAMS",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "lora",
|
||||
"name": "lora",
|
||||
"type": "REMOTE_LORA",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "model_params",
|
||||
"name": "model_params",
|
||||
"type": "VIDEO_PARAMS",
|
||||
"link": 4,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "fast_h3",
|
||||
"name": "fast_h3",
|
||||
"type": "FASTH3_DEPLOYMENT",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "latent_edit",
|
||||
"name": "latent_edit",
|
||||
"type": "LATENT_MASK_EDITING",
|
||||
"link": 3,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "url",
|
||||
"name": "url",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "url"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "model",
|
||||
"name": "model",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "model"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "prompt",
|
||||
"name": "prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "negative_prompt",
|
||||
"name": "negative_prompt",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "negative_prompt"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "width",
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "width"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "height",
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "height"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "fps",
|
||||
"name": "fps",
|
||||
"type": "INT",
|
||||
"widget": {
|
||||
"name": "fps"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "duration",
|
||||
"name": "duration",
|
||||
"type": "FLOAT",
|
||||
"widget": {
|
||||
"name": "duration"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": [
|
||||
5
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VLLMOmniGenerateVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"http://127.0.0.1:8000/v1",
|
||||
"MiniMaxAI/MiniMax-H3",
|
||||
"Partially restyle the clip and soundtrack",
|
||||
"",
|
||||
160,
|
||||
120,
|
||||
24,
|
||||
4.458
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "SaveVideo",
|
||||
"pos": [
|
||||
1700,
|
||||
300
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
160
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"link": 5,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"localized_name": "filename_prefix",
|
||||
"name": "filename_prefix",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "filename_prefix"
|
||||
},
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"localized_name": "format",
|
||||
"name": "format",
|
||||
"type": "COMFY_DYNAMICCOMBO_V3",
|
||||
"widget": {
|
||||
"name": "format"
|
||||
},
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"localized_name": "video",
|
||||
"name": "video",
|
||||
"type": "VIDEO",
|
||||
"links": []
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "SaveVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"vllm-omni/latent-edit",
|
||||
"mp4"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
2,
|
||||
0,
|
||||
4,
|
||||
0,
|
||||
"VIDEO"
|
||||
],
|
||||
[
|
||||
2,
|
||||
3,
|
||||
0,
|
||||
4,
|
||||
2,
|
||||
"MASK"
|
||||
],
|
||||
[
|
||||
3,
|
||||
4,
|
||||
0,
|
||||
6,
|
||||
6,
|
||||
"LATENT_MASK_EDITING"
|
||||
],
|
||||
[
|
||||
4,
|
||||
5,
|
||||
0,
|
||||
6,
|
||||
4,
|
||||
"VIDEO_PARAMS"
|
||||
],
|
||||
[
|
||||
5,
|
||||
6,
|
||||
0,
|
||||
7,
|
||||
0,
|
||||
"VIDEO"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {},
|
||||
"version": 0.4
|
||||
}
|
||||
@@ -1,219 +0,0 @@
|
||||
{
|
||||
"id": "6676f19a-7f41-4c80-8cb7-f80adcb4ac44",
|
||||
"revision": 0,
|
||||
"last_node_id": 9,
|
||||
"last_link_id": 4,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "LoadImage",
|
||||
"pos": [40, 20],
|
||||
"size": [270, 314],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{"localized_name": "image", "name": "image", "type": "COMBO", "widget": {"name": "image"}, "link": null},
|
||||
{"localized_name": "choose file to upload", "name": "upload", "type": "IMAGEUPLOAD", "widget": {"name": "upload"}, "link": null}
|
||||
],
|
||||
"outputs": [
|
||||
{"localized_name": "IMAGE", "name": "IMAGE", "type": "IMAGE", "links": [1]},
|
||||
{"localized_name": "MASK", "name": "MASK", "type": "MASK", "links": null}
|
||||
],
|
||||
"title": "First Frame",
|
||||
"properties": {"Node name for S&R": "LoadImage"},
|
||||
"widgets_values": ["first_frame.png", "image"]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "LoadImage",
|
||||
"pos": [40, 390],
|
||||
"size": [270, 314],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{"localized_name": "image", "name": "image", "type": "COMBO", "widget": {"name": "image"}, "link": null},
|
||||
{"localized_name": "choose file to upload", "name": "upload", "type": "IMAGEUPLOAD", "widget": {"name": "upload"}, "link": null}
|
||||
],
|
||||
"outputs": [
|
||||
{"localized_name": "IMAGE", "name": "IMAGE", "type": "IMAGE", "links": null},
|
||||
{"localized_name": "MASK", "name": "MASK", "type": "MASK", "links": null}
|
||||
],
|
||||
"title": "Last Frame (optional)",
|
||||
"properties": {"Node name for S&R": "LoadImage"},
|
||||
"widgets_values": ["last_frame.png", "image"]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [350, 20],
|
||||
"size": [390, 240],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"title": "MiniMax H3 I2V modes",
|
||||
"properties": {},
|
||||
"widgets_values": ["This workflow uses the remote vLLM-Omni Generate Video node.\n\n- First-frame only (default): keep First Frame connected.\n- Last-frame only: disconnect First Frame and connect Last Frame to last_frame.\n- First+last-frame: connect both image loaders.\n\nLeave the legacy frame and references inputs disconnected. Select images from ComfyUI's input folder. H3 runs at 24 FPS; the 5.167-second duration resolves to 124 frames, satisfying 17k+5."],
|
||||
"color": "#432",
|
||||
"bgcolor": "#000"
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "VLLMOmniDiffusionSampling",
|
||||
"pos": [350, 300],
|
||||
"size": [300, 226],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{"localized_name": "n", "name": "n", "type": "INT", "widget": {"name": "n"}, "link": null},
|
||||
{"localized_name": "num_inference_steps", "name": "num_inference_steps", "type": "INT", "widget": {"name": "num_inference_steps"}, "link": null},
|
||||
{"localized_name": "guidance_scale", "name": "guidance_scale", "type": "FLOAT", "widget": {"name": "guidance_scale"}, "link": null},
|
||||
{"localized_name": "true_cfg_scale", "name": "true_cfg_scale", "type": "FLOAT", "widget": {"name": "true_cfg_scale"}, "link": null},
|
||||
{"localized_name": "vae_use_slicing", "name": "vae_use_slicing", "type": "BOOLEAN", "widget": {"name": "vae_use_slicing"}, "link": null},
|
||||
{"localized_name": "vae_use_tiling", "name": "vae_use_tiling", "type": "BOOLEAN", "widget": {"name": "vae_use_tiling"}, "link": null},
|
||||
{"localized_name": "seed", "name": "seed", "type": "INT", "widget": {"name": "seed"}, "link": null}
|
||||
],
|
||||
"outputs": [
|
||||
{"localized_name": "diffusion sampling params", "name": "diffusion sampling params", "type": "SAMPLING_PARAMS", "links": [2]}
|
||||
],
|
||||
"title": "Base H3 Sampling (50 steps)",
|
||||
"properties": {"Node name for S&R": "VLLMOmniDiffusionSampling"},
|
||||
"widgets_values": [1, 50, 1, 1, false, false, 757358688076805, "fixed"]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "VLLMOmniMiniMaxH3Params",
|
||||
"pos": [350, 570],
|
||||
"size": [300, 82],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{"localized_name": "audio_flow_shift", "name": "audio_flow_shift", "type": "FLOAT", "widget": {"name": "audio_flow_shift"}, "link": null},
|
||||
{"localized_name": "flow_shift", "name": "flow_shift", "type": "FLOAT", "widget": {"name": "flow_shift"}, "link": null}
|
||||
],
|
||||
"outputs": [
|
||||
{"localized_name": "MiniMax-H3 video params", "name": "MiniMax-H3 video params", "type": "VIDEO_PARAMS", "links": [3]}
|
||||
],
|
||||
"title": "Base H3 Flow Shifts",
|
||||
"properties": {"Node name for S&R": "VLLMOmniMiniMaxH3Params"},
|
||||
"widgets_values": [3, 12]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "VLLMOmniRemoteLoRA",
|
||||
"pos": [700, 550],
|
||||
"size": [360, 150],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{"localized_name": "local_path", "name": "local_path", "type": "STRING", "widget": {"name": "local_path"}, "link": null},
|
||||
{"localized_name": "name", "name": "name", "type": "STRING", "widget": {"name": "name"}, "link": null},
|
||||
{"localized_name": "scale", "name": "scale", "type": "FLOAT", "widget": {"name": "scale"}, "link": null},
|
||||
{"localized_name": "int_id", "name": "int_id", "type": "INT", "widget": {"name": "int_id"}, "link": null}
|
||||
],
|
||||
"outputs": [
|
||||
{"localized_name": "lora", "name": "lora", "type": "REMOTE_LORA", "links": null}
|
||||
],
|
||||
"title": "Optional H3 Turbo Remote LoRA",
|
||||
"properties": {"Node name for S&R": "VLLMOmniRemoteLoRA"},
|
||||
"widgets_values": ["minimax_h3_fl2v_turbo_8step_v1.0_768p_bf16.safetensors", "h3-turbo-8step", 1, 0]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [700, 300],
|
||||
"size": [420, 210],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"title": "Optional Turbo settings",
|
||||
"properties": {},
|
||||
"widgets_values": ["Base H3 is configured by default: 50 steps, flow_shift 12, audio_flow_shift 3.\n\nFor the shown 8-step Diffusers Turbo artifact, set its full local_path on the server, start the server with the matching preloaded LoRA, connect this Remote LoRA to Generate Video, set num_inference_steps to 9, and set flow_shift to 6. Keep audio_flow_shift at 3 and scale at 1.\n\nDo not use the _comfyui_ repack or FastH3 here."],
|
||||
"color": "#432",
|
||||
"bgcolor": "#000"
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "VLLMOmniGenerateVideo",
|
||||
"pos": [1160, 80],
|
||||
"size": [440, 610],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{"localized_name": "frame", "name": "frame", "shape": 7, "type": "IMAGE", "link": null},
|
||||
{"localized_name": "first_frame", "name": "first_frame", "shape": 7, "type": "IMAGE", "link": 1},
|
||||
{"localized_name": "last_frame", "name": "last_frame", "shape": 7, "type": "IMAGE", "link": null},
|
||||
{"localized_name": "references", "name": "references", "shape": 7, "type": "VIDEO_REFERENCES", "link": null},
|
||||
{"localized_name": "sampling_params", "name": "sampling_params", "shape": 7, "type": "SAMPLING_PARAMS", "link": 2},
|
||||
{"localized_name": "lora", "name": "lora", "shape": 7, "type": "REMOTE_LORA", "link": null},
|
||||
{"localized_name": "model_params", "name": "model_params", "shape": 7, "type": "VIDEO_PARAMS", "link": 3},
|
||||
{"localized_name": "fast_h3", "name": "fast_h3", "shape": 7, "type": "FASTH3_DEPLOYMENT", "link": null},
|
||||
{"localized_name": "url", "name": "url", "type": "STRING", "widget": {"name": "url"}, "link": null},
|
||||
{"localized_name": "model", "name": "model", "type": "STRING", "widget": {"name": "model"}, "link": null},
|
||||
{"localized_name": "prompt", "name": "prompt", "type": "STRING", "widget": {"name": "prompt"}, "link": null},
|
||||
{"localized_name": "negative_prompt", "name": "negative_prompt", "type": "STRING", "widget": {"name": "negative_prompt"}, "link": null},
|
||||
{"localized_name": "width", "name": "width", "type": "INT", "widget": {"name": "width"}, "link": null},
|
||||
{"localized_name": "height", "name": "height", "type": "INT", "widget": {"name": "height"}, "link": null},
|
||||
{"localized_name": "fps", "name": "fps", "type": "INT", "widget": {"name": "fps"}, "link": null},
|
||||
{"localized_name": "duration", "name": "duration", "type": "FLOAT", "widget": {"name": "duration"}, "link": null}
|
||||
],
|
||||
"outputs": [
|
||||
{"localized_name": "video", "name": "video", "type": "VIDEO", "links": [4]}
|
||||
],
|
||||
"properties": {"Node name for S&R": "VLLMOmniGenerateVideo"},
|
||||
"widgets_values": [
|
||||
"http://localhost:8000/v1",
|
||||
"MiniMaxAI/MiniMax-H3",
|
||||
"A cinematic product shot begins exactly from the supplied keyframe. The subject moves naturally through a dark studio while blue and amber rim lights shift across reflective surfaces. The camera makes a slow controlled orbit. Audio: subtle room tone, precise mechanical details, and a restrained electronic score.",
|
||||
"",
|
||||
1344,
|
||||
768,
|
||||
24,
|
||||
5.167
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "SaveVideo",
|
||||
"pos": [1650, 170],
|
||||
"size": [374, 314],
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{"localized_name": "video", "name": "video", "type": "VIDEO", "link": 4},
|
||||
{"localized_name": "filename_prefix", "name": "filename_prefix", "type": "STRING", "widget": {"name": "filename_prefix"}, "link": null},
|
||||
{"localized_name": "format", "name": "format", "type": "COMBO", "widget": {"name": "format"}, "link": null},
|
||||
{"localized_name": "codec", "name": "codec", "type": "COMFY_DYNAMICCOMBO_V3", "widget": {"name": "codec"}, "link": null}
|
||||
],
|
||||
"outputs": [
|
||||
{"localized_name": "video", "name": "video", "type": "VIDEO", "links": null}
|
||||
],
|
||||
"properties": {},
|
||||
"widgets_values": ["video/MiniMax_H3_I2V", "auto", "auto"]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[1, 1, 0, 8, 1, "IMAGE"],
|
||||
[2, 4, 0, 8, 4, "SAMPLING_PARAMS"],
|
||||
[3, 5, 0, 8, 6, "VIDEO_PARAMS"],
|
||||
[4, 8, 0, 9, 0, "VIDEO"]
|
||||
],
|
||||
"groups": [
|
||||
{"id": 1, "title": "Portable first/last-frame inputs", "bounding": [10, -30, 760, 770], "color": "#3f789e", "flags": {}},
|
||||
{"id": 2, "title": "H3 sampling and optional Turbo", "bounding": [320, 270, 830, 460], "color": "#8a6d3b", "flags": {}},
|
||||
{"id": 3, "title": "Remote generation and audio-preserving save", "bounding": [1130, 30, 930, 700], "color": "#3f789e", "flags": {}}
|
||||
],
|
||||
"config": {},
|
||||
"extra": {"workflowRendererVersion": "LG", "ds": {"scale": 0.75, "offset": [180, 120]}},
|
||||
"version": 0.4
|
||||
}
|
||||
-5241
File diff suppressed because it is too large.
Load diff
@@ -52,7 +52,6 @@ const FAMILY_BY_OUTPUT = {
|
||||
REMOTE_LORA: "deployment",
|
||||
FASTH3_DEPLOYMENT: "deployment",
|
||||
VIDEO_REFERENCES: "media",
|
||||
LATENT_MASK_EDITING: "media",
|
||||
};
|
||||
|
||||
function familyOf(nodeData) {
|
||||
|
||||
@@ -41,17 +41,6 @@ LingBot-Video MoE transformer.
|
||||
- **Dense pipeline**: decoded-video MAE, MSE, PSNR, latency, and optional steady-state timings
|
||||
- **MoE transformer**: bitwise router, sparse-block, shared-expert, and full-transformer parity
|
||||
|
||||
### [LingBot-World](lingbot_world/README.md) — Realtime Streaming Cadence
|
||||
|
||||
Client-side benchmark for a `WS /v1/realtime/video` world-model session, where one
|
||||
request is a whole rollout and one AR block streams out as one video chunk. It
|
||||
measures a cadence rather than a throughput: there is exactly one session, and what
|
||||
matters is whether the next chunk arrives before the viewer finishes the last one.
|
||||
|
||||
- **Layout**: `lingbot_world/benchmark_lingbot_world_realtime.py` (WS client), `lingbot_world/workload.py` (rollout and metric math), `lingbot_world/configs/` (single-GPU eager and Ulysses-4 compiled deploy configs)
|
||||
- **Workload**: built-in image-conditioned rollout with a generated camera script, or a JSON spec with per-chunk actions and mid-rollout prompt updates
|
||||
- **Key metrics**: TTFC (time to first chunk), steady-state chunk inter-arrival percentiles, VIDEO_RTF (wall seconds per video second; lower is better), chunk-deadline attainment, and simulated playback underruns
|
||||
|
||||
### [Distributed](distributed/omni_connectors/README.md) — RDMA Connector Testing
|
||||
|
||||
RDMA environment setup and transfer tests for `MooncakeTransferEngineConnector`, including pytest-based single-node checks and manual cross-node benchmarks.
|
||||
@@ -66,13 +55,6 @@ Accuracy benchmarks for image generation/editing models, adapting external suite
|
||||
- **Layout**: `accuracy/text_to_image/` (GEBench), `accuracy/image_to_image/` (GEdit-Bench)
|
||||
- **Method**: generation and judge scoring both run through local `vllm-omni serve` endpoints
|
||||
|
||||
### [MammothModa2](mammoth_moda2/README.md) — Startup and Weight Loading
|
||||
|
||||
Startup / model-loading benchmark for the two-stage MammothModa2 (AR → DiT) deployment, plus a raw safetensors loading micro-benchmark.
|
||||
|
||||
- **Layout**: `mammoth_moda2/bench_startup.py` (single-process startup and first/steady request timing), `mammoth_moda2/parse_startup_log.py` (per-stage breakdown from engine logs), `mammoth_moda2/bench_storage_scenarios.sh` (cold/warm page cache on local disk vs. network storage), `mammoth_moda2/raw_load_bench.py` (safetensors → GPU without vLLM)
|
||||
- **Key metrics**: time to engine ready, per-stage spawn/init/weight-load/profile time, first vs. steady-state request latency
|
||||
|
||||
### Common serving metrics framework
|
||||
|
||||
`vllm_omni/benchmarks/` extends `vllm bench serve --omni` with Omni-specific datasets, backends, and multimodal metrics. Key metrics include:
|
||||
|
||||
@@ -1,6 +1,3 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
@@ -203,16 +200,7 @@ class VllmOmniImageClient:
|
||||
)
|
||||
last_response = response
|
||||
if response.status_code != 404:
|
||||
try:
|
||||
response.raise_for_status()
|
||||
except requests.HTTPError as error:
|
||||
# Keep the server's validation detail in the exception;
|
||||
# otherwise multipart image tests only report a bare
|
||||
# ``400 Bad Request`` and hide the actionable cause.
|
||||
raise requests.HTTPError(
|
||||
f"{error}: {response.text}",
|
||||
response=response,
|
||||
) from error
|
||||
response.raise_for_status()
|
||||
return decode_base64_image(response.json()["data"][0]["b64_json"])
|
||||
|
||||
assert last_response is not None
|
||||
|
||||
@@ -16,9 +16,9 @@ What changed:
|
||||
`/v1/images/edits` endpoint to produce benchmark outputs in the expected
|
||||
directory structure.
|
||||
- The evaluator keeps the same VIEScore-style decomposition:
|
||||
- `semantics_score`
|
||||
- `quality_score`
|
||||
- `overall_score = sqrt(semantics_score * quality_score)`
|
||||
- `semantics_score`
|
||||
- `quality_score`
|
||||
- `overall_score = sqrt(semantics_score * quality_score)`
|
||||
- Judge calls are routed to a local OpenAI-compatible model served by
|
||||
`vllm-omni`, not a remote provider.
|
||||
|
||||
|
||||
@@ -12,14 +12,14 @@ Upstream mapping:
|
||||
What changed:
|
||||
|
||||
- Generation calls the local OpenAI-compatible `vllm-omni` endpoints:
|
||||
- `/v1/images/generations` for text-only frame generation
|
||||
- `/v1/images/edits` for image-conditioned GUI transition generation
|
||||
- `/v1/images/generations` for text-only frame generation
|
||||
- `/v1/images/edits` for image-conditioned GUI transition generation
|
||||
- Evaluation still keeps the GEBench scoring dimensions:
|
||||
- `goal`
|
||||
- `logic`
|
||||
- `cons`
|
||||
- `ui`
|
||||
- `qual`
|
||||
- `goal`
|
||||
- `logic`
|
||||
- `cons`
|
||||
- `ui`
|
||||
- `qual`
|
||||
- Judge calls are also routed to a local OpenAI-compatible model served by
|
||||
`vllm-omni` instead of a remote service.
|
||||
|
||||
|
||||
@@ -164,22 +164,3 @@ batch may still pay compile or CUDA-graph capture cost.
|
||||
|
||||
For a Qwen-Image continuous-batching replay example, see
|
||||
[`performance_dashboard/qwen_image_serving_performance.md`](./performance_dashboard/qwen_image_serving_performance.md).
|
||||
|
||||
## HunyuanImage3 reference-prefix reuse
|
||||
|
||||
The prefix-cache benchmark uses the unified `vllm bench serve --omni` runner,
|
||||
not this legacy diffusion benchmark. It reuses the two-image IT2I input on a
|
||||
single DiT stage, comparing dense, paged without caching, and paged with caching.
|
||||
See [Shared-reference benchmark](../../docs/design/feature/prefix_caching.md#shared-reference-benchmark)
|
||||
for the configuration, commands, and separate partial-hit accuracy coverage.
|
||||
|
||||
The accuracy regression compares paged-no-cache, partial-hit and exact-repeat
|
||||
outputs against the checked-in official-repository reference-image goldens at
|
||||
**50 steps, CFG=2.5, seeds 43/45** (both CFG branches remain on CFGP1).
|
||||
It uses the AR-to-DiT image criteria: CLIP ≥ 90, SSIM ≥ 0.26, PSNR ≥ 12.5 dB.
|
||||
`HUNYUAN_IMAGE3_INCLUDE_DENSE=1` adds dense as another comparison to the goldens.
|
||||
It also requires actual reference-image hits and model-side query slicing,
|
||||
and checks bitwise repeatability for repeated requests at the same hit boundary.
|
||||
Images, logs, deployment YAMLs, hit traces and quality metrics are saved under
|
||||
pytest's temporary output directory. `HUNYUAN_IMAGE3_MODEL` selects a local model;
|
||||
DFX uses the repository's normal model/cache resolution.
|
||||
@@ -107,8 +107,6 @@ async def async_request_image_edits(
|
||||
|
||||
if extra_body.get("guidance_scale") is not None:
|
||||
form.add_field("guidance_scale", str(extra_body["guidance_scale"]))
|
||||
if extra_body.get("guidance_scale_2") is not None:
|
||||
form.add_field("guidance_scale_2", str(extra_body["guidance_scale_2"]))
|
||||
if extra_body.get("negative_prompt") is not None:
|
||||
form.add_field("negative_prompt", str(extra_body["negative_prompt"]))
|
||||
if extra_body.get("true_cfg_scale") is not None:
|
||||
|
||||
@@ -1,324 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Benchmark and exactness check for the Wan VAE decoder fast path.
|
||||
|
||||
Loads a diffusers Wan VAE (by default the Cosmos3 one), decodes seeded latents
|
||||
for the requested video size at each ``--fast-path`` level and reports wall
|
||||
time, peak memory and, against the ``off`` level, bitwise equality, max abs
|
||||
difference and PSNR. ``--profile`` adds a torch.profiler CUDA kernel table with
|
||||
the convolution / layout-transpose share.
|
||||
|
||||
Single GPU::
|
||||
|
||||
python benchmarks/diffusion/bench_wan_vae_decode.py --model nvidia/Cosmos3-Nano
|
||||
python benchmarks/diffusion/bench_wan_vae_decode.py --model nvidia/Cosmos3-Nano \
|
||||
--fast-path off,lossless --frames 33 --profile
|
||||
python benchmarks/diffusion/bench_wan_vae_decode.py --tiny --frames 9 --size 256x256
|
||||
|
||||
Multi-GPU VAE parallel decode (launch one process per GPU with ``torchrun``; the
|
||||
decode is timed across all ranks and only rank 0 prints)::
|
||||
|
||||
torchrun --nproc-per-node 2 benchmarks/diffusion/bench_wan_vae_decode.py --model nvidia/Cosmos3-Nano \
|
||||
--vae-patch-parallel-size 2 --vae-parallel-mode tile
|
||||
|
||||
``tile`` distributes the VAE's own spatial tiles over the ranks (it needs the
|
||||
video to exceed the tile size, 256x256 pixels by default) and ``spatial_shard_*``
|
||||
shards every decoder feature map along height or width with halo exchange. Both
|
||||
return the full video on rank 0 only, so equality and PSNR are computed there.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import math
|
||||
import os
|
||||
import time
|
||||
from collections.abc import Iterable
|
||||
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
|
||||
from vllm_omni.diffusion.distributed.autoencoders.autoencoder_kl_wan import (
|
||||
DistributedAutoencoderKLWan,
|
||||
OmniAutoencoderKLWan,
|
||||
)
|
||||
from vllm_omni.diffusion.distributed.autoencoders.wan_vae_fastpath import (
|
||||
VAE_FAST_PATH_LEVELS,
|
||||
install_wan_vae_fastpath,
|
||||
)
|
||||
|
||||
TINY_CONFIG = dict(
|
||||
base_dim=32,
|
||||
decoder_base_dim=64,
|
||||
z_dim=48,
|
||||
dim_mult=[1, 2, 4, 4],
|
||||
num_res_blocks=2,
|
||||
temperal_downsample=[False, True, True],
|
||||
is_residual=True,
|
||||
patch_size=2,
|
||||
in_channels=12,
|
||||
out_channels=12,
|
||||
scale_factor_temporal=4,
|
||||
scale_factor_spatial=16,
|
||||
)
|
||||
|
||||
DTYPES = {"bf16": torch.bfloat16, "fp16": torch.float16, "fp32": torch.float32}
|
||||
VAE_PARALLEL_MODES = ("tile", "spatial_shard_height", "spatial_shard_width")
|
||||
_CONV_TAGS = ("conv", "cudnn", "gemm", "cutlass", "xmma")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--model", default="nvidia/Cosmos3-Nano", help="HF id or local path of the pipeline")
|
||||
parser.add_argument("--subfolder", default="vae")
|
||||
parser.add_argument("--tiny", action="store_true", help="Use a small random VAE instead of loading weights")
|
||||
parser.add_argument("--size", default="1280x720", help="Output video WxH")
|
||||
parser.add_argument("--frames", type=int, default=189, help="Output frame count (1 + 4k)")
|
||||
parser.add_argument("--dtype", choices=sorted(DTYPES), default="bf16")
|
||||
parser.add_argument("--fast-path", default=",".join(VAE_FAST_PATH_LEVELS), help="Comma-separated levels")
|
||||
parser.add_argument("--warmup", type=int, default=1)
|
||||
parser.add_argument("--iters", type=int, default=2)
|
||||
parser.add_argument("--seed", type=int, default=0)
|
||||
parser.add_argument("--profile", action="store_true", help="Print a torch.profiler CUDA kernel table")
|
||||
parser.add_argument("--profile-rows", type=int, default=25)
|
||||
parser.add_argument("--save-output", default=None, help="Save the `off` decode output to this .pt path")
|
||||
parser.add_argument(
|
||||
"--vae-patch-parallel-size",
|
||||
type=int,
|
||||
default=1,
|
||||
help="VAE parallel decode over this many ranks (run under torchrun with as many processes)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--vae-parallel-mode",
|
||||
choices=VAE_PARALLEL_MODES,
|
||||
default="tile",
|
||||
help="tile: distribute the VAE's spatial tiles; spatial_shard_*: shard feature maps with halo exchange",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def init_distributed() -> tuple[int, int]:
|
||||
"""Join the torchrun process group (if any) and return ``(rank, world_size)``."""
|
||||
world_size = int(os.environ.get("WORLD_SIZE", "1"))
|
||||
if world_size <= 1:
|
||||
return 0, 1
|
||||
from vllm_omni.diffusion.distributed.parallel_state import (
|
||||
init_distributed_environment,
|
||||
initialize_model_parallel,
|
||||
)
|
||||
|
||||
rank = int(os.environ["RANK"])
|
||||
local_rank = int(os.environ.get("LOCAL_RANK", rank))
|
||||
init_distributed_environment(world_size=world_size, rank=rank, local_rank=local_rank)
|
||||
initialize_model_parallel(sequence_parallel_size=world_size, ulysses_degree=world_size)
|
||||
return rank, world_size
|
||||
|
||||
|
||||
def load_vae(
|
||||
args: argparse.Namespace, dtype: torch.dtype, device: torch.device, parallel: bool
|
||||
) -> OmniAutoencoderKLWan:
|
||||
cls = DistributedAutoencoderKLWan if parallel else OmniAutoencoderKLWan
|
||||
if args.tiny:
|
||||
torch.manual_seed(args.seed)
|
||||
vae = cls(**TINY_CONFIG).to(device=device, dtype=dtype).eval()
|
||||
if parallel:
|
||||
vae.init_distributed()
|
||||
else:
|
||||
vae = cls.from_pretrained(args.model, subfolder=args.subfolder, torch_dtype=dtype).to(device).eval()
|
||||
if parallel:
|
||||
# Same setup as the registry: parallel decode needs tiling on.
|
||||
vae.use_tiling = True
|
||||
vae.set_parallel_size(args.vae_patch_parallel_size, mode=args.vae_parallel_mode)
|
||||
return vae
|
||||
|
||||
|
||||
def make_latents(vae: OmniAutoencoderKLWan, args: argparse.Namespace, dtype: torch.dtype, device: torch.device):
|
||||
width, height = (int(v) for v in args.size.lower().split("x"))
|
||||
spatial = int(getattr(vae.config, "scale_factor_spatial", 16))
|
||||
temporal = int(getattr(vae.config, "scale_factor_temporal", 4))
|
||||
if width % spatial or height % spatial or (args.frames - 1) % temporal:
|
||||
raise SystemExit(f"size must be a multiple of {spatial} and frames must be 1 + {temporal}k")
|
||||
shape = (1, vae.config.z_dim, (args.frames - 1) // temporal + 1, height // spatial, width // spatial)
|
||||
generator = torch.Generator(device="cpu").manual_seed(args.seed)
|
||||
latents = torch.randn(shape, generator=generator).to(device=device, dtype=dtype)
|
||||
mean = getattr(vae.config, "latents_mean", None)
|
||||
std = getattr(vae.config, "latents_std", None)
|
||||
if mean is not None and std is not None:
|
||||
mean = torch.as_tensor(mean, device=device, dtype=dtype).view(1, -1, 1, 1, 1)
|
||||
std = torch.as_tensor(std, device=device, dtype=dtype).view(1, -1, 1, 1, 1)
|
||||
latents = latents * std + mean
|
||||
return latents
|
||||
|
||||
|
||||
@torch.inference_mode()
|
||||
def decode(vae: OmniAutoencoderKLWan, latents: torch.Tensor) -> torch.Tensor:
|
||||
return vae.decode(latents, return_dict=False)[0]
|
||||
|
||||
|
||||
def sync() -> None:
|
||||
"""Wait for this rank's GPU and, under torchrun, for every other rank."""
|
||||
torch.accelerator.synchronize()
|
||||
if dist.is_initialized():
|
||||
dist.barrier()
|
||||
|
||||
|
||||
def psnr(a: torch.Tensor, b: torch.Tensor) -> float:
|
||||
mse = torch.mean((a.float() - b.float()) ** 2).item()
|
||||
return math.inf if mse == 0 else 10 * math.log10(4.0 / mse) # outputs live in [-1, 1]
|
||||
|
||||
|
||||
def bits_equal(a: torch.Tensor, b: torch.Tensor) -> bool:
|
||||
if a.shape != b.shape or a.dtype != b.dtype:
|
||||
return False
|
||||
int_dtype = torch.int16 if a.element_size() == 2 else torch.int32
|
||||
return bool((a.contiguous().view(int_dtype) == b.contiguous().view(int_dtype)).all())
|
||||
|
||||
|
||||
def profile_decode(vae: OmniAutoencoderKLWan, latents: torch.Tensor, rows: int, verbose: bool) -> None:
|
||||
from torch.profiler import ProfilerActivity, profile
|
||||
|
||||
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=False) as prof:
|
||||
decode(vae, latents)
|
||||
sync()
|
||||
if not verbose:
|
||||
return
|
||||
events = [e for e in prof.key_averages() if e.device_time_total > 0]
|
||||
events.sort(key=lambda e: e.device_time_total, reverse=True)
|
||||
total = sum(e.device_time_total for e in events if e.device_type.name == "CUDA")
|
||||
if total == 0:
|
||||
total = sum(e.device_time_total for e in events)
|
||||
conv = sum(
|
||||
e.device_time_total
|
||||
for e in events
|
||||
if e.device_type.name == "CUDA" and any(tag in e.key.lower() for tag in _CONV_TAGS)
|
||||
)
|
||||
transpose = sum(e.device_time_total for e in events if e.device_type.name == "CUDA" and "nhwc" in e.key.lower())
|
||||
print(
|
||||
f" GPU kernel time: {total / 1e3:.1f} ms conv-like: {100 * conv / total:.1f}% "
|
||||
f"nchw<->nhwc transposes: {100 * transpose / total:.1f}%"
|
||||
)
|
||||
print(f" {'kernel':90s} {'ms':>9s} {'%':>6s} {'calls':>7s}")
|
||||
shown = 0
|
||||
for e in events:
|
||||
if e.device_type.name != "CUDA":
|
||||
continue
|
||||
share = 100 * e.device_time_total / total
|
||||
print(f" {e.key[:90]:90s} {e.device_time_total / 1e3:9.2f} {share:6.1f} {e.count:7d}")
|
||||
shown += 1
|
||||
if shown >= rows:
|
||||
break
|
||||
|
||||
|
||||
def run_level(
|
||||
args: argparse.Namespace,
|
||||
level: str,
|
||||
dtype: torch.dtype,
|
||||
device: torch.device,
|
||||
*,
|
||||
rank: int,
|
||||
parallel: bool,
|
||||
) -> tuple[torch.Tensor, dict]:
|
||||
vae = load_vae(args, dtype, device, parallel)
|
||||
report = install_wan_vae_fastpath(vae, level=level)
|
||||
latents = make_latents(vae, args, dtype, device)
|
||||
for _ in range(args.warmup):
|
||||
decode(vae, latents)
|
||||
sync()
|
||||
torch.accelerator.reset_peak_memory_stats()
|
||||
timings = []
|
||||
output = None
|
||||
for _ in range(args.iters):
|
||||
sync()
|
||||
start = time.perf_counter()
|
||||
output = decode(vae, latents)
|
||||
sync()
|
||||
timings.append(time.perf_counter() - start)
|
||||
peak = torch.tensor(torch.accelerator.max_memory_allocated() / 2**30, device=device)
|
||||
if dist.is_initialized():
|
||||
dist.all_reduce(peak, op=dist.ReduceOp.MAX) # the largest rank decides whether the config fits
|
||||
stats = {
|
||||
"installed": report.installed,
|
||||
"reason": report.reason,
|
||||
"fused_silu": report.fused_silu_dtypes,
|
||||
"patched": dict(report.patched),
|
||||
"time_s": min(timings),
|
||||
"peak_gib": peak.item(),
|
||||
}
|
||||
if args.profile:
|
||||
if rank == 0:
|
||||
print(f"[{level}] profile")
|
||||
profile_decode(vae, latents, args.profile_rows, verbose=rank == 0)
|
||||
assert output is not None
|
||||
del vae
|
||||
torch.accelerator.empty_cache()
|
||||
return output.detach(), stats
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if not torch.cuda.is_available():
|
||||
raise SystemExit("CUDA device required")
|
||||
rank, world_size = init_distributed()
|
||||
parallel = args.vae_patch_parallel_size > 1
|
||||
if parallel and world_size < args.vae_patch_parallel_size:
|
||||
raise SystemExit(
|
||||
f"--vae-patch-parallel-size {args.vae_patch_parallel_size} needs torchrun with at least that many "
|
||||
f"processes (WORLD_SIZE={world_size})"
|
||||
)
|
||||
device = torch.device("cuda", torch.accelerator.current_device_index()) if world_size > 1 else torch.device("cuda")
|
||||
dtype = DTYPES[args.dtype]
|
||||
levels: Iterable[str] = [level.strip() for level in args.fast_path.split(",") if level.strip()]
|
||||
for level in levels:
|
||||
if level not in VAE_FAST_PATH_LEVELS:
|
||||
raise SystemExit(f"unknown fast-path level {level!r}; choose from {VAE_FAST_PATH_LEVELS}")
|
||||
if "off" not in levels:
|
||||
levels = ["off", *levels]
|
||||
|
||||
if rank == 0:
|
||||
parallel_desc = f" vae_parallel={args.vae_parallel_mode}x{args.vae_patch_parallel_size}" if parallel else ""
|
||||
print(
|
||||
f"model={'tiny' if args.tiny else args.model} size={args.size} frames={args.frames} "
|
||||
f"dtype={args.dtype} world_size={world_size}{parallel_desc}"
|
||||
)
|
||||
golden = None
|
||||
baseline_time = None
|
||||
try:
|
||||
for level in levels:
|
||||
output, stats = run_level(args, level, dtype, device, rank=rank, parallel=parallel)
|
||||
if level == "off":
|
||||
golden = output
|
||||
baseline_time = stats["time_s"]
|
||||
if args.save_output and rank == 0:
|
||||
torch.save(output.cpu(), args.save_output)
|
||||
assert golden is not None
|
||||
if rank != 0:
|
||||
# Parallel decode assembles the video on rank 0 only; the other
|
||||
# ranks just keep the collectives in lockstep.
|
||||
del output
|
||||
continue
|
||||
speedup = baseline_time / stats["time_s"] if baseline_time else float("nan")
|
||||
line = (
|
||||
f"[{level:13s}] decode {stats['time_s'] * 1e3:9.1f} ms speedup {speedup:5.2f}x "
|
||||
f"peak {stats['peak_gib']:5.2f} GiB installed={stats['installed']}"
|
||||
)
|
||||
if level != "off":
|
||||
equal = torch.equal(output, golden)
|
||||
line += (
|
||||
f" torch_equal={equal} bits_equal={bits_equal(output, golden)} "
|
||||
f"max_abs_diff={(output.float() - golden.float()).abs().max().item():.3e} "
|
||||
f"psnr={psnr(output, golden):.2f} dB"
|
||||
)
|
||||
if stats["installed"]:
|
||||
line += f" fused_silu={stats['fused_silu'] or 'off'}"
|
||||
else:
|
||||
line += f" reason={stats['reason']}"
|
||||
print(line)
|
||||
del output
|
||||
torch.accelerator.empty_cache()
|
||||
finally:
|
||||
if dist.is_initialized():
|
||||
dist.destroy_process_group()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1107,11 +1107,6 @@ def calculate_metrics(
|
||||
error_outputs = [o for o in outputs if not o.success]
|
||||
|
||||
num_success = len(success_outputs)
|
||||
error_log_path = os.environ.get("BENCH_ERROR_LOG")
|
||||
if error_outputs and error_log_path:
|
||||
with open(error_log_path, "a") as f:
|
||||
for output in error_outputs:
|
||||
f.write(f"ERROR latency={output.latency:.3f} detail={output.error!r}\n")
|
||||
latencies = [o.latency for o in success_outputs]
|
||||
peak_memories = [o.peak_memory_mb for o in success_outputs if o.peak_memory_mb > 0]
|
||||
|
||||
|
||||
@@ -31,7 +31,6 @@ docker run -it \
|
||||
```
|
||||
|
||||
Parameter explanation:
|
||||
|
||||
- `--cap-add=SYS_PTRACE`: Allow reading system topology information
|
||||
- `--cap-add=IPC_LOCK`: Allow memory locking (required for RDMA memory registration)
|
||||
- `--security-opt seccomp=unconfined`: Disable seccomp restrictions
|
||||
@@ -61,7 +60,6 @@ When running single-node tests (producer and consumer on the same machine), ensu
|
||||
InfiniBand devices use LID (Local Identifier) for routing. Different devices have different LIDs and cannot communicate directly. If no device is specified, Mooncake may assign different devices to connectors, causing handshake failures.
|
||||
|
||||
Common error:
|
||||
|
||||
```
|
||||
[Handshake] Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num: Invalid argument [22]
|
||||
```
|
||||
@@ -251,7 +249,7 @@ python cross_node_mooncake_transfer_engine.py \
|
||||
### Transfer Modes
|
||||
|
||||
| Mode | Description | Example |
|
||||
| ------ | ------------- | --------- |
|
||||
|------|-------------|---------|
|
||||
| `copy` | Normal path — tensor copied to RDMA pool (default) | `--mode copy` |
|
||||
| `zerocopy` | Zero-copy path — data created directly in RDMA pool | `--mode zerocopy` |
|
||||
| `gpu` | GPU transfer — RDMA pool on GPU, uses GPUDirect | `--mode gpu --gpu-id 0` |
|
||||
@@ -283,7 +281,7 @@ python cross_node_mooncake_transfer_engine.py \
|
||||
### Cross-Node Test Options
|
||||
|
||||
| Option | Description | Default |
|
||||
| -------- | ------------- | --------- |
|
||||
|--------|-------------|---------|
|
||||
| `--role` | `producer` or `consumer` | Required |
|
||||
| `--local-host` | Local RDMA IP address | Required |
|
||||
| `--remote-host` | Remote RDMA IP address | Required |
|
||||
@@ -305,7 +303,6 @@ python cross_node_mooncake_transfer_engine.py \
|
||||
**Cause**: QP handshake failed, usually due to device configuration mismatch.
|
||||
|
||||
**Solution**:
|
||||
|
||||
```bash
|
||||
# Force using the same device
|
||||
export RDMA_DEVICE_NAME='mlx5_0'
|
||||
@@ -316,7 +313,6 @@ export RDMA_DEVICE_NAME='mlx5_0'
|
||||
**Cause**: Mooncake not installed or import failed.
|
||||
|
||||
**Solution**:
|
||||
|
||||
```bash
|
||||
# Check Mooncake installation
|
||||
python -c "from mooncake.engine import TransferEngine; print('OK')"
|
||||
@@ -333,7 +329,6 @@ uv pip install mooncake-transfer-engine
|
||||
**Cause**: Container lacks IB device access permissions.
|
||||
|
||||
**Solution**:
|
||||
|
||||
```bash
|
||||
docker run --device=/dev/infiniband --cap-add=IPC_LOCK ...
|
||||
```
|
||||
@@ -343,7 +338,6 @@ docker run --device=/dev/infiniband --cap-add=IPC_LOCK ...
|
||||
**Cause**: RDMA connection establishment failed or network latency.
|
||||
|
||||
**Solution**:
|
||||
|
||||
```bash
|
||||
# Check network status
|
||||
ibstat
|
||||
@@ -355,7 +349,6 @@ ibstatus
|
||||
**Cause**: CUDA environment not configured or GPU unavailable.
|
||||
|
||||
**Solution**:
|
||||
|
||||
```bash
|
||||
# Check CUDA
|
||||
python -c "import torch; print(torch.cuda.is_available())"
|
||||
@@ -369,7 +362,7 @@ docker run --gpus all ...
|
||||
## Environment Variables Reference
|
||||
|
||||
| Variable | Description | Example |
|
||||
| ---------- | ------------- | --------- |
|
||||
|----------|-------------|---------|
|
||||
| `RDMA_DEVICE_NAME` | Specify RDMA device name | `mlx5_0` |
|
||||
| `RDMA_TEST_HOST` | Specify test host IP | `10.0.0.1` |
|
||||
| `MC_TE_METRIC` | Enable Mooncake metrics | `1` |
|
||||
@@ -380,7 +373,7 @@ docker run --gpus all ...
|
||||
## Test Files Overview
|
||||
|
||||
| File | Description | Auto-discovered by pytest |
|
||||
| ------ | ------------- | -------------------------- |
|
||||
|------|-------------|--------------------------|
|
||||
| `test_mooncake_transfer_engine_rdma.py` | Integration tests for MooncakeTransferEngineConnector (basic, E2E, lifecycle, GPU) | Yes |
|
||||
| `test_mooncake_transfer_engine_buffer.py` | Memory pool and buffer management unit tests | Yes |
|
||||
| `cross_node_mooncake_transfer_engine.py` | Cross-node (multi-machine) testing script — run manually | No (filename does not start with `test_`) |
|
||||
@@ -388,7 +381,7 @@ docker run --gpus all ...
|
||||
### test_mooncake_transfer_engine_rdma.py — Test Classes
|
||||
|
||||
| Test Class | Memory Pool | Marker | Description |
|
||||
| ------------ | ------------- | -------- | ------------- |
|
||||
|------------|-------------|--------|-------------|
|
||||
| `TestBasicConnector` | CPU | — | Initialization, put tensor/bytes/object, cleanup, pool exhaustion |
|
||||
| `TestEndToEnd` | CPU | — | E2E RDMA transfer: tensor, bytes, object, zero-copy, large payload (100MB), mixed types, concurrency |
|
||||
| `TestLifecycle` | CPU | — | Close, context manager, double-close safety |
|
||||
@@ -398,7 +391,7 @@ docker run --gpus all ...
|
||||
### test_mooncake_transfer_engine_buffer.py — Test Classes
|
||||
|
||||
| Test Class | Marker | Description |
|
||||
| ------------ | -------- | ------------- |
|
||||
|------------|--------|-------------|
|
||||
| `TestBufferAllocator` | — | Basic alloc/free, alignment, exhaustion/recovery, thread safety |
|
||||
| `TestAllocatorInvariants` | `slow` | Double-free safety, overlap corruption detection, adjacent-block merging, fragmentation/defrag |
|
||||
| `TestManagedBuffer` | — | Tensor views, context manager |
|
||||
@@ -5,7 +5,7 @@ Benchmark GLM-Image T2I (text-to-image) and I2I (image-to-image) performance acr
|
||||
## Benchmarks
|
||||
|
||||
| Benchmark | Script | Description |
|
||||
| ----------- | -------- | ------------- |
|
||||
|-----------|--------|-------------|
|
||||
| HuggingFace Baseline | `huggingface/inference.py` | Single-GPU transformers + diffusers pipeline |
|
||||
| vLLM-Omni Offline | `vllm-omni/inference.py` | Offline inference with continuous batching |
|
||||
| vLLM-Omni Online | `benchmark_glm_image.py` | Online serving via `/v1/chat/completions` |
|
||||
@@ -27,7 +27,7 @@ CUDA_VISIBLE_DEVICES=0 python benchmarks/glm_image/huggingface/inference.py \
|
||||
### Options
|
||||
|
||||
| Flag | Default | Description |
|
||||
| ------ | --------- | ------------- |
|
||||
|------|---------|-------------|
|
||||
| `--model-path` | `zai-org/GLM-Image` | Model path |
|
||||
| `--mode` | `t2i` | `t2i` or `i2i` |
|
||||
| `--dataset-path` | `prompt/prompt.json` | Path to prompt.json |
|
||||
@@ -54,7 +54,7 @@ CUDA_VISIBLE_DEVICES=0,1 python benchmarks/glm_image/vllm-omni/inference.py \
|
||||
### Options
|
||||
|
||||
| Flag | Default | Description |
|
||||
| ------ | --------- | ------------- |
|
||||
|------|---------|-------------|
|
||||
| `--model-path` | `zai-org/GLM-Image` | Model path |
|
||||
| `--deploy-config` | - | Deploy config YAML |
|
||||
| `--mode` | `t2i` | `t2i` or `i2i` |
|
||||
@@ -100,7 +100,7 @@ python benchmarks/glm_image/benchmark_glm_image.py \
|
||||
### Options
|
||||
|
||||
| Flag | Default | Description |
|
||||
| ------ | --------- | ------------- |
|
||||
|------|---------|-------------|
|
||||
| `--mode` | `t2i` | `t2i` or `i2i` |
|
||||
| `--dataset` | `prompt` | `prompt`, `random`, or `custom` |
|
||||
| `--dataset-path` | - | JSON file path (required for `custom`) |
|
||||
@@ -134,7 +134,7 @@ Custom datasets use the same JSON format and can be provided via `--dataset-path
|
||||
All three benchmarks report per-stage pipeline timings (in milliseconds):
|
||||
|
||||
| Key | Description |
|
||||
| ----- | ------------- |
|
||||
|-----|-------------|
|
||||
| `preprocess_ms` | Input preprocessing (tokenization, multimodal encoding) |
|
||||
| `stage_0_gen_ms` | AR (autoregressive) model generation time |
|
||||
| `ar2diffusion_ms` | AR output to diffusion input conversion |
|
||||
@@ -148,7 +148,7 @@ The stages are ordered by execution: `preprocess → stage_0 (AR) → ar2diffusi
|
||||
Tested on 2x GPU with 10 prompts, 1024x1024, 50 denoising steps:
|
||||
|
||||
| Backend | Mode | Latency Mean (s) | Throughput (img/s) |
|
||||
| --------- | ------ | ------------------- | -------------------- |
|
||||
|---------|------|-------------------|--------------------|
|
||||
| HuggingFace | T2I | 72.6 | 0.014 |
|
||||
| HuggingFace | I2I | 70.9 | 0.014 |
|
||||
| vLLM-Omni Offline | T2I | 35.0 | 0.044 |
|
||||
|
||||
@@ -1,194 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Cold-cache torch.cuda timing and correctness gates for MAGI-2 microbenchmarks."""
|
||||
|
||||
import argparse
|
||||
import importlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import Any
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def emit(**fields: Any) -> None:
|
||||
LOGGER.info(json.dumps(fields, sort_keys=True))
|
||||
|
||||
|
||||
def add_arguments(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--tokens", type=int, nargs="+", default=[1, 4, 16, 64, 129, 4096])
|
||||
parser.add_argument("--dtype", choices=["bfloat16", "float16", "float32"], default="bfloat16")
|
||||
parser.add_argument("--num-tests", type=int, default=30, help="Measured samples per implementation.")
|
||||
parser.add_argument("--warmup", type=int, default=10, help="Interleaved warmup calls before sampling.")
|
||||
parser.add_argument("--seed", type=int, default=42)
|
||||
|
||||
|
||||
def load_musa(args: argparse.Namespace, parser: argparse.ArgumentParser) -> Any:
|
||||
if min(*args.tokens, args.num_tests) <= 0 or args.warmup < 0:
|
||||
parser.error("tokens and num-tests must be positive; warmup must be nonnegative")
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout)
|
||||
torch = importlib.import_module("torch")
|
||||
cuda = getattr(torch, "cuda")
|
||||
# torchada is needed only on MUSA; after it is loaded, use torch.cuda for
|
||||
# both CUDA and MUSA as documented by torchada.
|
||||
if hasattr(torch.version, "musa") and torch.version.musa is not None:
|
||||
try:
|
||||
importlib.import_module("torchada")
|
||||
except ModuleNotFoundError as error:
|
||||
if error.name != "torchada":
|
||||
raise
|
||||
has_gpu = cuda.is_available() or (
|
||||
hasattr(torch.version, "musa") and torch.version.musa is not None and torch.musa.is_available()
|
||||
)
|
||||
if not has_gpu:
|
||||
emit(status="skip", reason="CUDA/MUSA device is unavailable", torch=torch.__version__)
|
||||
return None
|
||||
torch.manual_seed(args.seed)
|
||||
cuda.manual_seed_all(args.seed)
|
||||
return torch
|
||||
|
||||
|
||||
def report_environment(torch: Any, args: argparse.Namespace, module: ModuleType) -> None:
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
packages = {}
|
||||
for name in ("torch", "torch_musa", "torchada", "triton", "vllm", "vllm-omni"):
|
||||
try:
|
||||
packages[name] = importlib.metadata.version(name)
|
||||
except importlib.metadata.PackageNotFoundError:
|
||||
packages[name] = None
|
||||
cuda = getattr(torch, "cuda")
|
||||
properties = cuda.get_device_properties(cuda.current_device())
|
||||
try:
|
||||
source_sha = subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip()
|
||||
source_dirty = bool(subprocess.check_output(["git", "-C", str(root), "status", "--porcelain"], text=True))
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
source_sha = os.environ.get("MAGI2_BENCH_SOURCE_SHA", "unknown")
|
||||
source_dirty = None
|
||||
emit(
|
||||
status="environment",
|
||||
command=sys.argv,
|
||||
source_root=str(root),
|
||||
source_sha=source_sha,
|
||||
source_dirty=source_dirty,
|
||||
kernel_source=module.__file__,
|
||||
device=cuda.get_device_name(cuda.current_device()),
|
||||
sm_count=getattr(properties, "multi_processor_count", None),
|
||||
packages=packages,
|
||||
env={name: os.environ.get(name) for name in ("MUSA_VISIBLE_DEVICES", "TORCHDYNAMO_DISABLE")},
|
||||
config=vars(args),
|
||||
mode="eager operator diagnostic; no compile or graph capture",
|
||||
comparison="forward_native versus the default CustomOp dispatch",
|
||||
timing="torch.cuda.Event; alternating native/fused calls; vanilla 512 MiB flush outside timed regions",
|
||||
timing_scope="whole callable, including device launch gaps; not isolated kernel duration",
|
||||
l2_flush=True,
|
||||
l2_flush_bytes=512 * 1024 * 1024,
|
||||
)
|
||||
|
||||
|
||||
class _LaunchProbe:
|
||||
def __init__(self, kernel: Any) -> None:
|
||||
self.kernel = kernel
|
||||
self.count = 0
|
||||
|
||||
def __getitem__(self, grid: Any) -> Any:
|
||||
self.count += 1
|
||||
return self.kernel[grid]
|
||||
|
||||
|
||||
def check_pair(
|
||||
torch: Any,
|
||||
reference: Callable,
|
||||
fused: Callable,
|
||||
module: ModuleType,
|
||||
kernels: tuple[str, ...],
|
||||
*,
|
||||
rtol: float,
|
||||
atol: float,
|
||||
) -> dict:
|
||||
cuda = getattr(torch, "cuda")
|
||||
expected = reference()
|
||||
probes = {name: _LaunchProbe(getattr(module, name)) for name in kernels}
|
||||
try:
|
||||
for name, probe in probes.items():
|
||||
setattr(module, name, probe)
|
||||
actual = fused()
|
||||
cuda.synchronize()
|
||||
finally:
|
||||
for name, probe in probes.items():
|
||||
setattr(module, name, probe.kernel)
|
||||
if any(probe.count != 1 for probe in probes.values()):
|
||||
raise RuntimeError(f"Expected one launch per fused kernel; got {[(k, p.count) for k, p in probes.items()]}")
|
||||
expected = expected if isinstance(expected, tuple) else (expected,)
|
||||
actual = actual if isinstance(actual, tuple) else (actual,)
|
||||
errors = []
|
||||
for result, baseline in zip(actual, expected, strict=True):
|
||||
for value in (baseline, result):
|
||||
if not torch.isfinite(value).all().item():
|
||||
raise AssertionError("Nonfinite benchmark output")
|
||||
if value.numel() > 1 and (value == value.flatten()[0]).all().item():
|
||||
raise AssertionError("Constant benchmark output")
|
||||
torch.testing.assert_close(result, baseline, rtol=rtol, atol=atol)
|
||||
difference = (result.float() - baseline.float()).abs()
|
||||
errors.append(
|
||||
{
|
||||
"max_abs": difference.max().item(),
|
||||
"max_rel": (difference / baseline.float().abs().clamp_min(1e-6)).max().item(),
|
||||
}
|
||||
)
|
||||
return {"parity": "pass", "errors": errors, "rtol": rtol, "atol": atol, "kernels": list(kernels)}
|
||||
|
||||
|
||||
def _percentile(samples: list[float], quantile: float) -> float:
|
||||
ordered = sorted(samples)
|
||||
position = (len(ordered) - 1) * quantile
|
||||
lower, upper = math.floor(position), math.ceil(position)
|
||||
return ordered[lower] + (ordered[upper] - ordered[lower]) * (position - lower)
|
||||
|
||||
|
||||
def measure_pair(torch: Any, reference: Callable, fused: Callable, args: argparse.Namespace) -> dict:
|
||||
cuda = getattr(torch, "cuda")
|
||||
functions = (reference, fused)
|
||||
order: list[int] = []
|
||||
|
||||
def alternate() -> Any:
|
||||
mode = len(order) % 2
|
||||
order.append(mode)
|
||||
return functions[mode]()
|
||||
|
||||
times_ms = []
|
||||
for _ in range(args.warmup):
|
||||
alternate()
|
||||
cuda.synchronize()
|
||||
order.clear()
|
||||
flush = torch.empty(512 * 1024 * 1024, dtype=torch.uint8, device="cuda")
|
||||
for _ in range(2 * args.num_tests):
|
||||
flush.zero_()
|
||||
cuda.synchronize()
|
||||
start = cuda.Event(enable_timing=True)
|
||||
end = cuda.Event(enable_timing=True)
|
||||
start.record()
|
||||
alternate()
|
||||
end.record()
|
||||
end.synchronize()
|
||||
times_ms.append(start.elapsed_time(end))
|
||||
samples: tuple[list[float], list[float]] = ([], [])
|
||||
for mode, milliseconds in zip(order[-len(times_ms) :], times_ms, strict=True):
|
||||
if not math.isfinite(milliseconds) or milliseconds <= 0:
|
||||
raise RuntimeError(f"Invalid GPU event sample: {milliseconds}")
|
||||
samples[mode].append(milliseconds * 1000)
|
||||
stats = {}
|
||||
for label, values in zip(("native", "fused"), samples, strict=True):
|
||||
if len(values) != args.num_tests:
|
||||
raise RuntimeError("GPU timing harness returned an unexpected sample count")
|
||||
p50 = _percentile(values, 0.5)
|
||||
stats[label] = {"median_us": p50, "p50_us": p50, "p90_us": _percentile(values, 0.9), "samples": len(values)}
|
||||
stats["speedup_p50"] = stats["native"]["p50_us"] / stats["fused"]["p50_us"]
|
||||
return stats
|
||||
@@ -1,194 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Cold-cache torch.cuda timing and correctness gates for MAGI-2 microbenchmarks."""
|
||||
|
||||
import argparse
|
||||
import importlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import Any
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def emit(**fields: Any) -> None:
|
||||
LOGGER.info(json.dumps(fields, sort_keys=True))
|
||||
|
||||
|
||||
def add_arguments(parser: argparse.ArgumentParser) -> None:
|
||||
parser.add_argument("--tokens", type=int, nargs="+", default=[1, 4, 16, 64, 129, 4096])
|
||||
parser.add_argument("--dtype", choices=["bfloat16", "float16", "float32"], default="bfloat16")
|
||||
parser.add_argument("--num-tests", type=int, default=30, help="Measured samples per implementation.")
|
||||
parser.add_argument("--warmup", type=int, default=10, help="Interleaved warmup calls before sampling.")
|
||||
parser.add_argument("--seed", type=int, default=42)
|
||||
|
||||
|
||||
def load_musa(args: argparse.Namespace, parser: argparse.ArgumentParser) -> Any:
|
||||
if min(*args.tokens, args.num_tests) <= 0 or args.warmup < 0:
|
||||
parser.error("tokens and num-tests must be positive; warmup must be nonnegative")
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout)
|
||||
torch = importlib.import_module("torch")
|
||||
cuda = getattr(torch, "cuda")
|
||||
# torchada is needed only on MUSA; after it is loaded, use torch.cuda for
|
||||
# both CUDA and MUSA as documented by torchada.
|
||||
if hasattr(torch.version, "musa") and torch.version.musa is not None:
|
||||
try:
|
||||
importlib.import_module("torchada")
|
||||
except ModuleNotFoundError as error:
|
||||
if error.name != "torchada":
|
||||
raise
|
||||
has_gpu = cuda.is_available() or (
|
||||
hasattr(torch.version, "musa") and torch.version.musa is not None and torch.musa.is_available()
|
||||
)
|
||||
if not has_gpu:
|
||||
emit(status="skip", reason="CUDA/MUSA device is unavailable", torch=torch.__version__)
|
||||
return None
|
||||
torch.manual_seed(args.seed)
|
||||
cuda.manual_seed_all(args.seed)
|
||||
return torch
|
||||
|
||||
|
||||
def report_environment(torch: Any, args: argparse.Namespace, module: ModuleType) -> None:
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
packages = {}
|
||||
for name in ("torch", "torch_musa", "torchada", "triton", "vllm", "vllm-omni"):
|
||||
try:
|
||||
packages[name] = importlib.metadata.version(name)
|
||||
except importlib.metadata.PackageNotFoundError:
|
||||
packages[name] = None
|
||||
cuda = getattr(torch, "cuda")
|
||||
properties = cuda.get_device_properties(cuda.current_device())
|
||||
try:
|
||||
source_sha = subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip()
|
||||
source_dirty = bool(subprocess.check_output(["git", "-C", str(root), "status", "--porcelain"], text=True))
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
source_sha = os.environ.get("MAGI2_BENCH_SOURCE_SHA", "unknown")
|
||||
source_dirty = None
|
||||
emit(
|
||||
status="environment",
|
||||
command=sys.argv,
|
||||
source_root=str(root),
|
||||
source_sha=source_sha,
|
||||
source_dirty=source_dirty,
|
||||
kernel_source=module.__file__,
|
||||
device=cuda.get_device_name(cuda.current_device()),
|
||||
sm_count=getattr(properties, "multi_processor_count", None),
|
||||
packages=packages,
|
||||
env={name: os.environ.get(name) for name in ("MUSA_VISIBLE_DEVICES", "TORCHDYNAMO_DISABLE")},
|
||||
config=vars(args),
|
||||
mode="eager operator diagnostic; no compile or graph capture",
|
||||
comparison="forward_native versus the default CustomOp dispatch",
|
||||
timing="torch.cuda.Event; alternating native/fused calls; vanilla 512 MiB flush outside timed regions",
|
||||
timing_scope="whole callable, including device launch gaps; not isolated kernel duration",
|
||||
l2_flush=True,
|
||||
l2_flush_bytes=512 * 1024 * 1024,
|
||||
)
|
||||
|
||||
|
||||
class _LaunchProbe:
|
||||
def __init__(self, kernel: Any) -> None:
|
||||
self.kernel = kernel
|
||||
self.count = 0
|
||||
|
||||
def __getitem__(self, grid: Any) -> Any:
|
||||
self.count += 1
|
||||
return self.kernel[grid]
|
||||
|
||||
|
||||
def check_pair(
|
||||
torch: Any,
|
||||
reference: Callable,
|
||||
fused: Callable,
|
||||
module: ModuleType,
|
||||
kernels: tuple[str, ...],
|
||||
*,
|
||||
rtol: float,
|
||||
atol: float,
|
||||
) -> dict:
|
||||
cuda = getattr(torch, "cuda")
|
||||
expected = reference()
|
||||
probes = {name: _LaunchProbe(getattr(module, name)) for name in kernels}
|
||||
try:
|
||||
for name, probe in probes.items():
|
||||
setattr(module, name, probe)
|
||||
actual = fused()
|
||||
cuda.synchronize()
|
||||
finally:
|
||||
for name, probe in probes.items():
|
||||
setattr(module, name, probe.kernel)
|
||||
if any(probe.count != 1 for probe in probes.values()):
|
||||
raise RuntimeError(f"Expected one launch per fused kernel; got {[(k, p.count) for k, p in probes.items()]}")
|
||||
expected = expected if isinstance(expected, tuple) else (expected,)
|
||||
actual = actual if isinstance(actual, tuple) else (actual,)
|
||||
errors = []
|
||||
for result, baseline in zip(actual, expected, strict=True):
|
||||
for value in (baseline, result):
|
||||
if not torch.isfinite(value).all().item():
|
||||
raise AssertionError("Nonfinite benchmark output")
|
||||
if value.numel() > 1 and (value == value.flatten()[0]).all().item():
|
||||
raise AssertionError("Constant benchmark output")
|
||||
torch.testing.assert_close(result, baseline, rtol=rtol, atol=atol)
|
||||
difference = (result.float() - baseline.float()).abs()
|
||||
errors.append(
|
||||
{
|
||||
"max_abs": difference.max().item(),
|
||||
"max_rel": (difference / baseline.float().abs().clamp_min(1e-6)).max().item(),
|
||||
}
|
||||
)
|
||||
return {"parity": "pass", "errors": errors, "rtol": rtol, "atol": atol, "kernels": list(kernels)}
|
||||
|
||||
|
||||
def _percentile(samples: list[float], quantile: float) -> float:
|
||||
ordered = sorted(samples)
|
||||
position = (len(ordered) - 1) * quantile
|
||||
lower, upper = math.floor(position), math.ceil(position)
|
||||
return ordered[lower] + (ordered[upper] - ordered[lower]) * (position - lower)
|
||||
|
||||
|
||||
def measure_pair(torch: Any, reference: Callable, fused: Callable, args: argparse.Namespace) -> dict:
|
||||
cuda = getattr(torch, "cuda")
|
||||
functions = (reference, fused)
|
||||
order: list[int] = []
|
||||
|
||||
def alternate() -> Any:
|
||||
mode = len(order) % 2
|
||||
order.append(mode)
|
||||
return functions[mode]()
|
||||
|
||||
times_ms = []
|
||||
for _ in range(args.warmup):
|
||||
alternate()
|
||||
cuda.synchronize()
|
||||
order.clear()
|
||||
flush = torch.empty(512 * 1024 * 1024, dtype=torch.uint8, device="cuda")
|
||||
for _ in range(2 * args.num_tests):
|
||||
flush.zero_()
|
||||
cuda.synchronize()
|
||||
start = cuda.Event(enable_timing=True)
|
||||
end = cuda.Event(enable_timing=True)
|
||||
start.record()
|
||||
alternate()
|
||||
end.record()
|
||||
end.synchronize()
|
||||
times_ms.append(start.elapsed_time(end))
|
||||
samples: tuple[list[float], list[float]] = ([], [])
|
||||
for mode, milliseconds in zip(order[-len(times_ms) :], times_ms, strict=True):
|
||||
if not math.isfinite(milliseconds) or milliseconds <= 0:
|
||||
raise RuntimeError(f"Invalid GPU event sample: {milliseconds}")
|
||||
samples[mode].append(milliseconds * 1000)
|
||||
stats = {}
|
||||
for label, values in zip(("native", "fused"), samples, strict=True):
|
||||
if len(values) != args.num_tests:
|
||||
raise RuntimeError("GPU timing harness returned an unexpected sample count")
|
||||
p50 = _percentile(values, 0.5)
|
||||
stats[label] = {"median_us": p50, "p50_us": p50, "p90_us": _percentile(values, 0.9), "samples": len(values)}
|
||||
stats["speedup_p50"] = stats["native"]["p50_us"] / stats["fused"]["p50_us"]
|
||||
return stats
|
||||
@@ -1,310 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Compare MAGI-2 routed MoE with equal preprocessing boundaries.
|
||||
|
||||
python -m benchmarks.kernels.benchmark_magi2_bf16_moe --tokens 2 4096 10000
|
||||
|
||||
Default (`--mode dispatch`): the same MAGI-2 layer is timed with its native
|
||||
reference and default BF16 fast path, including routing and all preprocessing.
|
||||
With `--mode routed`, routing probabilities/IDs are inputs to BOTH timed paths. Sorting,
|
||||
padding, packing, allocations and reduction remain inside their respective
|
||||
calls. The reference uses deterministic routed storage in diagnostic mode.
|
||||
Use --mode dispatch to measure the
|
||||
actual layer with the native and default fast paths, including the router. Unsupported
|
||||
dispatch fails explicitly; it is never silently replaced by another baseline.
|
||||
|
||||
Stage timings are standalone diagnostics, not additive or env speedup claims.
|
||||
These synthetic local shapes are not full-model performance measurements.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import statistics
|
||||
import subprocess
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
def parse_args(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--tokens", type=int, nargs="+", default=[2, 4096])
|
||||
parser.add_argument("--heads", type=int, default=3, help="EP4 local head count for MAGI-2 Preview.")
|
||||
parser.add_argument("--experts", type=int, default=256, help="Released experts per head.")
|
||||
parser.add_argument("--top-k", type=int, default=6)
|
||||
parser.add_argument("--warmup", type=int, default=10)
|
||||
parser.add_argument("--iters", type=int, default=50)
|
||||
parser.add_argument("--seed", type=int, default=419)
|
||||
parser.add_argument("--mode", choices=["routed", "dispatch"], default="dispatch")
|
||||
parser.add_argument(
|
||||
"--native-deterministic",
|
||||
action="store_true",
|
||||
help="Use the portable deterministic native reference (needed by Triton-MUSA 3.2).",
|
||||
)
|
||||
parser.add_argument("--stages", action="store_true")
|
||||
parser.add_argument("--output", type=Path)
|
||||
args = parser.parse_args(argv)
|
||||
if min(*args.tokens, args.heads, args.experts, args.top_k, args.iters) < 1 or args.warmup < 0:
|
||||
parser.error("Shapes/iters must be positive; warmup must be nonnegative")
|
||||
if args.top_k > args.experts:
|
||||
parser.error("top-k must not exceed experts per head")
|
||||
return args
|
||||
|
||||
|
||||
@contextmanager
|
||||
def env_switch(enabled):
|
||||
# Keep the legacy benchmark context API; dispatch is no longer opt-in.
|
||||
with patch.dict(os.environ, MAGI2_DETERMINISTIC="0"):
|
||||
yield
|
||||
|
||||
|
||||
def stats(values):
|
||||
if not values or any(not math.isfinite(x) or x <= 0 for x in values):
|
||||
raise ValueError("Missing/non-positive/non-finite timing sample")
|
||||
ordered = sorted(values)
|
||||
return {
|
||||
"mean_us": statistics.mean(values),
|
||||
"p50_us": statistics.median(values),
|
||||
"p90_us": ordered[math.ceil(0.9 * len(ordered)) - 1],
|
||||
"samples_us": values,
|
||||
}
|
||||
|
||||
|
||||
def measure(torch, calls, args):
|
||||
"""Alternating AB/BA; cold-cache flush is outside each timed region."""
|
||||
cuda = getattr(torch, "cuda")
|
||||
names = list(calls)
|
||||
sequence = names + names[::-1]
|
||||
wall_samples = []
|
||||
labels = []
|
||||
counter = 0
|
||||
|
||||
def invoke():
|
||||
nonlocal counter
|
||||
name = sequence[counter % len(sequence)]
|
||||
counter += 1
|
||||
cuda.synchronize()
|
||||
start = time.perf_counter_ns()
|
||||
output = calls[name]()
|
||||
cuda.synchronize()
|
||||
elapsed = (time.perf_counter_ns() - start) / 1000
|
||||
wall_samples.append(elapsed)
|
||||
labels.append(name)
|
||||
return output
|
||||
|
||||
count = len(names) * args.iters
|
||||
cache = torch.empty(512 * 1024 * 1024, device="cuda", dtype=torch.uint8)
|
||||
gpu_ms = []
|
||||
for index in range(len(names) * args.warmup + count):
|
||||
cache.zero_()
|
||||
cuda.synchronize()
|
||||
begin, end = cuda.Event(enable_timing=True), cuda.Event(enable_timing=True)
|
||||
begin.record()
|
||||
invoke()
|
||||
end.record()
|
||||
end.synchronize()
|
||||
if index >= len(names) * args.warmup:
|
||||
gpu_ms.append(begin.elapsed_time(end))
|
||||
labels, wall_samples = labels[-count:], wall_samples[-count:]
|
||||
result = {}
|
||||
for name in names:
|
||||
wall = [x for label, x in zip(labels, wall_samples) if label == name]
|
||||
event = [float(x) * 1000 for label, x in zip(labels, gpu_ms) if label == name]
|
||||
if len(wall) != args.iters or len(event) != args.iters:
|
||||
raise RuntimeError("Timing harness did not return the requested sample counts")
|
||||
result[name] = {"synchronized_wall": stats(wall), "gpu_event": stats(event)}
|
||||
return result
|
||||
|
||||
|
||||
def make_inputs(torch, tokens, args):
|
||||
# Fan-in-scaled synthetic weights. Unlike the old script, W_up and W_gate
|
||||
# have the same distribution and top-k IDs are unique within each token.
|
||||
torch.manual_seed(args.seed)
|
||||
x = torch.randn(tokens, args.heads, 256, device="cuda").to(torch.bfloat16)
|
||||
shape = (args.heads * args.experts, 256, 1280)
|
||||
gate, up = [(torch.randn(shape, device="cuda") / 16).to(torch.bfloat16) for _ in range(2)]
|
||||
down = (torch.randn(shape[0], 1280, 256, device="cuda") / math.sqrt(1280)).to(torch.bfloat16)
|
||||
logits = torch.randn(args.heads, tokens, args.experts, device="cuda")
|
||||
selected = logits.topk(args.top_k, dim=-1)
|
||||
probabilities = selected.values.softmax(-1)
|
||||
return x, probabilities, selected.indices, gate, up, down
|
||||
|
||||
|
||||
def check_and_capture(torch, moe, kernels, data, args):
|
||||
x, probabilities, indices, gate, up, down = data
|
||||
gather, weights, offsets = moe.global_sort_routes(probabilities, indices, args.experts)
|
||||
sorted_routes, launches = [], []
|
||||
scatter, invoke = kernels._deterministic_scatter, moe.invoke_fused_moe_bf16
|
||||
|
||||
def capture_scatter(routes, *params):
|
||||
sorted_routes.append(routes.detach().clone())
|
||||
return scatter(routes, *params)
|
||||
|
||||
def capture_launch(*params, **kwargs):
|
||||
launches.append((params, kwargs))
|
||||
return invoke(*params, **kwargs)
|
||||
|
||||
with patch.object(kernels, "_deterministic_scatter", capture_scatter):
|
||||
moe.triton_mh_moe_forward(x, gather, weights, offsets, gate, up, down, deterministic=True)
|
||||
packed_w13 = moe._pack_bf16_w13(gate, up)
|
||||
route_buffers = moe._allocate_bf16_route_buffers(
|
||||
x.shape[0] * args.heads * args.top_k,
|
||||
args.heads * args.experts,
|
||||
128,
|
||||
x.device,
|
||||
)
|
||||
with patch.object(moe, "invoke_fused_moe_bf16", capture_launch):
|
||||
actual = moe._bf16_fused_moe_forward(x, probabilities, indices, packed_w13, down, route_buffers)
|
||||
if len(sorted_routes) != 1 or len(launches) != 2:
|
||||
raise RuntimeError("Unexpected kernel dispatch: benchmark contract needs updating")
|
||||
flat_experts = (indices + torch.arange(args.heads, device=x.device)[:, None, None] * args.experts).flatten()
|
||||
order = flat_experts.argsort(stable=True)
|
||||
new_routes = launches[1][0][2].reshape(-1, 256).index_select(0, order)
|
||||
scatter_ids = gather.long() * args.heads + flat_experts[order] // args.experts
|
||||
expected = torch.zeros(x.shape[0] * args.heads, 256, device=x.device, dtype=torch.float32)
|
||||
expected.index_add_(0, scatter_ids, sorted_routes[0].float())
|
||||
expected = expected.to(x.dtype).view_as(x)
|
||||
errors = []
|
||||
# Same expert/FP32 reduction contract as test_bf16_moe_wiring.py, rather
|
||||
# than relaxing tolerances on the inherited BF16 scatter reduction.
|
||||
for result, reference in ((new_routes, sorted_routes[0]), (actual, expected)):
|
||||
if not (torch.isfinite(result).all() and torch.isfinite(reference).all()):
|
||||
raise RuntimeError("Non-finite MoE output")
|
||||
torch.testing.assert_close(result, reference, rtol=2e-2, atol=0.5)
|
||||
relative_l2 = (result.float() - reference.float()).norm() / reference.float().norm().clamp_min(1e-6)
|
||||
if relative_l2.item() >= 1e-3:
|
||||
raise AssertionError(f"MoE relative L2 {relative_l2.item()} >= 1e-3")
|
||||
errors.append(
|
||||
{"max_abs": (result.float() - reference.float()).abs().max().item(), "relative_l2": relative_l2.item()}
|
||||
)
|
||||
return launches, errors
|
||||
|
||||
|
||||
def benchmark(torch, moe, kernels, tokens, args):
|
||||
data = make_inputs(torch, tokens, args)
|
||||
x, probabilities, indices, gate, up, down = data
|
||||
launches, errors = check_and_capture(torch, moe, kernels, data, args)
|
||||
packed_w13 = moe._pack_bf16_w13(gate, up)
|
||||
route_count = tokens * args.heads * args.top_k
|
||||
route_buffers = moe._allocate_bf16_route_buffers(route_count, args.heads * args.experts, 128, x.device)
|
||||
|
||||
def reference():
|
||||
gather, weights, offsets = moe.global_sort_routes(probabilities, indices, args.experts)
|
||||
return moe.triton_mh_moe_forward(x, gather, weights, offsets, gate, up, down, deterministic=True)
|
||||
|
||||
calls = {
|
||||
"reference_deterministic_routed": reference,
|
||||
"candidate_routed": lambda: moe._bf16_fused_moe_forward(
|
||||
x, probabilities, indices, packed_w13, down, route_buffers
|
||||
),
|
||||
}
|
||||
if args.mode == "dispatch":
|
||||
config = moe.Magi2MultiHeadMoEConfig(
|
||||
args.heads * 256, args.heads, args.experts, args.top_k, 1280, torch.bfloat16
|
||||
)
|
||||
layer = moe.Magi2MultiHeadMoE(config).to(x.device)
|
||||
for parameter, value in ((layer.W_gate, gate), (layer.W_up, up), (layer.W_down, down)):
|
||||
parameter.copy_(value)
|
||||
layer.gate.normal_(std=1 / 16)
|
||||
|
||||
flat_x = x.reshape(tokens, -1)
|
||||
|
||||
def native_dispatch():
|
||||
x_heads = flat_x.view(tokens, args.heads, 256)
|
||||
probs, idx = layer._route(x_heads)
|
||||
gather, weights, offsets = moe.global_sort_routes(probs, idx, args.experts)
|
||||
return moe.triton_mh_moe_forward(
|
||||
x_heads,
|
||||
gather,
|
||||
weights,
|
||||
offsets,
|
||||
layer.W_gate,
|
||||
layer.W_up,
|
||||
layer.W_down,
|
||||
deterministic=args.native_deterministic,
|
||||
).reshape(tokens, -1)
|
||||
|
||||
calls = {"native_dispatch": native_dispatch, "fused_dispatch": lambda: layer(flat_x)}
|
||||
torch.testing.assert_close(calls["fused_dispatch"](), calls["native_dispatch"](), rtol=2e-2, atol=0.5)
|
||||
timings = measure(torch, calls, args)
|
||||
names = list(calls)
|
||||
result = {"status": "pass", "tokens": tokens, "mode": args.mode, "correctness": errors, "timings": timings}
|
||||
result["speedup_p50_wall"] = (
|
||||
timings[names[0]]["synchronized_wall"]["p50_us"] / timings[names[1]]["synchronized_wall"]["p50_us"]
|
||||
)
|
||||
if args.stages:
|
||||
route_ids = (indices.int() + torch.arange(args.heads, device=x.device)[:, None, None] * args.experts).reshape(
|
||||
-1, args.top_k
|
||||
)
|
||||
|
||||
def gems():
|
||||
for params, kwargs in launches:
|
||||
moe.invoke_fused_moe_bf16(*params, **kwargs)
|
||||
|
||||
result["standalone_stages_not_additive"] = measure(
|
||||
torch,
|
||||
{
|
||||
"reference_sort": lambda: moe.global_sort_routes(probabilities, indices, args.experts),
|
||||
"candidate_align": lambda: moe._align_bf16_routes(route_ids, args.heads * args.experts, 128),
|
||||
"candidate_pack_w13": lambda: torch.stack((gate.transpose(1, 2), up.transpose(1, 2)), dim=2).reshape(
|
||||
args.heads * args.experts, 2560, 256
|
||||
),
|
||||
"candidate_two_gemms_prepared": gems,
|
||||
},
|
||||
args,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
import torch
|
||||
|
||||
if hasattr(torch.version, "musa") and torch.version.musa is not None:
|
||||
importlib.import_module("torchada") # before importing vLLM / capturing CUDA APIs
|
||||
cuda = getattr(torch, "cuda")
|
||||
has_gpu = cuda.is_available() or (
|
||||
hasattr(torch.version, "musa") and torch.version.musa is not None and torch.musa.is_available()
|
||||
)
|
||||
if not has_gpu:
|
||||
print(json.dumps({"status": "skip", "reason": "No visible CUDA/MUSA device"}))
|
||||
return
|
||||
import vllm_omni.diffusion.models.magi2.fused_moe_kernels as kernels
|
||||
import vllm_omni.diffusion.models.magi2.mh_moe as moe
|
||||
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
try:
|
||||
sha = subprocess.check_output(
|
||||
["git", "-C", str(root), "rev-parse", "HEAD"], text=True, stderr=subprocess.DEVNULL
|
||||
).strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
sha = os.environ.get("MAGI2_BENCH_SOURCE_SHA", "unknown")
|
||||
report = {
|
||||
"device": cuda.get_device_name(),
|
||||
"sm_count": cuda.get_device_properties(0).multi_processor_count,
|
||||
"source_sha": sha,
|
||||
"module_path": moe.__file__,
|
||||
"torch": torch.__version__,
|
||||
"args": {**vars(args), "output": str(args.output)},
|
||||
"cache": "512 MiB flush outside each region",
|
||||
"capture": "eager; JIT warmed; no graph/compile",
|
||||
"results": [],
|
||||
}
|
||||
with torch.inference_mode():
|
||||
for tokens in args.tokens:
|
||||
result = benchmark(torch, moe, kernels, tokens, args)
|
||||
report["results"].append(result)
|
||||
print(json.dumps(result), flush=True)
|
||||
if args.output:
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(json.dumps(report, indent=2) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,141 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Compare native and fused MAGI-2 mHC post-processing on CUDA or MUSA.
|
||||
|
||||
Run from the checkout root with PyTorch and the selected backend dependencies.
|
||||
torchada is loaded automatically only on MUSA::
|
||||
|
||||
python -m benchmarks.kernels.benchmark_magi2_mhc \
|
||||
--tokens 1 4 16 64 129 4096 --hidden-size 3072 --num-tests 30
|
||||
|
||||
The released config supplies hidden=3072, four streams and 20 Sinkhorn steps.
|
||||
Token counts are synthetic; supply per-rank sizes from the target trace when
|
||||
comparing production shapes. FP32 post/residual logits are strided views of the
|
||||
same [tokens, 24] projection, as in MHCHandler.compute_logits. Mix inputs use
|
||||
the selected dtype. Both mix implementations consume identical coefficients.
|
||||
|
||||
The three comparisons cover coefficient preparation, stream mixing, and their
|
||||
combined pipeline (including intermediate allocations, excluding projection,
|
||||
normalization and apply_pre). This is an eager operator diagnostic, without
|
||||
compile or graph capture. It does not measure full model performance.
|
||||
No CUDA/MUSA device (or no PyTorch) produces a JSON skip record and exit status zero.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import math
|
||||
from functools import partial
|
||||
from typing import Any
|
||||
|
||||
from benchmarks.kernels._magi2_gpu_bench import (
|
||||
add_arguments,
|
||||
check_pair,
|
||||
emit,
|
||||
load_musa,
|
||||
measure_pair,
|
||||
report_environment,
|
||||
)
|
||||
|
||||
|
||||
def benchmark_shape(torch: Any, mhc: Any, args: argparse.Namespace, tokens: int) -> None:
|
||||
dtype = getattr(torch, args.dtype)
|
||||
device = torch.device("cuda")
|
||||
packed = torch.randn(tokens, 24, device=device, dtype=torch.float32)
|
||||
# Preserve the token stride of the model's shared projection allocation.
|
||||
post_inputs = (
|
||||
packed[:, 4:8],
|
||||
packed[:, 8:].view(tokens, 4, 4),
|
||||
torch.tensor([0.8], device=device),
|
||||
torch.randn(4, device=device),
|
||||
torch.tensor([1.1], device=device),
|
||||
torch.randn(4, 4, device=device),
|
||||
)
|
||||
streams = torch.randn(tokens, 4, args.hidden_size, device=device, dtype=dtype)
|
||||
branch = torch.randn(tokens, args.hidden_size, device=device, dtype=dtype)
|
||||
post_op, mix_op = mhc.MHCPostResidual(), mhc.MHCMix()
|
||||
scale = 1 / math.sqrt(4 * args.hidden_size)
|
||||
kwargs = dict(scale=scale, iterations=args.sinkhorn_iterations, epsilon=1e-12, out_dtype=dtype)
|
||||
prepare_reference = partial(post_op.forward_native, *post_inputs, **kwargs)
|
||||
prepare_fused = partial(post_op, *post_inputs, **kwargs)
|
||||
coefficients = prepare_reference()
|
||||
mix_reference = partial(mix_op.forward_native, streams, branch, *coefficients)
|
||||
mix_fused = partial(mix_op, streams, branch, *coefficients)
|
||||
post_kernel, mix_kernel = "_mhc_post_residual_kernel", "_mhc_mix_kernel"
|
||||
cases = {
|
||||
"post_residual": (prepare_reference, prepare_fused, (post_kernel,)),
|
||||
"mix": (mix_reference, mix_fused, (mix_kernel,)),
|
||||
"pipeline": (
|
||||
lambda: mix_op.forward_native(streams, branch, *prepare_reference()),
|
||||
lambda: mix_op(streams, branch, *prepare_fused()),
|
||||
(post_kernel, mix_kernel),
|
||||
),
|
||||
}
|
||||
rtol = {torch.bfloat16: 1.6e-2, torch.float16: 2e-3, torch.float32: 1e-5}[dtype]
|
||||
for name, (reference, fused, kernels) in cases.items():
|
||||
if args.op != "all" and args.op != name:
|
||||
continue
|
||||
# BF16 MUSA custom ops can quantize values near zero to the adjacent
|
||||
# representable value; retain the strict relative gate while allowing
|
||||
# one BF16 ulp in the absolute comparison.
|
||||
atol = (
|
||||
(1e-2 if dtype == torch.bfloat16 else 1e-6)
|
||||
if name == "post_residual"
|
||||
else (1e-2 if dtype == torch.bfloat16 else 1e-5)
|
||||
)
|
||||
parity = check_pair(torch, reference, fused, mhc, kernels, rtol=rtol, atol=atol)
|
||||
timing = measure_pair(torch, reference, fused, args)
|
||||
# Logical minimum traffic: input reads plus output writes. Native
|
||||
# intermediates and repeat parameter loads can add actual DRAM traffic.
|
||||
element_bytes = streams.element_size()
|
||||
prepare_bytes = tokens * 20 * (4 + element_bytes) + 22 * 4
|
||||
mix_bytes = tokens * (9 * args.hidden_size + 20) * element_bytes
|
||||
minimum_bytes = {"post_residual": prepare_bytes, "mix": mix_bytes, "pipeline": prepare_bytes + mix_bytes}[name]
|
||||
# Sigmoid/exp costs and max reductions are not represented as FLOPs.
|
||||
prepare_flops = tokens * (72 + 64 * args.sinkhorn_iterations)
|
||||
mix_flops = tokens * args.hidden_size * 36
|
||||
scalar_flops = {"post_residual": prepare_flops, "mix": mix_flops, "pipeline": prepare_flops + mix_flops}[name]
|
||||
emit(
|
||||
status="pass",
|
||||
op=name,
|
||||
tokens=tokens,
|
||||
hidden_size=args.hidden_size,
|
||||
num_streams=4,
|
||||
dtype=args.dtype,
|
||||
logits_dtype="float32",
|
||||
packed_shape=list(packed.shape),
|
||||
post_stride=list(post_inputs[0].stride()),
|
||||
residual_stride=list(post_inputs[1].stride()),
|
||||
scale=scale,
|
||||
sinkhorn_iterations=args.sinkhorn_iterations,
|
||||
epsilon=1e-12,
|
||||
minimum_io_bytes=minimum_bytes,
|
||||
scalar_flops_excluding_exp_and_max=scalar_flops,
|
||||
effective_fused_gbps=minimum_bytes / timing["fused"]["p50_us"] / 1000,
|
||||
**parity,
|
||||
**timing,
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
add_arguments(parser)
|
||||
parser.add_argument("--hidden-size", type=int, default=3072)
|
||||
parser.add_argument("--sinkhorn-iterations", type=int, default=20)
|
||||
parser.add_argument("--op", choices=["all", "post_residual", "mix", "pipeline"], default="all")
|
||||
args = parser.parse_args()
|
||||
if args.hidden_size <= 0 or not 0 <= args.sinkhorn_iterations <= 64:
|
||||
parser.error("hidden-size must be positive; sinkhorn-iterations must be in [0, 64]")
|
||||
torch = load_musa(args, parser)
|
||||
if torch is None:
|
||||
return
|
||||
cuda = getattr(torch, "cuda")
|
||||
from vllm_omni.diffusion.layers import mhc
|
||||
|
||||
report_environment(torch, args, mhc)
|
||||
with torch.inference_mode():
|
||||
for tokens in args.tokens:
|
||||
benchmark_shape(torch, mhc, args, tokens)
|
||||
cuda.empty_cache()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,87 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Compare native and fused MAGI-2 SwiGLU7 operators on CUDA or MUSA.
|
||||
|
||||
Run from the checkout root in an environment with PyTorch and the selected
|
||||
backend dependencies. torchada is loaded automatically only on MUSA::
|
||||
|
||||
python -m benchmarks.kernels.benchmark_magi2_swiglu7 \
|
||||
--tokens 1 4 16 64 129 4096 --intermediate-size 1280 --num-tests 30
|
||||
|
||||
The released MAGI-2 config supplies the 1280 intermediate width. Token counts
|
||||
are a synthetic sweep; pass per-rank sizes from the target trace for production
|
||||
comparisons. Input is contiguous interleaved gate/up [tokens, 2 * intermediate].
|
||||
Both paths evaluate FP32 arithmetic with one cast to the requested output dtype.
|
||||
This measures eager operator latency, not compiled serving performance.
|
||||
No CUDA/MUSA device (or no PyTorch) produces a JSON skip record and exit status zero.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
from functools import partial
|
||||
|
||||
from benchmarks.kernels._magi2_musa_bench import (
|
||||
add_arguments,
|
||||
check_pair,
|
||||
emit,
|
||||
load_musa,
|
||||
measure_pair,
|
||||
report_environment,
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
add_arguments(parser)
|
||||
parser.add_argument("--intermediate-size", type=int, default=1280)
|
||||
parser.add_argument("--out-dtype", choices=["bfloat16", "float16", "float32"], default=None)
|
||||
args = parser.parse_args()
|
||||
if args.intermediate_size <= 0:
|
||||
parser.error("intermediate-size must be positive")
|
||||
torch = load_musa(args, parser)
|
||||
if torch is None:
|
||||
return
|
||||
cuda = getattr(torch, "cuda")
|
||||
from vllm_omni.diffusion.layers import swiglu7
|
||||
|
||||
report_environment(torch, args, swiglu7)
|
||||
dtype = getattr(torch, args.dtype)
|
||||
out_dtype = getattr(torch, args.out_dtype or args.dtype)
|
||||
device = torch.device("cuda")
|
||||
rtol = {torch.bfloat16: 1.6e-2, torch.float16: 2e-3, torch.float32: 1e-5}[out_dtype]
|
||||
atol = 1e-5 if out_dtype == torch.float32 else 2e-3
|
||||
op = swiglu7.SwiGLU7()
|
||||
with torch.inference_mode():
|
||||
for tokens in args.tokens:
|
||||
# Values outside both clamp boundaries exercise the released formula.
|
||||
x = (torch.randn(tokens, 2 * args.intermediate_size, device=device) * 10).to(dtype)
|
||||
reference = partial(op.forward_native, x, out_dtype=out_dtype)
|
||||
fused = partial(op, x, out_dtype=out_dtype)
|
||||
parity = check_pair(torch, reference, fused, swiglu7, ("_swiglu7_kernel",), rtol=rtol, atol=atol)
|
||||
timing = measure_pair(torch, reference, fused, args)
|
||||
output_elements = tokens * args.intermediate_size
|
||||
minimum_io_bytes = (
|
||||
x.numel() * x.element_size() + output_elements * torch.empty((), dtype=out_dtype).element_size()
|
||||
)
|
||||
emit(
|
||||
status="pass",
|
||||
op="swiglu7",
|
||||
input_shape=list(x.shape),
|
||||
input_stride=list(x.stride()),
|
||||
dtype=args.dtype,
|
||||
out_dtype=args.out_dtype or args.dtype,
|
||||
alpha=1.702,
|
||||
limit=7.0,
|
||||
minimum_io_bytes=minimum_io_bytes,
|
||||
# Six scalar arithmetic operations and one exp per output;
|
||||
# clamps and transcendental costs are not modeled as FLOPs.
|
||||
scalar_flops_excluding_exp_and_clamps=6 * output_elements,
|
||||
effective_fused_gbps=minimum_io_bytes / timing["fused"]["p50_us"] / 1000,
|
||||
**parity,
|
||||
**timing,
|
||||
)
|
||||
del x, reference, fused
|
||||
cuda.empty_cache()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,6 +1,3 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
# ruff: noqa: N803
|
||||
|
||||
"""MoT (Mixture-of-Tokens) GEMM kernel benchmark and auto-tuning.
|
||||
@@ -13,7 +10,7 @@ Usage:
|
||||
python benchmarks/kernels/mot_linear_benchmarks.py \
|
||||
--model ByteDance-Seed/BAGEL-7B-MoT \
|
||||
--tp-size 1 --dtype w16a16 --tune \
|
||||
--save-dir vllm_omni/diffusion/models/bagel/mot/configs/
|
||||
--save-dir vllm_omni/diffusion/layers/mot/configs/
|
||||
|
||||
# Auto-tune with local model path (offline clusters):
|
||||
python benchmarks/kernels/mot_linear_benchmarks.py \
|
||||
@@ -46,7 +43,7 @@ from vllm.triton_utils import triton
|
||||
from vllm.utils.torch_utils import set_random_seed
|
||||
|
||||
# NOTE: you should use the same naming syetem for the kernel to load properly
|
||||
from vllm_omni.diffusion.models.bagel.mot.ops.mot_gemm import build_config_filename, get_device_name
|
||||
from vllm_omni.diffusion.layers.mot.ops.mot_gemm import build_config_filename, get_device_name
|
||||
|
||||
# clear the triton cache from time to time, usaully no need to change
|
||||
_CACHE_CLEAR_INTERVAL_ENV = "VLLM_MOT_TUNE_CACHE_CLEAR_INTERVAL"
|
||||
@@ -431,7 +428,7 @@ def benchmark_config(
|
||||
cache_flusher: torch.Tensor | None = None,
|
||||
) -> float:
|
||||
"""Run a MoT GEMM with the given tile config and return avg latency (us)."""
|
||||
from vllm_omni.diffusion.models.bagel.mot.ops.mot_gemm import invoke_mot_gemm
|
||||
from vllm_omni.diffusion.layers.mot.ops.mot_gemm import invoke_mot_gemm
|
||||
|
||||
text_indices, vae_indices, M = build_regular_indices(
|
||||
image_num=image_num,
|
||||
@@ -556,7 +553,7 @@ class BenchmarkWorker:
|
||||
use_int8_w8a16: bool,
|
||||
) -> tuple[dict[str, int], float]:
|
||||
set_random_seed(self.seed)
|
||||
from vllm_omni.diffusion.models.bagel.mot.ops.mot_gemm import (
|
||||
from vllm_omni.diffusion.layers.mot.ops.mot_gemm import (
|
||||
get_best_mot_config,
|
||||
)
|
||||
|
||||
|
||||
@@ -1,224 +0,0 @@
|
||||
# LingBot-World Benchmarks — Realtime Streaming Cadence
|
||||
|
||||
`benchmark_lingbot_world_realtime.py` measures LingBot-World 2.0 as it is
|
||||
actually deployed: one `WS /v1/realtime/video` session that streams one video
|
||||
chunk per AR block, for as long as the world keeps running.
|
||||
|
||||
## Why this is not the diffusion serving benchmark
|
||||
|
||||
[`benchmarks/diffusion`](../diffusion/README.md) measures "one request, one
|
||||
finished video" and reports request throughput. LingBot-World has neither shape:
|
||||
|
||||
- **One request is the whole rollout.** `prepare_encode` runs once, then every AR
|
||||
block is four DMD steps plus one clean-KV commit forward, and each block
|
||||
streams out as a chunk. Throughput in requests per second is meaningless.
|
||||
- **There is exactly one session.** The stepwise deploy config pins
|
||||
`max_num_seqs: 1`, and the AR-Diffusion consumer rejects a stage with more than
|
||||
one replica, because session-affine routing does not exist yet. There is no
|
||||
concurrency to sweep.
|
||||
- **What matters is cadence.** A deployment is healthy when chunk *N + 1* arrives
|
||||
before a viewer finishes watching chunk *N*. That is a latency distribution
|
||||
against a wall clock, not a throughput number.
|
||||
|
||||
## Quick start
|
||||
|
||||
Serve the model, then point the benchmark at it. The benchmark never starts a
|
||||
server.
|
||||
|
||||
```bash
|
||||
# Terminal 1 — one GPU, eager (portable smoke configuration)
|
||||
vllm serve robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
|
||||
--omni \
|
||||
--deploy-config benchmarks/lingbot_world/configs/single_gpu_eager.yaml \
|
||||
--port 8000
|
||||
|
||||
# Terminal 2
|
||||
pip install websockets
|
||||
python benchmarks/lingbot_world/benchmark_lingbot_world_realtime.py \
|
||||
--host 127.0.0.1 --port 8000 \
|
||||
--num-chunks 16 \
|
||||
--output-json /tmp/lingbot_world_realtime.json
|
||||
```
|
||||
|
||||
With no `--image`, the first frame comes from the shared vLLM image asset, so the
|
||||
default workload needs no setup and this repository carries no benchmark binary.
|
||||
Pass `--image /path/to/frame.png` to condition the rollout on your own frame.
|
||||
|
||||
### Realtime topology
|
||||
|
||||
The single-GPU config is a smoke configuration, not a realtime one: one card runs
|
||||
all five forwards of every block serially and nothing is compiled. Use the
|
||||
four-GPU config for numbers that mean anything about realtime behavior.
|
||||
|
||||
```bash
|
||||
vllm serve robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
|
||||
--omni \
|
||||
--deploy-config benchmarks/lingbot_world/configs/usp4_compiled.yaml \
|
||||
--port 8000
|
||||
|
||||
python benchmarks/lingbot_world/benchmark_lingbot_world_realtime.py \
|
||||
--port 8000 --num-chunks 40 --warmup-sessions 1 --sessions 3 \
|
||||
--target-fps 12 --output-json /tmp/lingbot_world_repeated.json
|
||||
```
|
||||
|
||||
`usp4_compiled.yaml` sets pure Ulysses sequence parallelism over four ranks and
|
||||
`enforce_eager: false`. The pipeline validates the parallel shape itself:
|
||||
`sequence_parallel_size` must equal `ulysses_degree`, `ring_degree` and
|
||||
`allgather_degree` must be 1, `ulysses_mode` must be `strict`, and pipeline,
|
||||
CFG, and VAE parallel sizes above 1 are rejected. This benchmark config sets
|
||||
`tensor_parallel_size: 1`; the pipeline validator does not reject larger TP sizes.
|
||||
|
||||
Compiled mode can pay compilation and capture costs on the first rollout.
|
||||
Use `--warmup-sessions 1` before measured repetitions. Excluding the first six
|
||||
chunks addresses the attention-window ramp; it does not guarantee that
|
||||
compilation and capture have finished.
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Meaning |
|
||||
| --- | --- |
|
||||
| `ttfc_ms` | **Time to first chunk**: `session.start` sent until the first chunk's bytes arrive. Pays for prompt encode, first-frame VAE encode, and any compile or capture. |
|
||||
| `interval_all` | Inter-arrival latency across every chunk: mean, median, std, min, p90, p95, p99, max. |
|
||||
| `interval_steady` | The same summary over steady-state chunks only. |
|
||||
| `video_rtf` | Wall seconds spent per second of video, matching vLLM-Omni's own RTF (`generation_s / video_duration`). **Lower is better; below 1.0 is real time.** |
|
||||
| `video_rtfx` | The reciprocal, under the name this repo uses for it: video seconds per wall second, where higher is better. |
|
||||
| `steady_video_rtf` | `video_rtf` restricted to steady-state chunks. |
|
||||
| `chunk_deadline_ms` | Wall-clock budget one steady chunk must meet to hold real time: 12 frames ÷ `--target-fps`. Override directly with `--chunk-slo-ms`. |
|
||||
| `slo_attainment` | Fraction of steady chunks that met the deadline. |
|
||||
| `playback` | Underrun count, total stall, and stall ratio from a simulated viewer. |
|
||||
|
||||
### Which frame rate counts as real time
|
||||
|
||||
**The checkpoint declares no frame rate.** There is none in `model_index.json`, the
|
||||
scheduler config, or the transformer config, and the pipeline never reads one. A
|
||||
chunk is twelve pixel frames; how long those frames *last* is a property of how the
|
||||
output is played, not of the model.
|
||||
|
||||
That choice moves the verdict without moving anything measured:
|
||||
|
||||
| basis | video per chunk | deadline | RTF at a measured 1357 ms/chunk |
|
||||
| --- | --- | --- | --- |
|
||||
| 16 fps | 0.750 s | 750 ms | 1.809 |
|
||||
| 12 fps | 1.000 s | 1000 ms | 1.357 |
|
||||
|
||||
So `--fps` (sent to the server, which uses it to label the muxed MP4) and
|
||||
`--target-fps` (the real-time basis for RTF, the deadline, and the playback
|
||||
simulation) are separate flags. `--target-fps` defaults to `--fps`; set it when the
|
||||
two differ, and the report prints both. State the basis whenever you quote a number
|
||||
— an RTF without its fps is not a measurement. (RTF is a cost here, so a bigger number is a
|
||||
worse result; a steady chunk's RTF is simply its interval divided by its video duration.)
|
||||
An RTF above 1 means the measured deployment is slower than real time at the
|
||||
stated target frame rate. The benchmark measures this failure as well as success.
|
||||
|
||||
### Why warmup chunks are not optional
|
||||
|
||||
The DiT attends over a sliding window of `sliding_window_num_frames = 18` latent
|
||||
frames with `sink_size = 9`, so the key count a block attends over keeps growing
|
||||
until eighteen latent frames of history exist — six chunks at three frames per
|
||||
block. **Chunk cost is not stationary before chunk six.** A mean over a short
|
||||
rollout is therefore lower than anything a running session will sustain, which is
|
||||
why `--warmup-chunks` defaults to 6 and steady-state metrics are reported
|
||||
separately. Keep `--num-chunks` comfortably above the warmup, or the benchmark
|
||||
tells you no chunk reached steady state. The terminal chunk skips next-chunk
|
||||
preparation, so it is excluded from steady metrics too; it remains in overall
|
||||
RTF, all-chunk intervals, and playback simulation.
|
||||
|
||||
The default 16-chunk rollout provides only nine steady intervals. Percentiles
|
||||
are interpolated sample summaries; the report warns when fewer than 100
|
||||
intervals contribute, and even 100 samples do not guarantee a stable p99.
|
||||
Use longer rollouts for tail analysis and at least `--sessions 3` after
|
||||
`--warmup-sessions 1` for reported comparisons. Session warmup pays compilation
|
||||
cost; chunk warmup separately excludes the attention-window ramp. The command
|
||||
above yields 33 steady intervals per session (indices 6 through 38), or 99 across
|
||||
three sessions. Report each session's mean and the spread across sessions; this
|
||||
sample still does not establish a reliable p99.
|
||||
|
||||
### Why playback is simulated
|
||||
|
||||
A p99 interval alone cannot say whether a viewer saw a stall: the same p99 is
|
||||
invisible behind a three-chunk buffer and a visible freeze behind none. The
|
||||
benchmark replays chunk arrivals against a wall clock — playback starts once
|
||||
`--playback-buffer-chunks` chunks have landed and then consumes video at `--target-fps`
|
||||
— and reports every moment the player ran dry.
|
||||
|
||||
### What the client-side number is worth
|
||||
|
||||
On a four-GPU Ulysses run the client's mean inter-arrival over all fifteen
|
||||
post-first-chunk intervals was **1326.743 ms**, against the server's own
|
||||
`StageRequestStats.inter_output_latency_ms` of **1327.479 ms** for the same
|
||||
request — a difference of **0.74 ms**. This cross-check shows agreement in
|
||||
average output cadence for that run, including warmup and the terminal chunk.
|
||||
It does not measure encoding or transport latency: a fixed delivery delay can
|
||||
cancel out of inter-arrival differences. Attributing that latency requires
|
||||
per-chunk timestamps at both boundaries or server-side profiling.
|
||||
|
||||
Compare matching interval populations when investigating discrepancies; both
|
||||
measurement bugs and delivery buffering can change the observed cadence.
|
||||
|
||||
### What this benchmark cannot see
|
||||
|
||||
Chunk metadata carries no server-side timestamps, so the benchmark cannot split
|
||||
DiT time from VAE decode time; use `--enable-diffusion-pipeline-profiler` on the
|
||||
server for that. It also reports no GPU memory, which is a server-side quantity.
|
||||
|
||||
## Custom rollouts
|
||||
|
||||
`--workload rollout.json` replaces the built-in workload, including mid-rollout
|
||||
prompt changes, which the CLI flags cannot express:
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": "The camera moves slowly forward through the scene.",
|
||||
"image": "/path/to/first_frame.png",
|
||||
"num_chunks": 24,
|
||||
"camera_pattern": "orbit",
|
||||
"fps": 16,
|
||||
"seed": 42,
|
||||
"prompt_updates": [
|
||||
{"after_chunk": 8, "prompt": "Rain begins to fall", "transition_chunks": 2}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Supply `camera_action_script` directly instead of `num_chunks` to control every
|
||||
frame's action. It must hold exactly one three-entry action list per chunk; one
|
||||
AR block is three latent frames, and the server rejects any other shape.
|
||||
Explicit `--image`, `--prompt`, `--negative-prompt`, `--width`, `--height`,
|
||||
`--fps`, `--seed`, and `--flow-shift` flags override the matching workload-file
|
||||
values. `--num-chunks` and `--camera-pattern` apply only to the built-in
|
||||
workload; when using `--workload`, set those fields in the file instead.
|
||||
An explicit `camera_action_script` takes precedence over the file's
|
||||
`num_chunks` and `camera_pattern`. Each prompt update must use a distinct
|
||||
`after_chunk` boundary so its interaction event ID is unique.
|
||||
|
||||
Camera actions change what the world does, not what it costs, so `--camera-pattern`
|
||||
(`forward`, `orbit`, `hold`) keeps a rollout representative rather than sweeping a
|
||||
cost dimension.
|
||||
|
||||
## Options worth knowing
|
||||
|
||||
| Flag | Purpose |
|
||||
| --- | --- |
|
||||
| `--target-fps` | Playback rate the real-time verdict is measured against, independent of the mux label. |
|
||||
| `--sessions N` | N sequential measured rollouts; the report adds spread across them. |
|
||||
| `--warmup-sessions N` | Unmeasured rollouts first, to pay compile and capture cost. |
|
||||
| `--print-chunks` | One line per chunk as it arrives. |
|
||||
| `--save-video PATH` | Remux the last measured session for eyeballing (needs vLLM-Omni). |
|
||||
| `--ping-interval` | `session.ping` cadence keeping the server's stall clock fresh during a slow first chunk. `0` disables. |
|
||||
| `--first-chunk-timeout` | Raise it when the server is still compiling or capturing graphs. |
|
||||
|
||||
## Output
|
||||
|
||||
`--output-json` writes the run's configuration, every chunk's arrival time and
|
||||
size, per-session metrics, and the aggregate, along with the git revision and
|
||||
hostname so a number can be traced back to what produced it.
|
||||
|
||||
## Tests
|
||||
|
||||
`tests/benchmarks/test_lingbot_world_realtime.py` covers the chunk and frame
|
||||
arithmetic against the server's own formula, the metric math, and the protocol
|
||||
handling against a scripted WebSocket server. It needs no GPU:
|
||||
|
||||
```bash
|
||||
pytest tests/benchmarks/test_lingbot_world_realtime.py
|
||||
```
|
||||
@@ -1,2 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
@@ -1,642 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Realtime streaming benchmark for LingBot-World 2.0 over ``WS /v1/realtime/video``.
|
||||
|
||||
Every other diffusion benchmark in this repository measures "one request, one
|
||||
finished video". LingBot-World is not that: one request is a whole rollout, and
|
||||
the thing a deployment lives or dies by is whether chunk *N + 1* keeps arriving
|
||||
before a viewer has finished watching chunk *N*. This benchmark therefore
|
||||
measures a **cadence**, not a throughput:
|
||||
|
||||
* ``TTFC`` -- session.start to the first chunk's bytes, which pays for prompt
|
||||
encode, the first-frame VAE encode, and any compile or capture the deploy
|
||||
config asks for;
|
||||
* **inter-arrival latency** over all chunks and, separately, over the steady
|
||||
state after the attention window saturates;
|
||||
* ``VIDEO_RTF`` -- wall seconds spent per second of video generated, matching how
|
||||
vLLM-Omni computes RTF everywhere else (``generation_s / video_duration``).
|
||||
**Lower is better and below 1.0 is real time**; ``VIDEO_RTFX`` is reported
|
||||
alongside it as the reciprocal, under the repo's own name for that;
|
||||
* **playback continuity** -- a simulated viewer with a start buffer, because a
|
||||
p99 that is harmless behind a three-chunk buffer is a visible stall behind
|
||||
none.
|
||||
|
||||
The server is not started here. Bring one up first (see the README), then point
|
||||
this script at it. Requires ``websockets``; ``--save-video`` additionally needs
|
||||
a vLLM-Omni installation for the remux helper.
|
||||
|
||||
Example:
|
||||
|
||||
python benchmarks/lingbot_world/benchmark_lingbot_world_realtime.py \\
|
||||
--host 127.0.0.1 --port 8000 \\
|
||||
--num-chunks 16 \\
|
||||
--output-json /tmp/lingbot_world_realtime.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import mimetypes
|
||||
import os
|
||||
import platform
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(REPO_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
from benchmarks.lingbot_world.workload import ( # noqa: E402
|
||||
CAMERA_PATTERNS,
|
||||
DEFAULT_FLOW_SHIFT,
|
||||
DEFAULT_FPS,
|
||||
DEFAULT_HEIGHT,
|
||||
DEFAULT_MODEL,
|
||||
DEFAULT_PROMPT,
|
||||
DEFAULT_SEED,
|
||||
DEFAULT_WARMUP_CHUNKS,
|
||||
DEFAULT_WIDTH,
|
||||
ChunkRecord,
|
||||
Workload,
|
||||
aggregate_metrics,
|
||||
compute_metrics,
|
||||
default_workload,
|
||||
load_workload,
|
||||
steady_chunk_deadline_ms,
|
||||
)
|
||||
|
||||
# The shared vLLM asset keeps the default workload runnable with no setup and
|
||||
# adds no binary to this repository. Any real photograph conditions a rollout.
|
||||
_DEFAULT_IMAGE_ASSET = "2560px-Gfp-wisconsin-madison-the-nature-boardwalk"
|
||||
_WS_PATH = "/v1/realtime/video"
|
||||
|
||||
|
||||
class BenchmarkError(RuntimeError):
|
||||
"""A session failed in a way that invalidates its numbers."""
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Input resolution
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def resolve_image_reference(value: str | None) -> str:
|
||||
"""Return an ``http(s)`` or ``data:`` URL for the session's first frame."""
|
||||
if value is None:
|
||||
from vllm.assets.image import ImageAsset
|
||||
|
||||
value = str(ImageAsset(_DEFAULT_IMAGE_ASSET).get_path("jpg"))
|
||||
if value.startswith(("http://", "https://", "data:")):
|
||||
return value
|
||||
path = Path(value).expanduser()
|
||||
if not path.is_file():
|
||||
raise BenchmarkError(f"--image is neither a URL nor a readable file: {value}")
|
||||
media_type = mimetypes.guess_type(path.name)[0] or "image/png"
|
||||
return f"data:{media_type};base64,{base64.b64encode(path.read_bytes()).decode()}"
|
||||
|
||||
|
||||
def build_workload(args: argparse.Namespace) -> Workload:
|
||||
if args.workload is not None:
|
||||
# The file names its own first frame, so resolving happens inside
|
||||
# load_workload -- never eagerly, which would fetch the default asset a
|
||||
# --workload run has no use for.
|
||||
return load_workload(
|
||||
args.workload,
|
||||
image_resolver=lambda value: resolve_image_reference(value if args.image is None else args.image),
|
||||
overrides={
|
||||
"width": args.width_override,
|
||||
"height": args.height_override,
|
||||
"fps": args.fps_override,
|
||||
"seed": args.seed_override,
|
||||
"prompt": args.prompt_override,
|
||||
"flow_shift": args.flow_shift,
|
||||
"negative_prompt": args.negative_prompt,
|
||||
},
|
||||
)
|
||||
return default_workload(
|
||||
image_reference=resolve_image_reference(args.image),
|
||||
num_chunks=args.num_chunks,
|
||||
camera_pattern=args.camera_pattern,
|
||||
prompt=args.prompt_override if args.prompt_override is not None else DEFAULT_PROMPT,
|
||||
width=args.width_override if args.width_override is not None else DEFAULT_WIDTH,
|
||||
height=args.height_override if args.height_override is not None else DEFAULT_HEIGHT,
|
||||
fps=args.fps_override if args.fps_override is not None else DEFAULT_FPS,
|
||||
seed=args.seed_override if args.seed_override is not None else DEFAULT_SEED,
|
||||
flow_shift=args.flow_shift if args.flow_shift is not None else DEFAULT_FLOW_SHIFT,
|
||||
negative_prompt=args.negative_prompt,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# One streaming session
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@dataclass
|
||||
class SessionResult:
|
||||
request_id: str | None
|
||||
records: list[ChunkRecord]
|
||||
media: bytes
|
||||
ttfc_s: float | None
|
||||
video_start_s: float | None
|
||||
wall_s: float
|
||||
done_event: dict[str, Any]
|
||||
|
||||
|
||||
async def _keepalive(websocket: Any, send_lock: asyncio.Lock, interval: float) -> None:
|
||||
"""Refresh the server's stall clock while a slow first chunk is generating.
|
||||
|
||||
A compiled deploy config records CUDA graphs on the first block, which can
|
||||
outlast the server's ~60 s stall timeout. ``session.ping`` is the documented
|
||||
way to say the client is still there.
|
||||
"""
|
||||
while True:
|
||||
await asyncio.sleep(interval)
|
||||
async with send_lock:
|
||||
await websocket.send(json.dumps({"type": "session.ping"}))
|
||||
|
||||
|
||||
async def run_session(
|
||||
url: str,
|
||||
model: str,
|
||||
workload: Workload,
|
||||
*,
|
||||
first_chunk_timeout: float,
|
||||
chunk_timeout: float,
|
||||
ping_interval: float,
|
||||
collect_media: bool,
|
||||
print_chunks: bool,
|
||||
) -> SessionResult:
|
||||
"""Drive one rollout and timestamp every chunk as its bytes land."""
|
||||
try:
|
||||
from websockets.asyncio.client import connect
|
||||
except ImportError: # pragma: no cover - exercised only without websockets
|
||||
try:
|
||||
from websockets import connect # type: ignore[attr-defined]
|
||||
except ImportError as exc:
|
||||
raise BenchmarkError("This benchmark needs the websockets package: pip install websockets") from exc
|
||||
|
||||
payload = workload.session_start_payload(model)
|
||||
records: list[ChunkRecord] = []
|
||||
media = bytearray()
|
||||
pending: dict[str, Any] | None = None
|
||||
request_id: str | None = None
|
||||
ttfc_s: float | None = None
|
||||
video_start_s: float | None = None
|
||||
last_arrival: float | None = None
|
||||
done_event: dict[str, Any] = {}
|
||||
send_lock = asyncio.Lock()
|
||||
keepalive: asyncio.Task[None] | None = None
|
||||
# Fire remaining prompt updates in chunk order as their boundary passes.
|
||||
updates = sorted(workload.prompt_updates, key=lambda update: update.after_chunk)
|
||||
update_index = 0
|
||||
|
||||
async with connect(url, max_size=None, ping_interval=None) as websocket:
|
||||
async with send_lock:
|
||||
await websocket.send(json.dumps(payload, ensure_ascii=False))
|
||||
started = time.perf_counter()
|
||||
if ping_interval > 0:
|
||||
keepalive = asyncio.create_task(_keepalive(websocket, send_lock, ping_interval))
|
||||
|
||||
try:
|
||||
while True:
|
||||
timeout = first_chunk_timeout if not records else chunk_timeout
|
||||
try:
|
||||
deadline = started + (last_arrival if last_arrival is not None else 0.0) + timeout
|
||||
remaining = deadline - time.perf_counter()
|
||||
if remaining <= 0:
|
||||
raise asyncio.TimeoutError
|
||||
message = await asyncio.wait_for(websocket.recv(), timeout=remaining)
|
||||
except asyncio.TimeoutError as exc:
|
||||
raise BenchmarkError(
|
||||
f"No media chunk for {timeout:g} s after {len(records)} chunk(s); "
|
||||
"the server may still be loading, compiling, or capturing graphs."
|
||||
) from exc
|
||||
now = time.perf_counter()
|
||||
|
||||
if isinstance(message, (bytes, bytearray)):
|
||||
if pending is None:
|
||||
raise BenchmarkError("Received a binary frame with no preceding video.chunk_metadata event.")
|
||||
if collect_media:
|
||||
media.extend(message)
|
||||
if pending.get("kind") == "media":
|
||||
index = int(pending["generation_chunk_index"])
|
||||
arrival = now - started
|
||||
if ttfc_s is None:
|
||||
ttfc_s = arrival
|
||||
records.append(
|
||||
ChunkRecord(
|
||||
index=index,
|
||||
arrival_s=arrival,
|
||||
inter_arrival_s=arrival - (last_arrival if last_arrival is not None else 0.0),
|
||||
byte_length=len(message),
|
||||
num_frames=int(pending["num_frames"]),
|
||||
)
|
||||
)
|
||||
last_arrival = arrival
|
||||
if print_chunks:
|
||||
print(
|
||||
f" chunk {index:3d} t={arrival:8.3f}s "
|
||||
f"interval={records[-1].inter_arrival_s * 1000:8.1f}ms "
|
||||
f"frames={records[-1].num_frames:3d} bytes={len(message)}",
|
||||
flush=True,
|
||||
)
|
||||
while update_index < len(updates) and updates[update_index].after_chunk <= index:
|
||||
async with send_lock:
|
||||
await websocket.send(json.dumps(updates[update_index].to_payload(), ensure_ascii=False))
|
||||
update_index += 1
|
||||
pending = None
|
||||
continue
|
||||
|
||||
event = json.loads(message)
|
||||
event_type = event.get("type")
|
||||
if event_type == "video.chunk_metadata":
|
||||
pending = event
|
||||
elif event_type == "video.start":
|
||||
request_id = event.get("request_id")
|
||||
video_start_s = now - started
|
||||
elif event_type == "error":
|
||||
# Say what was asked for alongside what the server refused:
|
||||
# some builds cap a rollout's num_frames, and the operator
|
||||
# needs to know which --num-chunks would fit.
|
||||
raise BenchmarkError(
|
||||
f"Server rejected the session after {len(records)} chunk(s): "
|
||||
f"{event.get('message', event)} "
|
||||
f"(requested {workload.num_chunks} chunks = num_frames {workload.num_frames})"
|
||||
)
|
||||
elif event_type == "session.done":
|
||||
done_event = event
|
||||
break
|
||||
finally:
|
||||
if keepalive is not None:
|
||||
keepalive.cancel()
|
||||
try:
|
||||
await keepalive
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
wall_s = time.perf_counter() - started
|
||||
|
||||
if not records:
|
||||
raise BenchmarkError("The session finished without delivering a single media chunk.")
|
||||
if done_event.get("stopped"):
|
||||
raise BenchmarkError(f"The session was stopped before it finished: {done_event}")
|
||||
observed = [record.index for record in records]
|
||||
if observed != list(range(len(records))):
|
||||
raise BenchmarkError(f"Chunk indices are not contiguous from zero: {observed}")
|
||||
if len(records) != workload.num_chunks:
|
||||
raise BenchmarkError(
|
||||
f"Expected {workload.num_chunks} chunks for num_frames={workload.num_frames}, received {len(records)}."
|
||||
)
|
||||
|
||||
return SessionResult(
|
||||
request_id=request_id,
|
||||
records=records,
|
||||
media=bytes(media),
|
||||
ttfc_s=ttfc_s,
|
||||
video_start_s=video_start_s,
|
||||
wall_s=wall_s,
|
||||
done_event=done_event,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Reporting
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _git_revision() -> str | None:
|
||||
try:
|
||||
return subprocess.check_output(
|
||||
["git", "-C", str(REPO_ROOT), "rev-parse", "HEAD"],
|
||||
text=True,
|
||||
stderr=subprocess.DEVNULL,
|
||||
).strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def _print_row(label: str, value: str) -> None:
|
||||
print(f"{label:<44}{value:>16}")
|
||||
|
||||
|
||||
def print_report(
|
||||
metrics: dict[str, Any],
|
||||
*,
|
||||
workload: Workload,
|
||||
endpoint: str,
|
||||
sessions: int,
|
||||
target_fps: float,
|
||||
) -> None:
|
||||
header = " LingBot-World realtime streaming benchmark "
|
||||
print("\n{s:{c}^{n}}".format(s=header, n=60, c="="))
|
||||
_print_row("Endpoint:", endpoint)
|
||||
_print_row("Geometry:", f"{workload.width}x{workload.height}")
|
||||
_print_row("Chunks per session:", str(workload.num_chunks))
|
||||
_print_row("Sessions:", str(sessions))
|
||||
frames = workload.num_frames
|
||||
_print_row("Real-time basis:", f"{target_fps:g} fps")
|
||||
_print_row("Video seconds per session:", f"{frames / target_fps:.2f} ({frames} frames)")
|
||||
if abs(target_fps - workload.fps) > 1e-9:
|
||||
_print_row("Mux label sent to server:", f"{workload.fps} fps")
|
||||
|
||||
if sessions == 1:
|
||||
print("{s:{c}^{n}}".format(s=" Cadence ", n=60, c="-"))
|
||||
if metrics.get("ttfc_ms") is not None:
|
||||
_print_row("TTFC (session.start -> first chunk):", f"{metrics['ttfc_ms']:.1f} ms")
|
||||
interval_all = metrics.get("interval_all") or {}
|
||||
interval_steady = metrics.get("interval_steady") or {}
|
||||
for title, summary in (("All chunks", interval_all), ("Steady state", interval_steady)):
|
||||
if not summary:
|
||||
continue
|
||||
print(f" {title} (n={int(summary['count'])}):")
|
||||
_print_row(" mean / median:", f"{summary['mean_ms']:.1f} / {summary['median_ms']:.1f} ms")
|
||||
_print_row(" std / max:", f"{summary['std_ms']:.1f} / {summary['max_ms']:.1f} ms")
|
||||
tail = f"{summary['p90_ms']:.0f} / {summary['p95_ms']:.0f} / {summary['p99_ms']:.0f} ms"
|
||||
_print_row(" p90 / p95 / p99:", tail)
|
||||
if summary["count"] < 100:
|
||||
print(
|
||||
" NOTE: fewer than 100 intervals; tail percentiles are descriptive, not reliable tail estimates."
|
||||
)
|
||||
|
||||
print("{s:{c}^{n}}".format(s=" Real time ", n=60, c="-"))
|
||||
_print_row("VIDEO_RTF (wall s / video s, <1 real time):", f"{metrics['video_rtf']:.3f}")
|
||||
_print_row("VIDEO_RTFX (video s / wall s):", f"{metrics['video_rtfx']:.3f}")
|
||||
if metrics.get("steady_video_rtf") is not None:
|
||||
_print_row("VIDEO_RTF steady state:", f"{metrics['steady_video_rtf']:.3f}")
|
||||
_print_row("Frames per second:", f"{metrics['frames_per_second']:.2f}")
|
||||
_print_row("Chunk deadline:", f"{metrics['chunk_deadline_ms']:.1f} ms")
|
||||
if metrics.get("slo_attainment") is not None:
|
||||
_print_row(
|
||||
"Steady chunks within deadline:",
|
||||
f"{metrics['slo_attainment'] * 100:.1f}% ({metrics['slo_violations']} late)",
|
||||
)
|
||||
if not metrics.get("steady_state_reached", False):
|
||||
print(
|
||||
f" NOTE: no chunk reached the steady state; the attention window needs "
|
||||
f"{metrics['warmup_chunks']} chunks. Raise --num-chunks."
|
||||
)
|
||||
|
||||
playback = metrics.get("playback") or {}
|
||||
print("{s:{c}^{n}}".format(s=" Playback continuity ", n=60, c="-"))
|
||||
_print_row("Start buffer:", f"{playback.get('buffer_chunks')} chunk(s)")
|
||||
_print_row("Underruns:", str(playback.get("underrun_count")))
|
||||
_print_row("Total stall:", f"{playback.get('total_stall_ms', 0.0):.1f} ms")
|
||||
_print_row("Stall ratio:", f"{playback.get('stall_ratio', 0.0) * 100:.2f}%")
|
||||
if playback.get("first_underrun_chunk") is not None:
|
||||
_print_row("First underrun at chunk:", str(playback["first_underrun_chunk"]))
|
||||
else:
|
||||
print("{s:{c}^{n}}".format(s=" Across sessions ", n=60, c="-"))
|
||||
if metrics.get("ttfc_ms_mean") is not None:
|
||||
_print_row("TTFC mean / max:", f"{metrics['ttfc_ms_mean']:.1f} / {metrics['ttfc_ms_max']:.1f} ms")
|
||||
if metrics.get("steady_interval_ms_mean") is not None:
|
||||
_print_row("Steady interval mean:", f"{metrics['steady_interval_ms_mean']:.1f} ms")
|
||||
_print_row("Steady interval spread:", f"{metrics['steady_interval_ms_spread']:.1f} ms")
|
||||
_print_row("VIDEO_RTF mean (lower is better):", f"{metrics['video_rtf_mean']:.3f}")
|
||||
_print_row(
|
||||
"VIDEO_RTF best / worst:",
|
||||
f"{metrics['video_rtf_best']:.3f} / {metrics['video_rtf_worst']:.3f}",
|
||||
)
|
||||
_print_row("Chunk deadline:", f"{metrics['chunk_deadline_ms']:.1f} ms")
|
||||
_print_row("Underruns (all sessions):", str(metrics["underrun_count"]))
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
def save_video(media: bytes, path: Path, *, fps: int) -> None:
|
||||
from vllm_omni.diffusion.utils.media_utils import finalize_streaming_video_bytes
|
||||
|
||||
playback_bytes = finalize_streaming_video_bytes(media, input_format="m4s", fps=fps)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp_path = path.with_suffix(path.suffix + ".tmp")
|
||||
tmp_path.write_bytes(playback_bytes)
|
||||
os.replace(tmp_path, path)
|
||||
print(f"Saved {len(media)} streamed bytes -> {len(playback_bytes)} playback bytes at {path}")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# CLI
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Benchmark LingBot-World 2.0 realtime streaming over WS /v1/realtime/video.",
|
||||
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
|
||||
)
|
||||
endpoint = parser.add_argument_group("endpoint")
|
||||
endpoint.add_argument("--host", default="127.0.0.1")
|
||||
endpoint.add_argument("--port", type=int, default=8000)
|
||||
endpoint.add_argument("--base-url", default=None, help="Full ws:// URL; overrides --host/--port.")
|
||||
endpoint.add_argument("--model", default=DEFAULT_MODEL)
|
||||
|
||||
workload = parser.add_argument_group("workload")
|
||||
workload.add_argument("--workload", default=None, help="JSON rollout spec; overrides the built-in workload.")
|
||||
workload.add_argument("--image", default=None, help="First frame: path or URL. Defaults to the shared vLLM asset.")
|
||||
workload.add_argument("--prompt", dest="prompt_override", default=None, help=f"Default: {DEFAULT_PROMPT!r}")
|
||||
workload.add_argument("--negative-prompt", default=None)
|
||||
workload.add_argument("--num-chunks", type=int, default=16, help="AR blocks per session; ignored with --workload.")
|
||||
workload.add_argument(
|
||||
"--camera-pattern",
|
||||
choices=CAMERA_PATTERNS,
|
||||
default="forward",
|
||||
help="Camera pattern for the built-in workload only; ignored with --workload.",
|
||||
)
|
||||
workload.add_argument("--width", dest="width_override", type=int, default=None, help=f"Default: {DEFAULT_WIDTH}")
|
||||
workload.add_argument("--height", dest="height_override", type=int, default=None, help=f"Default: {DEFAULT_HEIGHT}")
|
||||
workload.add_argument("--fps", dest="fps_override", type=int, default=None, help=f"Default: {DEFAULT_FPS}")
|
||||
workload.add_argument("--seed", dest="seed_override", type=int, default=None, help=f"Default: {DEFAULT_SEED}")
|
||||
workload.add_argument("--flow-shift", type=float, default=None, help=f"Default: {DEFAULT_FLOW_SHIFT}")
|
||||
|
||||
run = parser.add_argument_group("run")
|
||||
run.add_argument("--sessions", type=int, default=1, help="Sequential measured rollouts.")
|
||||
run.add_argument("--warmup-sessions", type=int, default=0, help="Unmeasured rollouts first (pays compile cost).")
|
||||
run.add_argument(
|
||||
"--warmup-chunks",
|
||||
type=int,
|
||||
default=DEFAULT_WARMUP_CHUNKS,
|
||||
help="Chunks excluded from steady state; the attention window saturates after 6.",
|
||||
)
|
||||
run.add_argument(
|
||||
"--chunk-slo-ms",
|
||||
type=float,
|
||||
default=None,
|
||||
help="Deadline a steady chunk must meet. Default: one chunk of video at --target-fps.",
|
||||
)
|
||||
run.add_argument(
|
||||
"--target-fps",
|
||||
type=float,
|
||||
default=None,
|
||||
help=(
|
||||
"Playback rate used for real-time factor, the chunk deadline, and the playback "
|
||||
"simulation. The checkpoint declares no frame rate, so this is a consumption "
|
||||
"choice, not a model property. Default: the --fps sent to the server."
|
||||
),
|
||||
)
|
||||
run.add_argument("--playback-buffer-chunks", type=int, default=1, help="Chunks buffered before playback starts.")
|
||||
run.add_argument("--first-chunk-timeout", type=float, default=900.0)
|
||||
run.add_argument("--chunk-timeout", type=float, default=300.0)
|
||||
run.add_argument("--ping-interval", type=float, default=20.0, help="session.ping cadence; 0 disables.")
|
||||
|
||||
output = parser.add_argument_group("output")
|
||||
output.add_argument("--output-json", default=None)
|
||||
output.add_argument("--save-video", default=None, help="Write the last measured session's video here.")
|
||||
output.add_argument("--print-chunks", action="store_true")
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
if args.sessions < 1:
|
||||
parser.error("--sessions must be at least 1.")
|
||||
if args.warmup_sessions < 0:
|
||||
parser.error("--warmup-sessions must be non-negative.")
|
||||
if args.warmup_chunks < 0:
|
||||
parser.error("--warmup-chunks must be non-negative.")
|
||||
if args.playback_buffer_chunks < 1:
|
||||
parser.error("--playback-buffer-chunks must be at least 1.")
|
||||
if args.target_fps is not None and args.target_fps <= 0:
|
||||
parser.error("--target-fps must be positive.")
|
||||
if args.workload is None and args.num_chunks < 1:
|
||||
parser.error("--num-chunks must be at least 1.")
|
||||
return args
|
||||
|
||||
|
||||
def endpoint_url(args: argparse.Namespace) -> str:
|
||||
if args.base_url is not None:
|
||||
base = args.base_url.rstrip("/")
|
||||
for prefix, replacement in (("http://", "ws://"), ("https://", "wss://")):
|
||||
if base.startswith(prefix):
|
||||
base = replacement + base[len(prefix) :]
|
||||
break
|
||||
return base if base.endswith(_WS_PATH) else base + _WS_PATH
|
||||
return f"ws://{args.host}:{args.port}{_WS_PATH}"
|
||||
|
||||
|
||||
async def run_benchmark(args: argparse.Namespace) -> dict[str, Any]:
|
||||
workload = build_workload(args)
|
||||
url = endpoint_url(args)
|
||||
# The mux label and the real-time basis are different questions; only the
|
||||
# latter decides whether a run passes.
|
||||
target_fps = args.target_fps if args.target_fps is not None else float(workload.fps)
|
||||
if max(1, args.warmup_chunks) >= workload.num_chunks - 1:
|
||||
print(
|
||||
f"WARNING: --warmup-chunks={args.warmup_chunks} leaves no nonterminal steady-state chunk in a "
|
||||
f"{workload.num_chunks}-chunk rollout; steady-state metrics will be empty.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
print(f"Connecting to {url}")
|
||||
print(
|
||||
f"Rollout: {workload.num_chunks} chunks, num_frames={workload.num_frames}, "
|
||||
f"{workload.video_seconds:.2f}s of video"
|
||||
)
|
||||
print(
|
||||
f"Real-time basis: {target_fps:g} fps -> a steady 12-frame chunk is "
|
||||
f"{steady_chunk_deadline_ms(target_fps):.1f} ms of video"
|
||||
+ ("" if args.target_fps is None else f" (mux label stays {workload.fps} fps)")
|
||||
)
|
||||
|
||||
for index in range(args.warmup_sessions):
|
||||
print(f"Warmup session {index + 1}/{args.warmup_sessions} (not measured)...")
|
||||
await run_session(
|
||||
url,
|
||||
args.model,
|
||||
workload,
|
||||
first_chunk_timeout=args.first_chunk_timeout,
|
||||
chunk_timeout=args.chunk_timeout,
|
||||
ping_interval=args.ping_interval,
|
||||
collect_media=False,
|
||||
print_chunks=False,
|
||||
)
|
||||
|
||||
session_metrics: list[dict[str, Any]] = []
|
||||
session_payloads: list[dict[str, Any]] = []
|
||||
last_result: SessionResult | None = None
|
||||
for index in range(args.sessions):
|
||||
if args.sessions > 1:
|
||||
print(f"Session {index + 1}/{args.sessions}...")
|
||||
result = await run_session(
|
||||
url,
|
||||
args.model,
|
||||
workload,
|
||||
first_chunk_timeout=args.first_chunk_timeout,
|
||||
chunk_timeout=args.chunk_timeout,
|
||||
ping_interval=args.ping_interval,
|
||||
collect_media=args.save_video is not None,
|
||||
print_chunks=args.print_chunks,
|
||||
)
|
||||
last_result = result
|
||||
metrics = compute_metrics(
|
||||
result.records,
|
||||
fps=target_fps,
|
||||
warmup_chunks=args.warmup_chunks,
|
||||
slo_ms=args.chunk_slo_ms,
|
||||
playback_buffer_chunks=args.playback_buffer_chunks,
|
||||
ttfc_s=result.ttfc_s,
|
||||
session_wall_s=result.wall_s,
|
||||
)
|
||||
session_metrics.append(metrics)
|
||||
session_payloads.append(
|
||||
{
|
||||
"session_index": index,
|
||||
"request_id": result.request_id,
|
||||
"video_start_ms": (result.video_start_s * 1000.0) if result.video_start_s is not None else None,
|
||||
"chunks": [record.to_dict() for record in result.records],
|
||||
"metrics": metrics,
|
||||
}
|
||||
)
|
||||
|
||||
aggregate = aggregate_metrics(session_metrics, fps=target_fps)
|
||||
print_report(aggregate, workload=workload, endpoint=url, sessions=args.sessions, target_fps=target_fps)
|
||||
|
||||
if args.save_video is not None and last_result is not None:
|
||||
save_video(last_result.media, Path(args.save_video), fps=workload.fps)
|
||||
|
||||
document = {
|
||||
"benchmark": "lingbot_world_realtime",
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"endpoint": url,
|
||||
"model": args.model,
|
||||
"git_revision": _git_revision(),
|
||||
"hostname": platform.node(),
|
||||
"workload": workload.describe(),
|
||||
"config": {
|
||||
"sessions": args.sessions,
|
||||
"warmup_sessions": args.warmup_sessions,
|
||||
"warmup_chunks": args.warmup_chunks,
|
||||
"camera_pattern": args.camera_pattern,
|
||||
"chunk_slo_ms": args.chunk_slo_ms,
|
||||
"target_fps": target_fps,
|
||||
"mux_fps": workload.fps,
|
||||
"playback_buffer_chunks": args.playback_buffer_chunks,
|
||||
"workload_file": args.workload,
|
||||
},
|
||||
"sessions": session_payloads,
|
||||
"metrics": aggregate,
|
||||
}
|
||||
if args.output_json is not None:
|
||||
path = Path(args.output_json)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(document, indent=2, sort_keys=True) + "\n")
|
||||
print(f"Wrote {path}")
|
||||
return document
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(argv)
|
||||
try:
|
||||
asyncio.run(run_benchmark(args))
|
||||
except BenchmarkError as error:
|
||||
print(f"ERROR: {error}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,45 +0,0 @@
|
||||
# Benchmark deploy config: LingBot-World 2.0 on one GPU, eager.
|
||||
#
|
||||
# This is the portable smoke configuration -- it matches the shipped
|
||||
# vllm_omni/deploy/lingbot_world_v2_stepwise.yaml so a number measured here is
|
||||
# comparable to the default deployment. It is NOT a realtime configuration: one
|
||||
# GPU runs all five forwards of every AR block serially and nothing is compiled,
|
||||
# so expect a chunk interval several times the real-time deadline.
|
||||
#
|
||||
# Use configs/usp4_compiled.yaml for the realtime topology.
|
||||
#
|
||||
# vllm serve robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
|
||||
# --omni --deploy-config benchmarks/lingbot_world/configs/single_gpu_eager.yaml \
|
||||
# --port 8000
|
||||
|
||||
pipeline: lingbot_world
|
||||
async_chunk: false
|
||||
distributed_executor_backend: mp
|
||||
dtype: bfloat16
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
devices: "0"
|
||||
# AR-Diffusion binds one session per runner invocation; the runner rejects
|
||||
# anything larger.
|
||||
max_num_seqs: 1
|
||||
enforce_eager: true
|
||||
model_class_name: LingBotWorldCausalDMDPipeline
|
||||
engine_backend: vllm_omni.experimental.ar_diffusion.engine.ARDiffusionEngine
|
||||
step_execution: true
|
||||
streaming_output: true
|
||||
default_sampling_params:
|
||||
# The causal-fast checkpoint pins a four-step DMD schedule; this sizes the
|
||||
# step budget before the pipeline is reached.
|
||||
num_inference_steps: 4
|
||||
output_type: pil
|
||||
extra_args:
|
||||
flow_shift: 5.0
|
||||
model_config:
|
||||
# The AR cache geometry is fixed at load time, so a benchmark request must
|
||||
# ask for exactly this resolution.
|
||||
ar_diffusion_height: 480
|
||||
ar_diffusion_width: 832
|
||||
ar_diffusion_kv_config:
|
||||
gpu_memory_fraction: 0.6
|
||||
warmup_cudagraph: true
|
||||
@@ -1,60 +0,0 @@
|
||||
# Benchmark deploy config: LingBot-World 2.0 on four GPUs, Ulysses SP-4, compiled.
|
||||
#
|
||||
# This is the realtime topology. Two things differ from the single-GPU config,
|
||||
# and both matter to the number the benchmark reports:
|
||||
#
|
||||
# parallel_config Pure Ulysses sequence parallelism over four ranks. The
|
||||
# pipeline validates this: LingBot requires
|
||||
# sequence_parallel_size == ulysses_degree, ring_degree and
|
||||
# allgather_degree of 1, ulysses_mode "strict", and
|
||||
# ulysses_a2a_permute disabled. Pipeline, CFG, and VAE
|
||||
# parallel sizes above 1 are rejected. This config uses
|
||||
# TP=1; the validator does not reject larger TP sizes.
|
||||
# enforce_eager false puts LingBotDMDBlockRunner on the compiled path
|
||||
# (regional torch.compile with cudagraph step marking). The
|
||||
# first rollout therefore pays compile and capture cost --
|
||||
# use --warmup-sessions 1 before measured repetitions.
|
||||
# Chunk warmup only excludes the attention-window ramp.
|
||||
#
|
||||
# Serve it under a GPU reservation, since it needs four cards:
|
||||
#
|
||||
# gpu run --gpus 4 -- vllm serve robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
|
||||
# --omni --deploy-config benchmarks/lingbot_world/configs/usp4_compiled.yaml \
|
||||
# --port 8000
|
||||
#
|
||||
# `devices` is the rank->card mapping inside the reservation, so it stays
|
||||
# "0,1,2,3" no matter which physical cards CUDA_VISIBLE_DEVICES selected.
|
||||
|
||||
pipeline: lingbot_world
|
||||
async_chunk: false
|
||||
distributed_executor_backend: mp
|
||||
dtype: bfloat16
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
devices: "0,1,2,3"
|
||||
max_num_seqs: 1
|
||||
enforce_eager: false
|
||||
model_class_name: LingBotWorldCausalDMDPipeline
|
||||
engine_backend: vllm_omni.experimental.ar_diffusion.engine.ARDiffusionEngine
|
||||
step_execution: true
|
||||
streaming_output: true
|
||||
parallel_config:
|
||||
tensor_parallel_size: 1
|
||||
sequence_parallel_size: 4
|
||||
ulysses_degree: 4
|
||||
ring_degree: 1
|
||||
default_sampling_params:
|
||||
num_inference_steps: 4
|
||||
output_type: pil
|
||||
extra_args:
|
||||
flow_shift: 5.0
|
||||
model_config:
|
||||
ar_diffusion_height: 480
|
||||
ar_diffusion_width: 832
|
||||
ar_diffusion_kv_config:
|
||||
# Each rank holds its own shard of the paged KV cache, so the per-card
|
||||
# fraction can stay at the single-GPU value; lower it if the four-rank
|
||||
# start OOMs on smaller cards.
|
||||
gpu_memory_fraction: 0.6
|
||||
warmup_cudagraph: true
|
||||
@@ -1,631 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Workload construction and metric math for the LingBot-World realtime benchmark.
|
||||
|
||||
Nothing here imports torch, vllm, or websockets, so the arithmetic that decides
|
||||
what a benchmark number *means* is unit-testable without a GPU or a server.
|
||||
|
||||
Two model properties drive every formula in this module, both read from the
|
||||
checkpoint's ``transformer/config.json`` rather than assumed:
|
||||
|
||||
``num_frames_per_block = 3``
|
||||
One AR block commits three latent frames, so one streamed chunk is three
|
||||
latent frames and a request's ``camera_action_script`` carries exactly one
|
||||
three-entry action list per chunk.
|
||||
|
||||
``sliding_window_num_frames = 18`` with ``sink_size = 9``
|
||||
Self-attention sees nine sink frames plus a nine-frame recent window, so the
|
||||
key count a block attends over keeps growing until eighteen latent frames of
|
||||
history exist -- six chunks. Chunk cost is therefore *not* stationary before
|
||||
chunk six, and a mean taken over a short rollout reports a number no steady
|
||||
session will ever produce. :data:`DEFAULT_WARMUP_CHUNKS` exists for that
|
||||
reason and is not a round-number guess.
|
||||
|
||||
The causal VAE expands the first latent frame to one pixel frame and every later
|
||||
latent frame to four, so chunk zero carries nine pixel frames and every chunk
|
||||
after it carries twelve. Video-duration math has to respect that asymmetry or
|
||||
real-time factor comes out wrong on short rollouts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
# transformer.config.num_frames_per_block: latent frames committed per AR block.
|
||||
FRAMES_PER_BLOCK = 3
|
||||
# vae_scale_factor_temporal: latent frames -> pixel frames, after the first.
|
||||
TEMPORAL_COMPRESSION = 4
|
||||
# (sink_size 9 + recent window 9) / FRAMES_PER_BLOCK: chunks until the attention
|
||||
# window saturates and per-chunk cost becomes stationary.
|
||||
DEFAULT_WARMUP_CHUNKS = 6
|
||||
|
||||
DEFAULT_MODEL = "robbyant/lingbot-world-v2-14b-causal-fast-diffusers"
|
||||
DEFAULT_PROMPT = "The camera moves slowly forward through the scene."
|
||||
# The AR KV cache geometry is fixed at load time, so a request must ask for
|
||||
# exactly the resolution the deploy config declares.
|
||||
DEFAULT_WIDTH = 832
|
||||
DEFAULT_HEIGHT = 480
|
||||
DEFAULT_FPS = 16
|
||||
DEFAULT_FLOW_SHIFT = 5.0
|
||||
DEFAULT_SEED = 42
|
||||
|
||||
_HOLD = "hold"
|
||||
_ACTION_KEYS = ("w", "a", "s", "d")
|
||||
|
||||
|
||||
def num_frames_for_chunks(num_chunks: int) -> int:
|
||||
"""Pixel-frame count whose rollout is exactly ``num_chunks`` chunks.
|
||||
|
||||
Inverse of :func:`chunks_for_num_frames`, which is the server's own
|
||||
``((num_frames - 1) // 4 + 1) // 3``.
|
||||
"""
|
||||
if num_chunks < 1:
|
||||
raise ValueError("num_chunks must be at least 1.")
|
||||
return (num_chunks * FRAMES_PER_BLOCK - 1) * TEMPORAL_COMPRESSION + 1
|
||||
|
||||
|
||||
def chunks_for_num_frames(num_frames: int) -> int:
|
||||
"""Chunks the server generates for ``num_frames``, mirroring its own formula."""
|
||||
if num_frames < 1:
|
||||
raise ValueError("num_frames must be at least 1.")
|
||||
return ((num_frames - 1) // TEMPORAL_COMPRESSION + 1) // FRAMES_PER_BLOCK
|
||||
|
||||
|
||||
def pixel_frames_in_chunk(chunk_index: int) -> int:
|
||||
"""Pixel frames carried by one chunk.
|
||||
|
||||
The causal decoder expands the session's opening latent frame once and every
|
||||
later latent frame four times, so chunk zero is short by three frames.
|
||||
"""
|
||||
if chunk_index < 0:
|
||||
raise ValueError("chunk_index must be non-negative.")
|
||||
if chunk_index == 0:
|
||||
return (FRAMES_PER_BLOCK - 1) * TEMPORAL_COMPRESSION + 1
|
||||
return FRAMES_PER_BLOCK * TEMPORAL_COMPRESSION
|
||||
|
||||
|
||||
def camera_actions_for_chunk(pattern: str, chunk_index: int) -> list[list[str]]:
|
||||
"""One three-entry action list for ``chunk_index``.
|
||||
|
||||
Camera actions select camera embeddings; they change what the world does,
|
||||
not how much arithmetic it costs. The pattern therefore exists to keep a
|
||||
benchmark rollout representative rather than to sweep a cost dimension.
|
||||
"""
|
||||
if chunk_index < 0:
|
||||
raise ValueError("chunk_index must be non-negative.")
|
||||
if pattern == "forward":
|
||||
return [["w"] for _ in range(FRAMES_PER_BLOCK)]
|
||||
if pattern == _HOLD:
|
||||
return [[] for _ in range(FRAMES_PER_BLOCK)]
|
||||
if pattern == "orbit":
|
||||
key = _ACTION_KEYS[chunk_index % len(_ACTION_KEYS)]
|
||||
return [[key] for _ in range(FRAMES_PER_BLOCK)]
|
||||
raise ValueError(f"Unknown camera pattern {pattern!r}; expected one of {sorted(CAMERA_PATTERNS)}.")
|
||||
|
||||
|
||||
CAMERA_PATTERNS = ("forward", "orbit", _HOLD)
|
||||
|
||||
|
||||
def build_camera_script(num_chunks: int, pattern: str) -> list[list[list[str]]]:
|
||||
"""Full ``camera_action_script``: one action list per generated chunk."""
|
||||
if num_chunks < 1:
|
||||
raise ValueError("num_chunks must be at least 1.")
|
||||
return [camera_actions_for_chunk(pattern, index) for index in range(num_chunks)]
|
||||
|
||||
|
||||
def _validate_camera_script(script: Sequence[Any]) -> list[list[list[str]]]:
|
||||
validated: list[list[list[str]]] = []
|
||||
for chunk_index, actions in enumerate(script):
|
||||
if not isinstance(actions, Sequence) or isinstance(actions, (str, bytes)):
|
||||
raise ValueError(f"camera_action_script[{chunk_index}] must be a list of three action lists.")
|
||||
if len(actions) != FRAMES_PER_BLOCK:
|
||||
raise ValueError(
|
||||
f"camera_action_script[{chunk_index}] must hold exactly {FRAMES_PER_BLOCK} action lists "
|
||||
f"(got {len(actions)}); one AR block is {FRAMES_PER_BLOCK} latent frames."
|
||||
)
|
||||
frames: list[list[str]] = []
|
||||
for frame_index, frame in enumerate(actions):
|
||||
if not isinstance(frame, Sequence) or isinstance(frame, (str, bytes)):
|
||||
raise ValueError(f"camera_action_script[{chunk_index}][{frame_index}] must be a list of strings.")
|
||||
if any(not isinstance(action, str) for action in frame):
|
||||
raise ValueError(f"camera_action_script[{chunk_index}][{frame_index}] must contain only strings.")
|
||||
frames.append([str(action) for action in frame])
|
||||
validated.append(frames)
|
||||
if not validated:
|
||||
raise ValueError("camera_action_script must not be empty.")
|
||||
return validated
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PromptUpdate:
|
||||
"""A mid-rollout ``session.interaction`` scheduled on a chunk boundary."""
|
||||
|
||||
after_chunk: int
|
||||
prompt: str
|
||||
transition_chunks: int = 3
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.after_chunk < 0:
|
||||
raise ValueError("PromptUpdate.after_chunk must be non-negative.")
|
||||
if not self.prompt.strip():
|
||||
raise ValueError("PromptUpdate.prompt must contain non-whitespace text.")
|
||||
if self.transition_chunks < 0:
|
||||
raise ValueError("PromptUpdate.transition_chunks must be non-negative.")
|
||||
|
||||
def to_payload(self) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "session.interaction",
|
||||
"interaction": {
|
||||
"event_id": f"chunk-{self.after_chunk}",
|
||||
"event": {"prompt": self.prompt},
|
||||
"transition_chunks": self.transition_chunks,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Workload:
|
||||
"""One rollout: what the benchmark asks the server to generate."""
|
||||
|
||||
prompt: str
|
||||
image_reference: str
|
||||
camera_script: list[list[list[str]]]
|
||||
width: int = DEFAULT_WIDTH
|
||||
height: int = DEFAULT_HEIGHT
|
||||
fps: int = DEFAULT_FPS
|
||||
seed: int = DEFAULT_SEED
|
||||
flow_shift: float = DEFAULT_FLOW_SHIFT
|
||||
negative_prompt: str | None = None
|
||||
prompt_updates: tuple[PromptUpdate, ...] = ()
|
||||
extra_params: Mapping[str, Any] = field(default_factory=dict)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.prompt.strip():
|
||||
raise ValueError("Workload.prompt must contain non-whitespace text.")
|
||||
if not self.image_reference:
|
||||
raise ValueError("Workload.image_reference is required; LingBot-World is image-conditioned.")
|
||||
if self.width <= 0 or self.height <= 0 or self.width % 16 or self.height % 16:
|
||||
raise ValueError("Workload width/height must be positive multiples of 16.")
|
||||
if self.fps <= 0:
|
||||
raise ValueError("Workload.fps must be positive.")
|
||||
object.__setattr__(self, "camera_script", _validate_camera_script(self.camera_script))
|
||||
seen_update_chunks: set[int] = set()
|
||||
for update in self.prompt_updates:
|
||||
if update.after_chunk in seen_update_chunks:
|
||||
raise ValueError(f"Duplicate prompt update after chunk {update.after_chunk}; event IDs must be unique.")
|
||||
seen_update_chunks.add(update.after_chunk)
|
||||
if update.after_chunk >= self.num_chunks:
|
||||
raise ValueError(
|
||||
f"Prompt update after chunk {update.after_chunk} never fires in a {self.num_chunks}-chunk rollout."
|
||||
)
|
||||
|
||||
@property
|
||||
def num_chunks(self) -> int:
|
||||
return len(self.camera_script)
|
||||
|
||||
@property
|
||||
def num_frames(self) -> int:
|
||||
return num_frames_for_chunks(self.num_chunks)
|
||||
|
||||
@property
|
||||
def video_seconds(self) -> float:
|
||||
return self.num_frames / self.fps
|
||||
|
||||
def session_start_payload(self, model: str) -> dict[str, Any]:
|
||||
"""The ``session.start`` event for ``WS /v1/realtime/video``."""
|
||||
extra_params: dict[str, Any] = {
|
||||
"flow_shift": self.flow_shift,
|
||||
"camera_action_script": self.camera_script,
|
||||
}
|
||||
extra_params.update(self.extra_params)
|
||||
payload: dict[str, Any] = {
|
||||
"type": "session.start",
|
||||
"model": model,
|
||||
"prompt": self.prompt,
|
||||
"image_reference": {"image_url": self.image_reference},
|
||||
"width": self.width,
|
||||
"height": self.height,
|
||||
"num_frames": self.num_frames,
|
||||
"fps": self.fps,
|
||||
"seed": self.seed,
|
||||
"extra_params": extra_params,
|
||||
}
|
||||
if self.negative_prompt is not None:
|
||||
payload["negative_prompt"] = self.negative_prompt
|
||||
return payload
|
||||
|
||||
def describe(self) -> dict[str, Any]:
|
||||
return {
|
||||
"num_chunks": self.num_chunks,
|
||||
"num_frames": self.num_frames,
|
||||
"width": self.width,
|
||||
"height": self.height,
|
||||
"fps": self.fps,
|
||||
"seed": self.seed,
|
||||
"flow_shift": self.flow_shift,
|
||||
"video_seconds": self.video_seconds,
|
||||
"prompt_updates": [
|
||||
{
|
||||
"after_chunk": update.after_chunk,
|
||||
"transition_chunks": update.transition_chunks,
|
||||
}
|
||||
for update in self.prompt_updates
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def default_workload(
|
||||
*,
|
||||
image_reference: str,
|
||||
num_chunks: int,
|
||||
camera_pattern: str = "forward",
|
||||
prompt: str = DEFAULT_PROMPT,
|
||||
width: int = DEFAULT_WIDTH,
|
||||
height: int = DEFAULT_HEIGHT,
|
||||
fps: int = DEFAULT_FPS,
|
||||
seed: int = DEFAULT_SEED,
|
||||
flow_shift: float = DEFAULT_FLOW_SHIFT,
|
||||
negative_prompt: str | None = None,
|
||||
) -> Workload:
|
||||
"""Built-in rollout: a fixed prompt and a generated camera script."""
|
||||
return Workload(
|
||||
prompt=prompt,
|
||||
image_reference=image_reference,
|
||||
camera_script=build_camera_script(num_chunks, camera_pattern),
|
||||
width=width,
|
||||
height=height,
|
||||
fps=fps,
|
||||
seed=seed,
|
||||
flow_shift=flow_shift,
|
||||
negative_prompt=negative_prompt,
|
||||
)
|
||||
|
||||
|
||||
def load_workload(
|
||||
path: str | Path,
|
||||
*,
|
||||
image_resolver: Any = None,
|
||||
overrides: Mapping[str, Any] | None = None,
|
||||
) -> Workload:
|
||||
"""Read a rollout spec from JSON.
|
||||
|
||||
``image_resolver`` turns the spec's ``image`` field into an ``http(s)`` or
|
||||
``data:`` URL; the caller owns that because inlining a local file is I/O.
|
||||
Keys in ``overrides`` that are not ``None`` win over the file, so CLI flags
|
||||
stay meaningful next to ``--workload``.
|
||||
"""
|
||||
document = json.loads(Path(path).read_text())
|
||||
if not isinstance(document, dict):
|
||||
raise ValueError("A workload file must contain a JSON object.")
|
||||
|
||||
merged = dict(document)
|
||||
for key, value in (overrides or {}).items():
|
||||
if value is not None:
|
||||
merged[key] = value
|
||||
|
||||
camera_script = merged.get("camera_action_script")
|
||||
if camera_script is None:
|
||||
num_chunks = merged.get("num_chunks")
|
||||
if num_chunks is None:
|
||||
raise ValueError("A workload file needs either camera_action_script or num_chunks.")
|
||||
camera_script = build_camera_script(int(num_chunks), str(merged.get("camera_pattern", "forward")))
|
||||
|
||||
image = merged.get("image") or merged.get("image_reference")
|
||||
if image is None:
|
||||
raise ValueError("A workload file needs an image or image_reference field.")
|
||||
image_reference = image_resolver(str(image)) if image_resolver is not None else str(image)
|
||||
|
||||
updates_field = merged.get("prompt_updates") or ()
|
||||
prompt_updates = tuple(
|
||||
PromptUpdate(
|
||||
after_chunk=int(entry["after_chunk"]),
|
||||
prompt=str(entry["prompt"]),
|
||||
transition_chunks=int(entry.get("transition_chunks", 3)),
|
||||
)
|
||||
for entry in updates_field
|
||||
)
|
||||
|
||||
extra_params = merged.get("extra_params") or {}
|
||||
if not isinstance(extra_params, Mapping):
|
||||
raise ValueError("Workload extra_params must be a JSON object.")
|
||||
|
||||
return Workload(
|
||||
prompt=str(merged.get("prompt", DEFAULT_PROMPT)),
|
||||
image_reference=image_reference,
|
||||
camera_script=camera_script,
|
||||
width=int(merged.get("width", DEFAULT_WIDTH)),
|
||||
height=int(merged.get("height", DEFAULT_HEIGHT)),
|
||||
fps=int(merged.get("fps", DEFAULT_FPS)),
|
||||
seed=int(merged.get("seed", DEFAULT_SEED)),
|
||||
flow_shift=float(merged.get("flow_shift", DEFAULT_FLOW_SHIFT)),
|
||||
negative_prompt=merged.get("negative_prompt"),
|
||||
prompt_updates=prompt_updates,
|
||||
extra_params=dict(extra_params),
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Metrics
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChunkRecord:
|
||||
"""One media chunk as the client observed it."""
|
||||
|
||||
index: int
|
||||
arrival_s: float
|
||||
"""Seconds from ``session.start`` being sent to this chunk's bytes arriving."""
|
||||
inter_arrival_s: float
|
||||
"""Seconds since the previous chunk; for chunk zero this equals ``arrival_s``."""
|
||||
byte_length: int
|
||||
num_frames: int
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"index": self.index,
|
||||
"arrival_s": self.arrival_s,
|
||||
"inter_arrival_ms": self.inter_arrival_s * 1000.0,
|
||||
"byte_length": self.byte_length,
|
||||
"num_frames": self.num_frames,
|
||||
}
|
||||
|
||||
|
||||
def percentile(values: Sequence[float], fraction: float) -> float:
|
||||
"""Linearly interpolated percentile, so a 10-sample run still reports a p95.
|
||||
|
||||
Implemented here rather than pulled from numpy to keep this module importable
|
||||
in a bare environment; the benchmark's own numbers must not depend on which
|
||||
optional package happens to be installed.
|
||||
"""
|
||||
if not values:
|
||||
raise ValueError("percentile() needs at least one value.")
|
||||
if not 0.0 <= fraction <= 1.0:
|
||||
raise ValueError("percentile fraction must be within [0, 1].")
|
||||
ordered = sorted(values)
|
||||
if len(ordered) == 1:
|
||||
return ordered[0]
|
||||
position = fraction * (len(ordered) - 1)
|
||||
lower = int(position)
|
||||
upper = min(lower + 1, len(ordered) - 1)
|
||||
weight = position - lower
|
||||
return ordered[lower] * (1.0 - weight) + ordered[upper] * weight
|
||||
|
||||
|
||||
def _stddev(values: Sequence[float]) -> float:
|
||||
if len(values) < 2:
|
||||
return 0.0
|
||||
mean = sum(values) / len(values)
|
||||
return (sum((value - mean) ** 2 for value in values) / (len(values) - 1)) ** 0.5
|
||||
|
||||
|
||||
def _latency_summary(values: Sequence[float]) -> dict[str, float]:
|
||||
"""Millisecond summary of a set of inter-arrival intervals."""
|
||||
if not values:
|
||||
return {}
|
||||
milliseconds = [value * 1000.0 for value in values]
|
||||
return {
|
||||
"count": float(len(milliseconds)),
|
||||
"mean_ms": sum(milliseconds) / len(milliseconds),
|
||||
"median_ms": percentile(milliseconds, 0.5),
|
||||
"std_ms": _stddev(milliseconds),
|
||||
"min_ms": min(milliseconds),
|
||||
"p90_ms": percentile(milliseconds, 0.90),
|
||||
"p95_ms": percentile(milliseconds, 0.95),
|
||||
"p99_ms": percentile(milliseconds, 0.99),
|
||||
"max_ms": max(milliseconds),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PlaybackResult:
|
||||
"""What a viewer would have seen, given a start buffer and a playback rate."""
|
||||
|
||||
underrun_count: int
|
||||
total_stall_s: float
|
||||
playback_span_s: float
|
||||
first_underrun_chunk: int | None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"underrun_count": self.underrun_count,
|
||||
"total_stall_ms": self.total_stall_s * 1000.0,
|
||||
"playback_span_s": self.playback_span_s,
|
||||
"stall_ratio": (self.total_stall_s / self.playback_span_s) if self.playback_span_s > 0 else 0.0,
|
||||
"first_underrun_chunk": self.first_underrun_chunk,
|
||||
}
|
||||
|
||||
|
||||
def simulate_playback(
|
||||
records: Sequence[ChunkRecord],
|
||||
*,
|
||||
fps: float,
|
||||
buffer_chunks: int = 1,
|
||||
) -> PlaybackResult:
|
||||
"""Replay arrivals against a wall clock and count stalls.
|
||||
|
||||
Playback starts when ``buffer_chunks`` chunks have arrived and then consumes
|
||||
video at ``fps``. A chunk that has not arrived by the moment the player needs
|
||||
it is an underrun, and the player resumes only when the bytes land -- which
|
||||
is what a viewer experiences, and what a single p99 number cannot tell you:
|
||||
the same p99 is harmless behind a three-chunk buffer and fatal behind none.
|
||||
"""
|
||||
if fps <= 0:
|
||||
raise ValueError("fps must be positive.")
|
||||
if buffer_chunks < 1:
|
||||
raise ValueError("buffer_chunks must be at least 1.")
|
||||
if not records:
|
||||
return PlaybackResult(0, 0.0, 0.0, None)
|
||||
if buffer_chunks > len(records):
|
||||
raise ValueError(f"buffer_chunks={buffer_chunks} exceeds the {len(records)} chunks received.")
|
||||
|
||||
ordered = sorted(records, key=lambda record: record.index)
|
||||
start = ordered[buffer_chunks - 1].arrival_s
|
||||
cursor = start
|
||||
underruns = 0
|
||||
total_stall = 0.0
|
||||
first_underrun: int | None = None
|
||||
|
||||
for record in ordered:
|
||||
if record.arrival_s > cursor:
|
||||
stall = record.arrival_s - cursor
|
||||
underruns += 1
|
||||
total_stall += stall
|
||||
if first_underrun is None:
|
||||
first_underrun = record.index
|
||||
cursor = record.arrival_s
|
||||
cursor += record.num_frames / fps
|
||||
|
||||
return PlaybackResult(
|
||||
underrun_count=underruns,
|
||||
total_stall_s=total_stall,
|
||||
playback_span_s=cursor - start,
|
||||
first_underrun_chunk=first_underrun,
|
||||
)
|
||||
|
||||
|
||||
def steady_chunk_deadline_ms(fps: float) -> float:
|
||||
"""Wall-clock budget a steady chunk must meet to hold real time.
|
||||
|
||||
``fps`` here is the rate the frames are *intended to be played at*, which the
|
||||
checkpoint does not declare: nothing in ``model_index.json``, the scheduler
|
||||
config, or the transformer config carries a frame rate, and the pipeline
|
||||
never reads one. It is a property of how the output is consumed, so both
|
||||
this deadline and real-time factor scale linearly with the choice and the
|
||||
caller has to state it rather than inherit it from a muxing default.
|
||||
"""
|
||||
if fps <= 0:
|
||||
raise ValueError("fps must be positive.")
|
||||
return pixel_frames_in_chunk(1) / fps * 1000.0
|
||||
|
||||
|
||||
def compute_metrics(
|
||||
records: Sequence[ChunkRecord],
|
||||
*,
|
||||
fps: float,
|
||||
warmup_chunks: int = DEFAULT_WARMUP_CHUNKS,
|
||||
slo_ms: float | None = None,
|
||||
playback_buffer_chunks: int = 1,
|
||||
ttfc_s: float | None = None,
|
||||
session_wall_s: float | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Summarize one session.
|
||||
|
||||
``warmup_chunks`` are excluded from the steady-state block because the
|
||||
attention window has not saturated before then (see the module docstring).
|
||||
The terminal chunk also stays out of steady metrics because it skips
|
||||
next-chunk preparation. All chunks stay in the playback simulation, which is
|
||||
what a session really costs end to end.
|
||||
"""
|
||||
if not records:
|
||||
raise ValueError("compute_metrics() needs at least one chunk record.")
|
||||
if warmup_chunks < 0:
|
||||
raise ValueError("warmup_chunks must be non-negative.")
|
||||
|
||||
ordered = sorted(records, key=lambda record: record.index)
|
||||
deadline_ms = slo_ms if slo_ms is not None else steady_chunk_deadline_ms(fps)
|
||||
|
||||
all_intervals = [record.inter_arrival_s for record in ordered[1:]]
|
||||
# The terminal chunk skips next-chunk preparation and is not representative
|
||||
# of sustained generation. Keep it in end-to-end and playback metrics.
|
||||
steady = [record for record in ordered[:-1] if record.index >= warmup_chunks]
|
||||
steady_intervals = [record.inter_arrival_s for record in steady if record.index > 0]
|
||||
|
||||
total_frames = sum(record.num_frames for record in ordered)
|
||||
video_seconds = total_frames / fps
|
||||
wall_seconds = session_wall_s if session_wall_s is not None else ordered[-1].arrival_s
|
||||
|
||||
metrics: dict[str, Any] = {
|
||||
"chunks_received": len(ordered),
|
||||
"warmup_chunks": warmup_chunks,
|
||||
"steady_chunks": len(steady_intervals),
|
||||
"steady_state_reached": bool(steady_intervals),
|
||||
"total_frames": total_frames,
|
||||
"total_bytes": sum(record.byte_length for record in ordered),
|
||||
"video_seconds": video_seconds,
|
||||
"wall_seconds": wall_seconds,
|
||||
# RTF is processing time over content duration, as vLLM-Omni computes it
|
||||
# (`patch.py`: generation_s / video_duration; `compute_audio_rtf`: latency
|
||||
# / audio_duration, "SLO red line < 1"). LOWER IS BETTER and below 1.0 is
|
||||
# real time. RTFX is the reciprocal the repo reports under its own name,
|
||||
# kept here because "fraction of real time achieved" reads better on a
|
||||
# dashboard -- but the two must never be confused for one another.
|
||||
"video_rtf": (wall_seconds / video_seconds) if video_seconds > 0 else 0.0,
|
||||
"video_rtfx": (video_seconds / wall_seconds) if wall_seconds > 0 else 0.0,
|
||||
"frames_per_second": (total_frames / wall_seconds) if wall_seconds > 0 else 0.0,
|
||||
"chunk_deadline_ms": deadline_ms,
|
||||
"interval_all": _latency_summary(all_intervals),
|
||||
"interval_steady": _latency_summary(steady_intervals),
|
||||
}
|
||||
if ttfc_s is not None:
|
||||
metrics["ttfc_ms"] = ttfc_s * 1000.0
|
||||
|
||||
if steady_intervals:
|
||||
steady_frames = sum(record.num_frames for record in steady if record.index > 0)
|
||||
steady_wall = sum(steady_intervals)
|
||||
steady_video = steady_frames / fps
|
||||
metrics["steady_video_rtf"] = (steady_wall / steady_video) if steady_video > 0 else 0.0
|
||||
metrics["steady_video_rtfx"] = (steady_video / steady_wall) if steady_wall > 0 else 0.0
|
||||
within = sum(1 for value in steady_intervals if value * 1000.0 <= deadline_ms)
|
||||
metrics["slo_attainment"] = within / len(steady_intervals)
|
||||
metrics["slo_violations"] = len(steady_intervals) - within
|
||||
|
||||
metrics["playback"] = simulate_playback(
|
||||
ordered,
|
||||
fps=fps,
|
||||
buffer_chunks=playback_buffer_chunks,
|
||||
).to_dict()
|
||||
metrics["playback"]["buffer_chunks"] = playback_buffer_chunks
|
||||
return metrics
|
||||
|
||||
|
||||
def aggregate_metrics(sessions: Iterable[Mapping[str, Any]], *, fps: float) -> dict[str, Any]:
|
||||
"""Combine per-session metrics across sequential repeats.
|
||||
|
||||
Steady intervals are pooled rather than averaging per-session means, so a
|
||||
session that happened to produce fewer steady chunks does not carry the same
|
||||
weight as a full one.
|
||||
"""
|
||||
entries = list(sessions)
|
||||
if not entries:
|
||||
raise ValueError("aggregate_metrics() needs at least one session.")
|
||||
if len(entries) == 1:
|
||||
single = dict(entries[0])
|
||||
single["sessions"] = 1
|
||||
return single
|
||||
|
||||
deadline_ms = entries[0]["chunk_deadline_ms"]
|
||||
if any(entry["chunk_deadline_ms"] != deadline_ms for entry in entries):
|
||||
raise ValueError("Cannot aggregate sessions with different chunk deadlines.")
|
||||
ttfcs = [float(entry["ttfc_ms"]) for entry in entries if entry.get("ttfc_ms") is not None]
|
||||
# Lower is better, so "max" is the worst session, not the best.
|
||||
rtfs = [float(entry["video_rtf"]) for entry in entries]
|
||||
steady_means = [
|
||||
float(entry["interval_steady"]["mean_ms"]) for entry in entries if entry.get("interval_steady", {}).get("count")
|
||||
]
|
||||
steady_count = sum(entry.get("interval_steady", {}).get("count", 0) for entry in entries)
|
||||
steady_total_ms = sum(
|
||||
entry["interval_steady"]["mean_ms"] * entry["interval_steady"]["count"]
|
||||
for entry in entries
|
||||
if entry.get("interval_steady", {}).get("count")
|
||||
)
|
||||
return {
|
||||
"sessions": len(entries),
|
||||
"chunks_received": sum(int(entry["chunks_received"]) for entry in entries),
|
||||
"total_frames": sum(int(entry["total_frames"]) for entry in entries),
|
||||
"video_seconds": sum(float(entry["video_seconds"]) for entry in entries),
|
||||
"wall_seconds": sum(float(entry["wall_seconds"]) for entry in entries),
|
||||
"video_rtf_mean": sum(rtfs) / len(rtfs),
|
||||
"video_rtf_best": min(rtfs),
|
||||
"video_rtf_worst": max(rtfs),
|
||||
"ttfc_ms_mean": (sum(ttfcs) / len(ttfcs)) if ttfcs else None,
|
||||
"ttfc_ms_max": max(ttfcs) if ttfcs else None,
|
||||
"steady_interval_ms_mean": (steady_total_ms / steady_count) if steady_count else None,
|
||||
"steady_interval_ms_spread": (max(steady_means) - min(steady_means)) if steady_means else None,
|
||||
"underrun_count": sum(int(entry["playback"]["underrun_count"]) for entry in entries),
|
||||
"chunk_deadline_ms": deadline_ms,
|
||||
}
|
||||
@@ -1,124 +0,0 @@
|
||||
# MammothModa2 startup benchmark
|
||||
|
||||
Measures `Omni()` initialization and first/subsequent image requests for
|
||||
MammothModa2-Preview. This is a measurement baseline for
|
||||
[#7075](https://github.com/vllm-project/vllm-omni/issues/7075); it changes no runtime code.
|
||||
|
||||
## Run
|
||||
|
||||
From the repository root, with the model downloaded and vLLM-Omni installed:
|
||||
|
||||
```bash
|
||||
set -o pipefail
|
||||
python benchmarks/mammoth_moda2/bench_startup.py \
|
||||
--model /path/MammothModa2-Preview \
|
||||
--deploy-config vllm_omni/deploy/mammoth_moda2.yaml \
|
||||
--height 512 --width 512 --seed 42 \
|
||||
--extra-body '{"text_guidance_scale": 4.0, "cfg_range": [0.0, 1.0], "num_inference_steps": 8}' \
|
||||
--repeat 1 --label baseline --output-json baseline.json --save-image baseline.png \
|
||||
2>&1 | tee baseline.log
|
||||
python benchmarks/mammoth_moda2/parse_startup_log.py baseline.log --require-complete --markdown
|
||||
```
|
||||
|
||||
Use `OMP_NUM_THREADS=16` to set CPU threads and `--parallel-stage-init` to
|
||||
initialize stages concurrently. For first/subsequent requests, use
|
||||
`--height 1024 --width 1024 --repeat 3` and 50 inference steps. Memory polling
|
||||
is opt-in (`--sample-memory`); collect it in separate runs to avoid perturbing
|
||||
the timing comparison. `--save-image` saves only the first request's image.
|
||||
Compare that image across runs; a fixed seed alone does not guarantee equality
|
||||
across execution modes or software versions.
|
||||
|
||||
## Measurement boundaries
|
||||
|
||||
| Metric | Boundary / limitation |
|
||||
| --- | --- |
|
||||
| Imports | Before torch import → benchmark imports complete; excludes interpreter startup |
|
||||
| Time to ready | `Omni()` call → return; excludes client imports and requests |
|
||||
| Time to first image | Benchmark clock start → first `generate()` return, including request setup |
|
||||
| First / subsequent request | Each `generate()` call → return; same prompt and settings |
|
||||
| Stage spawn / device init | Log timestamps: launch → `world_size=` → `Starting to load model`; 1 s resolution |
|
||||
| Weight load | Engine-reported `Loading weights took`; read/cast/copy are merged |
|
||||
| Model setup estimate | Reported model-load time minus weight-load time; not independently instrumented |
|
||||
| Profile + KV + warmup | One merged interval after loading; compilation/capture may be nested within it |
|
||||
| Compile / graph capture | Explicit engine log durations when present; N/A for eager execution |
|
||||
| Memory | Optional 1 s samples: sum of process-tree RSS and maximum device memory used across GPUs reported by `nvidia-smi`; includes other GPU users and can miss transient peaks |
|
||||
|
||||
Stage timelines overlap with parallel initialization; do not sum them to
|
||||
calculate time to ready. Missing log fields stay unmeasured. The parser marks
|
||||
logs without `BENCH_JSON` as incomplete; `--require-complete` makes this an error.
|
||||
|
||||
## Reference results
|
||||
|
||||
A800-SXM4-80GB; Ubuntu 22.04; 16-CPU cgroup quota on a 128-core host;
|
||||
vLLM 0.28.0+cu129, torch 2.13.0+cu129, vLLM-Omni `34aa1d2a`.
|
||||
Checkpoint: 34.5 GiB, 8 shards, mostly fp32, on local NVMe.
|
||||
Committed eager deploy config, GPU memory fractions 0.5 / 0.3.
|
||||
|
||||
The 2026-09-10 rerun used the corrected benchmark: one warm-up process,
|
||||
then three rounds of four configurations with order rotated between rounds.
|
||||
The checkpoint page cache was warmed before the matrix; no cache eviction
|
||||
was requested during it. Each fresh process sent one 512×512, 8-step request
|
||||
(guidance 4.0, seed 42). Memory polling was disabled. All 12 measured runs
|
||||
are retained in [the results](reference_results.json).
|
||||
Values below are `Omni()` seconds, mean ± sample standard deviation, n=3;
|
||||
client imports took approximately another 10 s.
|
||||
|
||||
| CPU threads | Serial init | Parallel init |
|
||||
| --- | --- | --- |
|
||||
| Default (64) | 60.1 ± 0.2 | 41.1 ± 1.0 |
|
||||
| 16 | 56.5 ± 0.9 | 36.7 ± 0.2 |
|
||||
|
||||
On this configuration, parallel initialization reduced time to ready by about
|
||||
32%; combining it with 16 threads reduced it by about 39%. This supports
|
||||
trying these settings on similar deployments, not changing defaults for all
|
||||
models or hardware. Fixed 512×512 samples matched across all 12 measured runs;
|
||||
this is a smoke check, not a broad quality evaluation.
|
||||
|
||||
For context, in the earlier serial baseline AR / DiT spent approximately 11 / 11 s in
|
||||
spawn and imports, 6 / 6 s in device setup, 6 / 4 s loading weights, and
|
||||
5 / 2 s in profile + KV + warmup. About 6 s followed the last stage becoming
|
||||
ready. These coarse log intervals locate work; they are not independent
|
||||
operator timings.
|
||||
|
||||
Separate observations from the earlier experiment (not rerun in this matrix):
|
||||
|
||||
- Requested page-cache eviction: 80.8 s; FlashInfer JIT cache removed:
|
||||
129.1 s (means, n=3 each). These are separate cache conditions, not guaranteed
|
||||
cold-machine startup or isolated disk/JIT costs.
|
||||
- Three 1024×1024, 50-step eager requests in one process took
|
||||
95.2 / 94.9 / 94.6 s. No large first-request penalty was observed in that run.
|
||||
- With AR `enforce_eager: false` and serial init, time to ready was
|
||||
62.5 ± 0.1 s (warm caches, n=3); logged graph capture was 4 s / 0.20 GiB.
|
||||
The model reported torch.compile unsupported. KV capacity fell from about
|
||||
15.7 to 2.0 GiB, and one parallel-init trial failed. The reported non-KV
|
||||
profiling difference is not a measurement of graph allocation. Graph-mode
|
||||
samples differed from eager. This needs separate memory and quality analysis
|
||||
before recommending the configuration.
|
||||
|
||||
## Loading diagnostic and storage scenarios
|
||||
|
||||
```bash
|
||||
OMP_NUM_THREADS=16 python benchmarks/mammoth_moda2/raw_load_bench.py \
|
||||
--model /path/MammothModa2-Preview --shards 6,7,8 --mode cpu_cast
|
||||
# Compare --mode gpu_cast in a fresh process with the same cache conditions.
|
||||
|
||||
MODEL_LOCAL=/local/MammothModa2-Preview OUT_DIR=./startup-bench \
|
||||
bash benchmarks/mammoth_moda2/bench_storage_scenarios.sh
|
||||
# Optional MODEL_NFS=/nfs/MammothModa2-Preview; SCENARIOS="local-cold local-warm".
|
||||
```
|
||||
|
||||
The diagnostic clones each mmap-backed tensor into owned CPU memory before
|
||||
casting. `materialize_s` includes page faults, CPU allocation and copying;
|
||||
`cast_s` measures dtype conversion; `h2d_s` includes GPU allocation and copy.
|
||||
CUDA context/operation warmup is outside the timer. Total also includes file
|
||||
opening and bookkeeping. This extra CPU copy differs from the engine loader,
|
||||
so these numbers do not decompose its loading time or measure disk bandwidth.
|
||||
The revised diagnostic was run three times each for CPU cast with 64 / 16
|
||||
threads and GPU cast with 16 threads; per-run values are in the results file.
|
||||
Both CPU materialization and conversion were faster with 16 threads here; this
|
||||
does not isolate how much of the engine-level improvement comes from either.
|
||||
|
||||
Storage scenarios explicitly read shards before warm runs and request
|
||||
`posix_fadvise(DONTNEED)` before cold runs. Eviction is not guaranteed, and
|
||||
server-side NFS caches are uncontrolled. The script stops on failure and
|
||||
summarizes only logs from the current invocation.
|
||||
Loaded 100 of 1693 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user