[Doc] Add Cosmos3-Nano ROCm recipe (1x MI350X) (#5634)

Signed-off-by: zjli2013 <leezhengjiang@126.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
zhengjiaandCursor authored and GitHub committed 2026-08-05 22:50:15 +08:00
1 parent 9f978923f2
commit b4581b293b
1 file changed
+150
+150
View File
@@ -423,6 +423,156 @@ ffprobe -v error -show_entries stream=codec_type,nb_frames,width,height cosmos3_
once in `vllm_omni/model_extras/cosmos3.py` and forwarded through `--extra-body`;
unknown keys for the model are dropped.
## ROCm
### 1x MI350X (Online serving)
#### Environment
- OS: Ubuntu 22.04+
- Python: 3.12+
- Driver / runtime: ROCm, HIP 7.2, **gfx942 or gfx950** (validated on gfx950 / MI350X,
which exposes 252 GiB of HBM to PyTorch)
- PyTorch: 2.11.0 (ROCm build)
- vLLM version: match the repository requirements from your current checkout
- vLLM-Omni version or commit: use the commit you are deploying from
No ROCm-specific patch or flag is needed: Cosmos3 runs on the stock code path.
#### Command
The NVIDIA command works unchanged. Guardrails need the gated
`nvidia/Cosmos-1.0-Guardrail` repo, so the quick path is to disable them (you are
then responsible for license compliance):
```bash
vllm serve nvidia/Cosmos3-Nano \
--omni \
--no-guardrails \
--host 0.0.0.0 --port 8000 \
--init-timeout 1800
```
For 720p video, add `--vae-use-tiling`: it cuts peak VRAM by ~68% for ~13%
latency, which is the difference between needing a 120 GiB card and a 40 GiB one.
`--quantization fp8` and `--enable-layerwise-offload` are also supported; see the
Notes for the measured cost of each. For extra GPUs use `--ulysses-degree N` or
`--tensor-parallel-size N` (not validated on ROCm yet).
#### Verification
Use the same `curl` requests as the CUDA section above. The ROCm images do not
ship `ffprobe`, so the CUDA section's `ffprobe` line does not work here. `imageio`
and `pyav` are already installed and report the same thing:
```bash
python -c "from PIL import Image; im=Image.open('cosmos3_t2i.png'); print(im.size, im.mode)"
python -c "import imageio.v3 as iio; print(iio.improps('cosmos3_t2v.mp4', plugin='pyav').shape)"
```
Expect `(1024, 1024) RGB` and `(189, 720, 1280, 3)` for the requests above.
Re-running the same request with the same seed reproduces a byte-identical mp4,
so an md5 comparison across two runs also works as a smoke check.
#### Notes
- **Warm up once per output shape before timing anything.** The first generation
of a shape pays a one-time cost that is cached on disk and reused by later
processes in the same container: a ~250 s aiter attention-kernel JIT build plus
a ~390 s build of the VAE decode path *for that shape*, so warming 1024² images
does nothing for 189-frame video. At 720p / 189 frames / 35 steps this is
**540 s cold vs 161 s warm, with byte-identical output**.
- **Measured on 1x MI350X (bf16, guardrails off, warm):** T2I 1024² @ 50 steps
**~2.7 s**; T2V 1280×720 / 189 frames @ 35 steps **~161 s**, of which ~92% is
the DiT denoise loop and ~4% VAE decode, so optimization effort belongs in the
denoise loop. The optional flags act on different terms of the memory bill and
are therefore complementary rather than redundant:
| Flag | Latency | Peak reserved | Peak allocated | Acts on |
|---|---|---|---|---|
| *(none)* | 161 s | 120 GiB | 95 GiB | — |
| `--vae-use-tiling` | 183 s (+13.5%) | **38 GiB** (−68%) | **36 GiB** (−62%) | decode activations |
| `--enable-layerwise-offload` | 162 s (+0.8%) | 84 GiB (−30%) | 69 GiB (−27%) | weights |
| `--quantization fp8` | 150 s (−6.6%) | 107 GiB (−11%) | 82 GiB (−14%) | weights (online, no calibration) |
Reserved is the caching-allocator high water mark and runs ~2–25 GiB above
allocated; allocated is the figure to compare against another platform.
- **`--vae-use-tiling` is by far the biggest lever for video.** Of the 95 GiB
allocated by default only ~35 GiB is weights and fixed overhead; the remaining
**60 GiB is activations from decoding all 189 frames in one piece**, which is
why fp8 and offload cannot touch it and why tiling, which leaves under 5 GiB of
them, wins so much. `--vae-use-slicing` is a no-op for a single video request:
peak memory is bit-identical to the default.
- **`--enable-layerwise-offload` is close to free on this part:** per-block DiT
compute is long enough to hide the weight transfer, so on a card this size it
buys headroom for concurrency or higher resolution rather than the
speed-for-memory trade it is on small cards.
- **Attention backend:** the numbers above are aiter FlashAttention, not SDPA.
ROCm picks aiter when it is installed and the encoded device capability
satisfies `90 < major * 10 + minor < 100` (gfx942 and gfx950); otherwise it
falls back to `TORCH_SDPA` and says so only at debug level, which is easy to
miss. If you are reporting numbers, assert the choice instead of trusting the
default: `RocmOmniPlatform.get_diffusion_attn_backend_cls(None, 128)` returns
the resolved path.
- **Determinism is per-configuration:** same seed and same flags reproduce a
byte-identical mp4 across processes, but different configurations do not match
each other — including offload, which nominally only relocates weights, so do
not use it as a bit-exact regression baseline. Quality was not evaluated for
any configuration.
### 1x MI350X (Offline generation)
#### Environment
Same as the online serving section above.
#### Command
The standard task examples run unchanged:
```bash
# Text-to-image
python examples/offline_inference/text_to_image/text_to_image.py \
--model nvidia/Cosmos3-Nano \
--prompt "A photorealistic red sports car at golden hour, cinematic lighting." \
--negative-prompt "blurry, distorted, low quality" \
--height 1024 --width 1024 --num-inference-steps 50 --guidance-scale 7.0 \
--extra-body '{"flow_shift": 3.0, "guardrails": false}' \
--output cosmos3_t2i.png
# Text-to-video
python examples/offline_inference/text_to_video/text_to_video.py \
--model nvidia/Cosmos3-Nano \
--prompt "A robot arm is cleaning a plate in the kitchen." \
--negative-prompt "blurry, distorted, low quality" \
--height 720 --width 1280 --num-frames 189 --fps 24 \
--num-inference-steps 35 --guidance-scale 6.0 --flow-shift 10.0 \
--extra-body '{"max_sequence_length": 4096, "guardrails": false,
"use_resolution_template": false, "use_duration_template": false}' \
--output cosmos3_t2v.mp4
```
Both scripts print `Total generation time` and `Worker peak GPU memory
(reserved)`, which is where the latency and reserved figures in the Notes above
come from. Add `--vae-use-tiling`, `--quantization fp8` or
`--enable-layerwise-offload` to reproduce the other rows. The allocated column
needs `VLLM_LOGGING_LEVEL=DEBUG`, which enables a per-request line reporting
reserved, allocated and pool overhead together.
#### Verification
Same as the online serving section above.
#### Notes
- Run each command twice and read the second run; the first pays the one-time
compilation described above. If you run in a container, keep the container
alive between runs (`docker run --rm` throws the JIT cache away and you pay
the ~250 s kernel build every time).
- Latency and memory figures are identical to the online serving section — they
were taken from this path, because a single process has no client/server
timing ambiguity.
## NPU
### 1x Ascend 910B / 910C (Atlas A2 / A3) — Online serving