diff --git a/recipes/README.md b/recipes/README.md index 8ac877567..6b80f8a03 100644 --- a/recipes/README.md +++ b/recipes/README.md @@ -42,6 +42,7 @@ recipes/ | -------- | ------ | ---------- | | [`Baidu/ERNIE-Image.md`](./Baidu/ERNIE-Image.md) | Text-to-image online serving (ERNIE-Image 8B) | 1x or 2x RTX 4090 24GB | | [`Bagel/BAGEL-7B-MoT.md`](./Bagel/BAGEL-7B-MoT.md) | Text-to-image with shared online/offline examples | 1x A100 80GB / 2x CUDA GPUs | +| [`black-forest-labs/FLUX.1-schnell.md`](./black-forest-labs/FLUX.1-schnell.md) | Offline text-to-image generation with CPU offload | 1x NVIDIA GeForce RTX 5090 32GB | | [`Boogu/Boogu-Image.md`](./Boogu/Boogu-Image.md) | Text-to-image online serving (Boogu-Image-0.1-Base) | 1x A100/H100 40GB+ | | [`BosonAI/Higgs-Audio-V3-TTS.md`](./BosonAI/Higgs-Audio-V3-TTS.md) | Online + offline multilingual TTS with voice cloning | 1x H100 80GB | | [`ByteDance/Lance.md`](./ByteDance/Lance.md) | Unified AR+diffusion: text/img/video gen + understanding (Lance 3B) | 1x B300 / A100 80GB | diff --git a/recipes/black-forest-labs/FLUX.1-schnell.md b/recipes/black-forest-labs/FLUX.1-schnell.md new file mode 100644 index 000000000..9ae6f4817 --- /dev/null +++ b/recipes/black-forest-labs/FLUX.1-schnell.md @@ -0,0 +1,160 @@ +# FLUX.1-schnell + +> Offline text-to-image generation on one NVIDIA GeForce RTX 5090 32 GB + +## Summary + +- Vendor: Black Forest Labs +- Model: `black-forest-labs/FLUX.1-schnell` +- Task: Text-to-image generation +- Mode: Offline inference with the shared text-to-image runner +- Maintainer: Community + +## When to use this recipe + +Use this recipe as a personally validated starting point for generating a +1024x1024 image with FLUX.1-schnell on one NVIDIA GeForce RTX 5090. The tested +configuration uses BF16, four denoising steps and model-level CPU offload. + +This recipe covers single-image generation. It does not attempt to optimize +throughput with quantization or cache acceleration. + +## References + +- Model card: +- Canonical offline text-to-image guide: + [`docs/user_guide/examples/offline_inference/text_to_image.md`](../../docs/user_guide/examples/offline_inference/text_to_image.md) +- Shared runnable example: + [`examples/offline_inference/text_to_image`](../../examples/offline_inference/text_to_image) +- CPU offload guide: + [`docs/user_guide/diffusion/cpu_offload.md`](../../docs/user_guide/diffusion/cpu_offload.md) +- Community recipe tracker: + [#2645](https://github.com/vllm-project/vllm-omni/issues/2645) + +## Hardware Support + +## GPU + +### 1x NVIDIA GeForce RTX 5090 32 GB + +This configuration was personally validated end to end on September 9, 2026. + +#### Environment + +- OS: Ubuntu 22.04.3 x86_64 +- Python: 3.12.7 +- GPU: 1x NVIDIA GeForce RTX 5090, 32,607 MiB reported by `nvidia-smi` +- Host RAM: approximately 62 GiB (64 GB class) +- NVIDIA driver: 595.71.05; CUDA 13.2 reported by `nvidia-smi` +- PyTorch: 2.13.0+cu130; PyTorch CUDA runtime: 13.0 +- vLLM: 0.28.0 +- vLLM-Omni: `0.28.1.dev112+ga31690041` +- Diffusers: 0.40.0 +- Transformers: 5.14.1 + +#### Command + +Run the shared example from the vLLM-Omni repository root in an environment +with vLLM-Omni installed: + +```bash +python examples/offline_inference/text_to_image/text_to_image.py \ + --model black-forest-labs/FLUX.1-schnell \ + --prompt 'A red fox standing in fresh snow, cinematic photography' \ + --height 1024 --width 1024 \ + --num-inference-steps 4 --guidance-scale 0 --seed 42 \ + --enable-cpu-offload \ + --output flux-schnell-rtx5090.png +``` + +The command uses the public model ID for portability. The measured runs used +already downloaded FLUX.1-schnell weights; initial Hub download and +authentication were not part of the inference validation. + +#### Verification + +Verify that the runner succeeded and inspect the saved image: + +```bash +python - <<'PY' +import hashlib +from pathlib import Path + +from PIL import Image, ImageStat + +path = Path("flux-schnell-rtx5090.png") +with Image.open(path) as image: + image.load() + assert image.format == "PNG" + assert image.size == (1024, 1024) + rgb = image.convert("RGB") + stats = ImageStat.Stat(rgb) + assert max(stats.stddev) > 1, "Image is nearly uniform" + print(f"format={image.format}") + print(f"size={image.width}x{image.height}") + print(f"mode={image.mode}") + print(f"bytes={path.stat().st_size}") + print("rgb_mean=" + ",".join(f"{value:.3f}" for value in stats.mean)) + print(f"rgb_extrema={rgb.getextrema()}") + +print(f"sha256={hashlib.sha256(path.read_bytes()).hexdigest()}") +PY +``` + +The repeated experiment completed 36 image requests (6 additional warmups and +30 measured outputs) with successful process exits. Every PNG passed decoding, +1024x1024 dimension and non-uniform pixel checks. SHA-256 hashes were recorded +for the artifacts; matching those hashes is not required across environments. +These checks establish file integrity, not a quantitative image-quality score. + +#### Performance + +The experiment used three independent processes. Each process initialized the +model once, generated two warmup images, then generated ten measured images +sequentially. All 30 measured images used BF16, CPU offload, 1024x1024, four +steps, guidance scale 0 and TP=1, without concurrent requests. + +| Generation measurement | Result | +| --- | ---: | +| Measured images | 30 | +| Mean latency | 25.8411 seconds | +| P95 latency (nearest-rank) | 26.1446 seconds | + +| Startup / resource measurement | Maximum | +| --- | ---: | +| Model initialization (seconds) | 64.73 | +| Full-lifetime RAM PSS peak (MiB) | 49306 | +| Full-lifetime device GPU memory peak (MiB) | 28121 | +| Measured PyTorch reserved GPU memory peak (MiB) | 27360 | + +Latency measures only `Omni.generate()`, excluding image saving and +verification. The two warmup images per process are excluded. P95 uses the +nearest-rank method over the 30 measured images. Model initialization includes +the engine's built-in dummy warmup; weights were already downloaded and OS file +caches were not cleared. + +RAM and whole-device GPU memory were sampled throughout each process lifetime +at a target interval of 200 ms. RAM is the maximum simultaneous sum of PSS for +the inference process and its workers, read from Linux `smaps_rollup`. +Whole-device GPU memory is the maximum reported by `nvidia-smi`. Both are +sampled peaks and may miss shorter spikes. + +The PyTorch value is the maximum worker-reported `peak_memory_mb` across the +30 measured requests. Despite the field name, its unit is MiB; it measures the +PyTorch allocator's reserved-memory high-water mark rather than whole-device +usage. + +#### Notes + +- `--enable-cpu-offload` swaps the transformer and text encoders between host + and device memory at phase boundaries. It reduces GPU residency but adds + transfer overhead and requires sufficient host RAM. The minimum host RAM + requirement was not measured. +- CPU offload does not imply CPU-only computation. The tested model-level + offload backend keeps the VAE on the GPU. +- Only offline single-image generation on one GPU with BF16, 1024x1024, + four steps and CPU offload was validated. Online serving, FP8, other + resolutions, concurrency, execution without offload and smaller host RAM + configurations are outside this recipe's validated scope. +- The upstream revision of the downloaded weights was not recorded, so the + measurements do not establish exact checkpoint reproducibility.