vllm-omni/recipes/JD/JoyAI-Image-Edit.md

# JoyAI-Image-Edit

> Text-guided single-image editing with the native JoyAI-Image-Edit diffusion pipeline

## Summary

- Vendor: JD / JoyImage Team
- Model: `jdopensource/JoyAI-Image-Edit-Diffusers`
- Task: Text-guided single-image editing
- Mode: Offline inference and OpenAI-compatible online serving
- Maintainer: Community

## When to use this recipe

Use this recipe when you want to run JoyAI-Image-Edit through the shared
image-to-image examples with a known-good baseline configuration. JoyAI-Image-Edit
is a large single-image editing model: it requires one reference image per
request, snaps requested sizes to Joy's supported resolution buckets, and has a
validated 1024x1024 baseline on an 80 GB NVIDIA GPU.

## References

- Upstream model:
  [`jdopensource/JoyAI-Image-Edit-Diffusers`](https://huggingface.co/jdopensource/JoyAI-Image-Edit-Diffusers)
- Upstream project:
  [`jd-opensource/JoyAI-Image`](https://github.com/jd-opensource/JoyAI-Image)
- Related offline example:
  [`examples/offline_inference/image_to_image/image_edit.py`](../../examples/offline_inference/image_to_image/image_edit.py)
- Related online example:
  [`examples/online_serving/image_to_image/README.md`](../../examples/online_serving/image_to_image/README.md)
- User guide:
  [`docs/user_guide/examples/offline_inference/image_to_image.md`](../../docs/user_guide/examples/offline_inference/image_to_image.md),
  [`docs/user_guide/examples/online_serving/image_to_image.md`](../../docs/user_guide/examples/online_serving/image_to_image.md)

## Hardware Support

This recipe documents the CUDA configuration used to validate the PR's JoyAI
integration. Other GPU types should be treated as unvalidated until you rerun
the smoke and parity checks below.

## GPU

### 1x NVIDIA H800 PCIe 80GB

#### Environment

- OS: Linux
- Python: `3.12.3`
- Driver / runtime: NVIDIA driver `580.82.07`, CUDA runtime `13.0`
- PyTorch: `2.11.0+cu130`
- Diffusers: `0.39.0.dev0` with `JoyImageEditPipeline` available
- vLLM: `0.21.0`
- vLLM-Omni: use the commit you are deploying from
- GPU: NVIDIA H800 PCIe, 81559 MiB

#### Online Serving

Start the OpenAI-compatible server from the repository root:

```bash
CUDA_VISIBLE_DEVICES=0 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm serve jdopensource/JoyAI-Image-Edit-Diffusers \
  --omni \
  --port 8092 \
  --init-timeout 1200
```

The server is ready when the log shows `Application startup complete.`

If you need extra GPU headroom, use one of the memory-saving variants below.
These variants are useful on tighter 80 GB deployments, but the latency and
quality numbers in this recipe are from the baseline command above.

```bash
# Pipeline-level CPU offload for large components.
CUDA_VISIBLE_DEVICES=0 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm serve jdopensource/JoyAI-Image-Edit-Diffusers \
  --omni \
  --port 8092 \
  --init-timeout 1200 \
  --enable-cpu-offload
```

```bash
# DiT blockwise offload. This trades latency for lower peak GPU memory.
CUDA_VISIBLE_DEVICES=0 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
vllm serve jdopensource/JoyAI-Image-Edit-Diffusers \
  --omni \
  --port 8092 \
  --init-timeout 1200 \
  --enable-layerwise-offload
```

#### Online Verification

Download the public JoyAI plate example:

```bash
wget -O /tmp/joy_plate.jpg \
  https://raw.githubusercontent.com/jd-opensource/JoyAI-Image/main/test_images/test_1.jpg
```

Send one image and one prompt. JoyAI-Image-Edit supports exactly one input image
per request.

```bash
IMG_B64=$(base64 -w0 /tmp/joy_plate.jpg)

curl -s http://localhost:8092/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg img "$IMG_B64" '{
    messages: [{
      role: "user",
      content: [
        {type: "text", text: "Turn the plate blue"},
        {type: "image_url", image_url: {url: ("data:image/jpeg;base64," + $img)}}
      ]
    }],
    extra_body: {
      height: 1024,
      width: 1024,
      num_inference_steps: 30,
      guidance_scale: 4.0,
      seed: 0
    }
  }')" \
  | jq -r '.choices[0].message.content[0].image_url.url' \
  | cut -d',' -f2- \
  | base64 -d > /tmp/joyai_image_edit_plate_online.png
```

Expected result: `/tmp/joyai_image_edit_plate_online.png` is a 1024x1024 PNG
where the plate color changes to blue while the surrounding scene is mostly
preserved. Use a client timeout of at least 600 seconds for first-run or cold
server checks.

#### Offline Command

Run the same smoke case without starting a server:

```bash
CUDA_VISIBLE_DEVICES=0 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python examples/offline_inference/image_to_image/image_edit.py \
  --model jdopensource/JoyAI-Image-Edit-Diffusers \
  --image /tmp/joy_plate.jpg \
  --prompt "Turn the plate blue" \
  --negative-prompt "" \
  --height 1024 \
  --width 1024 \
  --num-inference-steps 30 \
  --guidance-scale 4.0 \
  --seed 0 \
  --enforce-eager \
  --init-timeout 1200 \
  --output /tmp/joyai_image_edit_plate_offline.png
```

For a local checkpoint mirror, replace `--model` with the local model directory.
The PR validation used the same command shape with
`/autodl-fs/data/joyai_models/JoyAI-Image-Edit-Diffusers`.

#### Validated Results

The table below is from the 1x H800 PCIe environment above. Resolution requests
were snapped to the Joy 1024x1024 bucket, batch size was 1, and CFG was 4.0.
`vLLM peak` is the observed peak GPU memory for the vLLM-Omni run.

| Case | Steps | Seed | vLLM latency | Diffusers latency | vLLM peak | Diffusers peak | SSIM | PSNR |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `plate_seed123` | 30 | 123 | 50.87 s | 52.74 s | 57.52 GiB | 57.28 GiB | 0.9869 | 34.7051 |
| `plate_seed0` | 30 | 0 | 51.02 s | 52.01 s | 57.52 GiB | 57.28 GiB | 0.9946 | 41.4351 |
| `astronaut_seed0` | 40 | 0 | 67.90 s | 69.22 s | 57.52 GiB | 57.28 GiB | 0.9948 | 38.2498 |
| `crane_removal_seed0` | 40 | 0 | 68.00 s | 69.27 s | 57.52 GiB | 57.28 GiB | 0.9975 | 50.0135 |
| `object_move_seed0` | 40 | 0 | 68.06 s | 69.32 s | 57.52 GiB | 57.28 GiB | 0.9969 | 46.1138 |
| `object_rotation_seed0` | 40 | 0 | 67.94 s | 69.28 s | 57.52 GiB | 57.28 GiB | 0.9898 | 35.5365 |
| `camera_control_seed0` | 40 | 0 | 68.15 s | 69.51 s | 57.52 GiB | 57.28 GiB | 0.9952 | 41.7697 |

#### Output Size Behavior

JoyAI-Image-Edit does not use arbitrary image sizes directly. The pipeline maps
the input or requested `height` / `width` to the nearest supported Joy bucket,
then resizes and center-crops the reference image to that bucket before
denoising.

- Provide both `height` and `width`, or omit both.
- For square outputs, request `height=1024` and `width=1024`.
- A request such as `512x512` is also square and is snapped to the 1024x1024 Joy
  bucket, so the final image size will not match the literal request.
- Non-square requests are snapped to the nearest aspect-ratio bucket supported
  by the model.

#### Guidance Parameters

Use `guidance_scale` for JoyAI classifier-free guidance, matching Diffusers.
The pipeline defaults to `4.0` when the parameter is omitted. `true_cfg_scale`
remains supported as an alias; if both parameters are supplied, their values
must match. Values greater than `1.0` enable CFG; an omitted negative prompt is
treated as an empty string. Values at or below `1.0` disable CFG.

The offline example passes `--guidance-scale 4.0` explicitly for clarity; if it
is omitted, the JoyAI pipeline uses the same `4.0` default.

#### Known Limitations

- Single image only: JoyAI-Image-Edit rejects requests with zero or multiple
  input images.
- Single prompt only: batched prompt requests are rejected.
- HSDP is not supported yet. Starting with HSDP enabled raises a `ValueError`
  before the generic HSDP setup reaches the transformer.
- Tensor Parallelism is not supported yet. Keep `tensor_parallel_size=1`;
  larger values raise a `ValueError` during pipeline initialization.
- LoRA is not included for this JoyAI pipeline.
- Cache-DiT and TeaCache are not included for this JoyAI pipeline. Do not treat
  the generic image-to-image cache flags as validated JoyAI acceleration knobs.
- Text encoder quantization is not included.
- The validation above is CUDA-only; ROCm and NPU configurations are not
  documented as supported by this recipe.