vllm-omni/benchmarks/accuracy/text_to_image/README.md

GEBench on vLLM-Omni

This integration adapts the upstream stepfun-ai/GEBench scripts into vllm-omni/benchmarks/accuracy/text_to_image.

Upstream mapping:

• scripts/generate.py -> run_gebench.py generate
• scripts/evaluate.py -> run_gebench.py evaluate
• upstream prompt / judge logic -> gbench.py

What changed:

• Generation calls the local OpenAI-compatible vllm-omni endpoints:
  • /v1/images/generations for text-only frame generation
  • /v1/images/edits for image-conditioned GUI transition generation
• Evaluation still keeps the GEBench scoring dimensions:
  • goal
  • logic
  • cons
  • ui
  • qual
• Judge calls are also routed to a local OpenAI-compatible model served by vllm-omni instead of a remote service.

Expected dataset layout:

• Clone the benchmark dataset from Hugging Face into a local directory:

git clone https://huggingface.co/datasets/stepfun-ai/GEBench /path/to/GEBench

Example usage:

python benchmarks/accuracy/text_to_image/run_gebench.py generate \
  --dataset-root /path/to/GEBench \
  --output-root benchmarks/accuracy/text_to_image/outputs \
  --base-url http://127.0.0.1:8000 \
  --model Tongyi-MAI/Z-Image-Turbo \
  --data-type type3

python benchmarks/accuracy/text_to_image/run_gebench.py evaluate \
  --dataset-root /path/to/GEBench \
  --output-root benchmarks/accuracy/text_to_image/outputs \
  --data-type type3 \
  --judge-base-url http://127.0.0.1:8000 \
  --judge-model Qwen/Qwen2.5-VL-7B-Instruct \
  --judge-api-key EMPTY

python benchmarks/accuracy/text_to_image/run_gebench.py summarize \
  --output-root benchmarks/accuracy/text_to_image/outputs

Example summary output:

{
  "generation": {
    "count": 20,
    "by_type": {
      "type3": {"count": 10},
      "type4": {"count": 10}
    }
  },
  "evaluation": {
    "count": 20,
    "overall_mean": 0.52,
    "by_type": {
      "type3": {"count": 10, "overall_mean": 0.50, "overall_mean_100": 50.0},
      "type4": {"count": 10, "overall_mean": 0.54, "overall_mean_100": 54.0}
    }
  }
}

Example generated images to inspect:

• benchmarks/accuracy/text_to_image/outputs/03_trajectory_text_fictionalapp/<lang_device>/<sample>/frame0.png
• benchmarks/accuracy/text_to_image/outputs/03_trajectory_text_fictionalapp/<lang_device>/<sample>/frame5.png
• benchmarks/accuracy/text_to_image/outputs/04_trajectory_text_realapp/<lang_device>/<sample>/frame0.png
• benchmarks/accuracy/text_to_image/outputs/04_trajectory_text_realapp/<lang_device>/<sample>/frame5.png

What to expect:

• overall_mean is normalized to 0.0 ~ 1.0; higher is better.
• frame0.png is the initial GUI frame; frame5.png is the final trajectory frame most often used for quick inspection.
• For full debugging, inspect the whole frame0.png ... frame5.png sequence for one sample directory.

Notes:

• GEBench upstream leaves type3/type4 generation unfinished. This integration fills that gap with a trajectory runner that generates frame0.png followed by frame1.png ... frame5.png.
• Type1/2/5 require an image-edit capable model exposed through vllm-omni serve.
• summarize will report both generated coverage and any existing evaluation summary files.