evaly bench

One-command model benchmarking: generate, score, and report.

evaly bench [OPTIONS]

The bench command is Evalytic's primary workflow. It generates images from prompts using fal.ai models, scores them with a VLM judge, and outputs rich reports. Everything in a single command.

Basic Usage

# Quickest start: uses demo model + demo prompt
evaly bench -y

# Single model, inline prompt
evaly bench -m flux-schnell -p "A cat on a windowsill" --yes

# Multiple models, prompts from file
evaly bench -m flux-schnell -m flux-dev -m flux-pro -p prompts.json

# With HTML report and browser review
evaly bench -m flux-schnell flux-dev -p prompts.json -o report.html --review

# Disable auto-enabled metrics
evaly bench -m flux-schnell -p "Hello World" --no-metrics

Options

Running evaly bench --help groups options into Essential and Advanced sections. Most workflows only need the essential ones.

Essential Options

Advanced Options

Smart Defaults

To minimize the barrier to your first benchmark, bench can run with almost no arguments:

No models specified

If you omit --models in an interactive terminal, Evalytic asks if you want a demo run with flux-schnell. You can also set default models in evalytic.toml:

# evalytic.toml
[bench]
models = ["flux-schnell", "flux-dev"]

No prompts specified

If you omit --prompts (and --inputs/--images), a demo prompt is used automatically: "A cat sitting on a windowsill at sunset". You can also set default prompts in evalytic.toml:

# evalytic.toml
[bench]
prompts = "prompts.json"

Prompts Format

Prompts can be provided as an inline string or as a JSON file:

// prompts.json — array of strings
[
  "A photorealistic cat on a windowsill at sunset",
  "A modern minimalist logo for 'ACME Corp'",
  "Product photo: white sneakers on marble"
]

Dataset Mode

Use --dataset to run a benchmark from a dataset file instead of plain prompts. Datasets can include metadata and expected scores for regression detection. The --dataset flag cannot be combined with --prompts or --inputs.

# Run bench from a dataset file
evaly bench -m flux-schnell --dataset golden.json -y

Add --check-expected to compare actual scores against the expected values defined in the dataset. Dimensions that fall below expected are flagged in the terminal output:

# Check results against expected scores
evaly bench -m flux-schnell --dataset golden.json --check-expected -y

Model Registry

Run evaly bench --list-models to see all registered models. Common ones:

You can also pass full fal.ai endpoint paths directly:

evaly bench -m fal-ai/flux-realism -p "A portrait photo"

Output Formats

Terminal (always)

Rich-formatted table with per-model, per-dimension scores. Suppressed with --no-terminal.

JSON

Machine-readable report with all scores, metadata, and cost breakdown. Used for CI/CD integration with evaly gate.

evaly bench -m flux-schnell -p "A cat" -o report.json --yes

HTML

Interactive report with image grids, score charts, and judge reasoning.

evaly bench -m flux-schnell flux-dev -p prompts.json -o report.html --review

Output Directory

Use --output-dir to write all outputs to a timestamped subfolder. Each run creates a directory like bench-abc123_20260227-143000/ containing report.html, report.json, and errors.log (only when errors occurred).

evaly bench -m flux-schnell flux-dev -p prompts.json --output-dir ./reports -y

# Creates: reports/bench-abc123_20260227-143000/
#   report.html
#   report.json
#   errors.log    (only if errors occurred)

--output-dir works alongside --output — both can be used in the same command. You can also set a default in evalytic.toml:

# evalytic.toml
[bench]
output_dir = "./reports"

Error Log

When generation or scoring errors occur (API timeouts, content policy violations, judge parse failures), they are collected into a structured errors.log file inside the output directory. The terminal also shows a brief error summary at the end of each run.

Examples

Compare 3 models with JSON + HTML output

evaly bench \
    -m flux-schnell -m flux-dev -m flux-pro \
    -p prompts.json \
    -o report.json -o report.html \
    --review

Text rendering evaluation

evaly bench \
    -m flux-pro \
    -p "A sign that says 'OPEN 24/7'" \
    -d text_rendering -d visual_quality

Reproducible benchmark with seed

evaly bench \
    -m flux-dev \
    -p prompts.json \
    --seed 42 \
    --image-size landscape_16_9

Image-to-image benchmark

# Compare img2img models with face identity metric
evaly bench \
    -m flux-kontext -m seedream-edit -m reve-edit \
    -i inputs.json \
    -d identity_preservation \
    --metrics face \
    -o report.html --review

Use GPT-5.2 as judge instead of Gemini

evaly bench \
    -m flux-schnell \
    -p "A sunset over the ocean" \
    -j openai/gpt-5.2

Local judge with Ollama

evaly bench \
    -m flux-schnell \
    -p "A cat" \
    -j ollama/qwen2.5-vl:7b

Consensus mode (multi-judge)

Use --judges with 2-3 comma-separated judge names for more reliable scores via adaptive consensus:

# 2 judges — average when they agree, flag disputes
evaly bench \
    -m flux-schnell \
    -p "A cat" \
    --judges "gemini-2.5-flash,gpt-5.2"

# 3 judges — disputed dimensions get a tiebreaker (median)
evaly bench \
    -m flux-schnell flux-dev \
    -p prompts.json \
    --judges "gemini-2.5-flash,gpt-5.2,claude-haiku-4-5"