evaly init

Interactive setup wizard for API keys, config file, and optional demo benchmark.

evaly init [OPTIONS]

The init command walks you through setting up Evalytic in a new project. It collects API keys, creates an evalytic.toml config file, and optionally runs a quick demo benchmark to verify everything works.

Fastest way to get started. Run evaly init instead of manually creating config files and setting environment variables.

Options

Flag	Type	Description
--force	FLAG	Overwrite existing `evalytic.toml` without asking.
--skip-demo	FLAG	Skip the optional demo benchmark at the end.

Wizard Steps

Step 1: Use case selection

The wizard asks what you'll use Evalytic for, to tailor API key requirements and next-step suggestions:

$ evaly init

  Welcome to Evalytic!
  This wizard will set up your configuration.

  What will you use Evalytic for?
    1. Compare text-to-image models (Flux, SDXL, Recraft...)
    2. Compare image-editing models (Kontext, Seedream...)
    3. Score existing images (no generation)
    4. CI/CD quality gate

  Enter number [1]: 1

Options 1 and 2 (model comparison) require a fal.ai API key for image generation. Options 3 and 4 only need a judge key (Gemini).

Step 2: API key collection

The wizard checks your environment first. Keys already set via .env or environment variables are detected automatically:

  API Keys
  Google Gemini API key: AIza***wxyz (from environment, keeping)

  fal.ai API key
  Get one at: https://fal.ai/dashboard/keys
  Enter key: fal_xxxxx
  Valid!

Step 3: Config file generation

An evalytic.toml is created in your current directory:

# evalytic.toml -- generated by `evalytic init`

[keys]
fal = "fal_xxxxx"

[bench]
judge = "gemini-2.5-flash"

Keys already present in the environment are not written to the file to avoid duplication.

Step 4: Optional demo benchmark

If you have both API keys, the wizard offers to run a quick benchmark to verify your setup:

  Run a quick demo benchmark (~$0.003)? [Y/n] y

  Generating... flux-schnell: 1/1
  Scoring... gemini-2.5-flash: 2/2

    ┌────────────────┬─────────────────┬──────────────────┬─────────┐
    │ Model          │ visual_quality  │ prompt_adherence │ Overall │
    ├────────────────┼─────────────────┼──────────────────┼─────────┤
    │ flux-schnell   │ 4.5             │ 4.0              │ 4.3     │
    └────────────────┴─────────────────┴──────────────────┴─────────┘

Use --skip-demo to skip this step in non-interactive environments.

Step 5: Next steps

The wizard shows tailored next steps based on your use case:

  Next Steps
    evaly bench -m flux-schnell -m flux-dev -p "A cat" -y
    evaly bench -m flux-schnell --prompts prompts.json -y

    evaly --help        Full command list
    https://docs.evalytic.ai   Documentation

Examples

Quick setup (interactive)

evaly init

Overwrite existing config

evaly init --force

CI/CD: non-interactive setup

# Set keys via env, then create config without prompts
export FAL_KEY=fal_xxx
export GEMINI_API_KEY=AIxxx
evaly init --force --skip-demo

evaly dataset evaly config