Evaluation Quickstart

Quantiles lets teams run benchmarks and evaluations from the command line, inspect sample-level outputs, compare runs against baselines, and define custom evaluations with minimal setup.

Install the CLI

To install the Quantiles CLI, run the following command:

curl -fsSL https://cli.quantiles.io/install.sh | bash

Because Quantiles is local by default, your project directory holds the database for run history, metadata, samples, and metrics, all under .quantiles/.

Run a sample benchmark

Run a simpleqa-verified sample benchmark with the built-in demo model to validate the installation and inspect how Quantiles runs evaluations and records inputs, outputs, and results.

qt run simpleqa-verified

The SimpleQA-verified benchmark tests 1,000 short-form factuality prompts for testing parametric knowledge. See the arXiv paper  for details.

The above command uses a demo model which generates random text and does not use external model API calls, and doesn’t incur usage charges. Runs using the demo model are only useful for validating the workflow. See Built-in Benchmarks for more details.

Inspect the run

The first Quantiles run in a new workspace will have a run_id of 1. Use the run_id to inspect the complete run record, including workflow inputs, outputs, sample-level results, metrics, and execution metadata, in JSON format.

qt show 1 --json

You can omit the --json flag to see a human-readable summary of the run.

Use qt list to view run history and find a run_id to analyze

Run a benchmark against your own model

Use --input to run a built-in benchmark, such as SimpleQA Verified, with a model other than the default demo model. See Model Configuration for supported providers and model identifiers.

First, set the credential required by the provider. This example uses OpenAI:

export OPENAI_API_KEY="<your_openai_api_key>"

Then pass the model directly to the evaluation:

qt run simpleqa-verified --input '{"model":"openai:gpt-5.6-luna"}'

This model override applies only to the current evaluation run. To reuse a model across evaluations, configure it in a quantiles.toml configuration file.

Create a custom no-code evaluation

Custom no-code evaluations let you evaluate AI systems on datasets and tasks specific to your use case without writing or maintaining custom evaluation code. Define the evaluation in a quantiles.toml or .quantiles.toml configuration file, then run, analyze, and compare it using the standard Quantiles workflow.

Quantiles currently supports the following custom no-code evaluation styles. See the example for each style for configuration instructions.

• Exact match: an evaluation that compares the model’s response with an expected answer.
• Multiple choice: an evaluation that extracts and scores the model’s selection from a configured set of answer choices.

Create a Custom Code Evaluation

A custom code evaluation is a Python program to measure AI behavior that is highly specific to your product, workflow, prompt, dataset, rubric, or release process, and that behavior cannot be captured by a custom no-code evaluation or an existing built-in benchmark.

See Custom Evaluations for a complete walkthrough.

Compare Evaluation Runs

Use the qt CLI to compare and analyze the differences between two eval runs:

qt compare <run_id_a> <run_id_b>

See Compare Evals for full documentation on run comparison.

Use Coding Agents to Run Evaluation Workflows

For an agent-assisted workflow, install the Quantiles skill and use your preferred coding agent to run evaluations. The skill works with agents such as Codex, Claude, and others.

See the following guides to get started with coding-agent workflows in Quantiles.

• Agent Quickstart

  Use coding agents to create, run, inspect, compare, and analyze evals

• Agent Overview

  Learn how agents fit into the Quantiles evaluation workflow

Useful Commands

See the CLI Reference for full command details and output examples.