Built-in Benchmarks

Built-in benchmarks are ready-to-run evals with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark.

Built-in benchmarks use the demo model by default to validate the installation and inspect evaluation workflow, execution steps, recorded inputs and outputs, scoring behavior, and reported metrics without calling an external model API or incurring usage charges.

Use the following command pattern to run a built-in benchmark:

qt run <benchmark>

Use the Custom no-code evaluations documentation to learn how to create a fully custom evaluation from a dataset without writing or maintaining Python code.

Available Built-in Benchmarks

Quantiles includes built-in benchmarks to make common evaluations easier to run. These benchmarks are open source and commonly used across industry evaluation workflows.

When given a built-in benchmark, the qt run command does the following:

1. Selects the benchmark dataset and loads its examples into the local run.
2. Executes the benchmark with the configured sampler.
3. Scores each example with the benchmark’s evaluation metric.
4. Saves per-example inputs, outputs, and step records in the run history.
5. Emits aggregate metrics that you can inspect with qt show and can later compare against other runs with qt compare.

Customize built-in benchmarks

All built-in benchmarks can be customized with a configuration file. Configure a built-in benchmark under its [benchmarks.<benchmark_name>] section in quantiles.toml or .quantiles.toml. The section supports the following fields:

Example

The following example limits the number of samples evaluated and specifies the provider and model used for inference (such as OpenAI or Anthropic) for the built-in benchmark PubmedQA. When qt run pubmedqa is run, the CLI will detect and apply the following customizations.

# This config block customizes the built-in PubMedQA benchmark.
 
# You could customize other built-in benchmarks, such as
# simpleqa-verified, with a similar block.
[benchmarks.pubmedqa]
 
# `type` defaults to "builtin", so you can omit this line
# if you want.
type = "builtin"
 
# Restrict the benchmark to run only the first 50 of the
# 1000 samples in the PubMedQA benchmark
samples = 50
 
# Use OpenAI's GPT 5.6 model instead of the built-in
# demo model. This requires an OPENAI_API_KEY environment
# variable, and OpenAI will charge you for usage.
model = "openai:gpt-5.6"

--input flag

Use --input to apply one-time overrides to a single benchmark run, such as a sample limit or model. Pass the overrides as a JSON object on the command line:

# Limit the number of samples run
qt run <benchmark> --input '{"samples":10}'
 
# Use a provider-backed model instead of the demo model
qt run <benchmark> --input '{"model":"openai:gpt-5.6"}'

These values apply only to the current qt run invocation. To reuse settings across runs, define samples, model, or other supported fields in the benchmark’s quantiles.toml configuration.

Run a smoke test

We recommend using the --input flag to run a small sample count (e.g., 10) to verify that any benchmark or evaluation runs end to end with your selected model before committing time or inference cost to a full evaluation.

#Limit the number to 10 samples for a PubMed QA benchmark run
qt run pubmedqa --input '{"limit":10}'

Inspect the benchmark run

Inspect the full run record, including inputs, outputs, sample level results, and metadata:

qt show <run_id>

Use qt list to view run history and find a run_id to pass to qt show

Compare Benchmark Runs

Use the qt CLI to compare and analyze the differences between two benchmark runs:

qt compare <run_id_a> <run_id_b>

See Compare Evals for full documentation on run comparison.

Resume Interrupted Benchmark Runs

Quantiles evaluation and benchmark workflows are designed to recover from interruptions (e.g., rate limits, timeouts, interrupted processes). If a benchmark is interrupted after completing some steps, you can resume the run using the run_id:

qt resume <run_id>

See Resume Runs for the full recovery workflow.

Add a Built-in Benchmark

If there is an open-source benchmark you would like to add as a built-in benchmark, file an issue in the Quantiles GitHub repo .

Helpful requests include the benchmark name, source dataset or repository, license and any reference implementation.