Skip to Content
Core Concepts

The Quantiles configuration file

Quantiles uses a single quantiles.toml or .quantiles.toml file in the current working directory to configure built-in benchmarks and define custom evaluations. The file specifies how evaluations are loaded and executed, including datasets, models, prompts, scoring methods, inputs, and runtime settings.

Only one of the two filenames can exist in the same directory. If both exist, the CLI will exit with an error.

When to use a configuration file

Create or configure a quantiles.toml or .quantiles.toml configuration file when you want to do any of the following:

Note: Built-in benchmarks can run without a configuration file using their default settings. For example, qt run simpleqa-verified runs the full SimpleQA Verified benchmark with the demo model.

File name and location

The qt CLI looks for either quantiles.toml or .quantiles.toml in the current working directory. If neither exists, it searches ancestor directories and uses the first matching file it finds. To configure Quantiles for your project, we recommend adding one of these files to the project’s working directory.

File structure

Every evaluation definition or built-in override lives under its own [benchmarks.<eval_name>] section. The section key is the evaluation name passed to qt run <eval_name>.

For example, if you want to override default parameters for the built-in PubMedQA benchmark, add the following section to your configuration file:

# Configure the sample limit and model for the built-in PubMedQA benchmark.
 
[benchmarks.pubmedqa]
samples = 50
 
# Use OpenAI's GPT 5.6 Luna model instead of the built-in
# demo model. This requires an OPENAI_API_KEY environment
# variable, and OpenAI will charge you for usage.
model = "openai:gpt-5.6-luna"

Configuring built-in and custom evaluations

Use a configuration file to create and/or configure the following evaluation types. See the documentation for each type for supported fields and examples.

Configure a model

Built-in benchmarks and custom no-code evaluations accept an optional model field, which can be set in the configuration file or supplied at runtime through --input.

The following example configures a custom evaluation named my-eval to use an OpenAI model:

# Example configuration for running my-eval with an OpenAI model
[benchmarks.my-eval]
 
# The value before the first colon identifies the provider.
# The remaining value is the provider-specific model identifier.
model = "openai:gpt-5.6-luna"

If model is not specified, Quantiles uses the evaluation’s built-in demo model.

See Model Configuration for supported providers and prefixes, credential environment variables, and troubleshooting.

CLI --input overrides

To apply a one-time configuration override, pass --input to qt run. These values are recorded in the local run history but are not written to the configuration file or applied to subsequent runs. For example, the following command overrides the model for one evaluation run:

qt run my-eval --input '{"model":"openai:gpt-5.6"}'

Warning: —input overrides config input for keys: model

In --json mode, the warning is included in the JSON output under the warning key.

Configuration validation

The CLI validates benchmark configuration sections before execution:

  • built-in sections:

    • May not contain command or input fields.
    • May include built-in fields such as samples, model, and max_workers.

  • custom_code sections:

    • Must have a non-empty command array.
    • May not contain built-in-only fields such as samples or model.

  • custom_nocode sections:

    • Must include dataset, style, and prompt_template_file.
    • prompt_template_file must point to an existing file.
    • Must set style.type to either exact_match or multiple_choice.
    • When style.type = "exact_match", a style.golden_column field must be present.
    • When style.type = "multiple_choice", a valid choice source, answer source, and a non-empty list of unique choice_labels must be present in the style dictionary.
    • May not contain command, input, or other unsupported fields.

Validation failures produce clear error messages before any run is created.

How qt resume uses the configuration file

When you run qt resume <run_id>, the CLI reuses the same configuration from that run, including input, from the database. It also re-reads the command from the config file. This setup means the following:

  • You do not need to re-submit input when resuming a previous run.
  • If you update the command between qt run and qt resume, the resumed run uses the updated command, but retains the input values used in the original run.
  • If a custom_code or custom_nocode config section is removed after a qt run, resuming that run will fail with a clear error.

See Restart and Resume Runs for the full recovery workflow.

Last updated on