Custom no-code evaluations

Custom no-code evaluations let you evaluate AI systems on datasets and tasks specific to your use case without writing or maintaining custom evaluation code. Define the evaluation in a quantiles.toml or .quantiles.toml configuration file, then run, analyze, and compare it using the standard Quantiles workflow.

For highly specialized evaluations such as multi-step workflows, LLM judge-based evaluations, or agent evaluations, use a custom code evaluation instead.

Custom no-code evaluation styles

Custom no-code evaluations have two styles, defined by the style.type field:

• exact_match: an evaluation that compares the model’s response with an expected answer.
• multiple_choice: an evaluation that extracts and scores the model’s selection from a configured set of answer choices.

The following required and optional fields apply to both exact-match and multiple-choice evaluations:

The built-in demo model (model = "random") is for workflow validation only. It generates random responses for exact_match evaluations and selects uniformly from style.choice_labels for multiple_choice evaluations. Do not use its results to assess model quality.

Exact match

Use the exact_match evaluation style when each dataset row has a single expected answer that should match the model response exactly. Quantiles scores the model response based on whether it matches that answer exactly.

In addition to the fields above, this evaluation style requires:

The style.golden_column field identifies the dataset column containing the expected responses. Each value in that column must be a string, number, or boolean, which Quantiles converts to text. Before comparison, Quantiles removes leading and trailing whitespace from both the expected response and the model response. All other text is compared case-sensitively.

Example

The following example configures an exact-match evaluation in quantiles.toml. It assumes that each dataset row contains a prompt column with the model input and an answer column with the expected response.

1. Define the evaluation in quantiles.toml

# Example of an exact-match custom no-code evaluation.
[benchmarks.exact_match_nocode_eval]
 
# Type must be set to "custom_nocode" or the section defaults to "builtin".
type = "custom_nocode"
 
# Style sets how responses are parsed and scored.
style = { type = "exact_match", golden_column = "answer" }
 
# Identifies the Hugging Face dataset to evaluate.
dataset = { name = "your-org/your-dataset" }
 
# This example uses the built-in demo model. Replace it with a
# provider-prefixed model (e.g., OpenAI or Anthropic) for real inference.
# Provider-backed models require provider-specific credentials and may
# incur usage charges.
model = "random"
 
# Path to an existing Jinja prompt template.
prompt_template_file = "prompts/exact-match.txt"
 
# Limit the number of samples to be evaluated. Omit this if you
# want to run all samples in that evaluation.
limit = 100

2. Create the Jinja prompt template

Create prompts/exact-match.txt relative to the directory containing quantiles.toml or .quantiles.toml:

{{ row.prompt }}

Dataset fields are available through row. Use dot notation, such as {{ row.prompt }}, for simple field names. Use bracket notation, such as {{ row["Prompt Text"] }}, for field names containing spaces or other characters that do not work cleanly with dot notation.

3. Run the evaluation

Run the configured evaluation by name:

qt run exact_match_nocode_eval

After the evaluation finishes, Quantiles prints a run_id that can be used to inspect aggregate metrics and sample-level results:

qt show <run_id> --json

Multiple choice

Use the multiple_choice evaluation style when each dataset row contains a set of answer choices and identifies the correct choice. Quantiles scores the model response based on the choice it selects.

In addition to the fields above, configure the following required and optional fields:

Example

The following example configures a multiple-choice evaluation in quantiles.toml. It assumes that each dataset row contains a question column with the prompt, an options column with the answer choices, and an answer_label column with the correct choice label. Each answer_label value must match one of the configured style.choice_labels.

1. Define the evaluation in quantiles.toml

# Example of a multiple-choice custom no-code evaluation.
[benchmarks.multiple_choice_nocode_eval]
 
# Type must be set to "custom_nocode" or the section defaults to "builtin".
type = "custom_nocode"
 
# Identifies the Hugging Face dataset to evaluate.
dataset = { name = "your-org/your-multiple-choice-dataset" }
 
# This example uses the built-in demo model. Replace it with a
# provider-prefixed model (e.g., OpenAI or Anthropic) for real inference.
# Provider-backed models require provider-specific credentials and may
# incur usage charges.
model = "random"
 
# Path to an existing Jinja prompt template.
prompt_template_file = "prompts/multiple-choice.txt"
 
# Limit the number of samples to be evaluated. Omit this if you
# want to run all samples in that evaluation.
limit = 10
 
# Style sets how responses are parsed and scored.
[benchmarks.multiple_choice_nocode_eval.style]
type = "multiple_choice"
 
# Read the answer choices from the "options" dataset column.
choices = { column = "options" }
 
# Assign these labels to the choices in order.
choice_labels = ["A", "B", "C", "D"]
 
# Read the expected label from the "answer_label" dataset column.
answer = { label_column = "answer_label" }

2. Create the Jinja prompt template

Create prompts/multiple-choice.txt relative to the directory containing quantiles.toml or .quantiles.toml:

{{ row.question }}
 
{% for choice in choices %}
{{ choice.label }}. {{ choice.text }}
{% endfor %}
Answer with only the letter of the correct choice.

Reminder: Dataset fields are available through row. Use dot notation, such as {{ row.question }}, for simple field names. Use bracket notation, such as {{ row["Question Text"] }}, for field names containing spaces or other characters that do not work cleanly with dot notation. Multiple-choice templates also receive the normalized answer choices as choices.

Explicitly ask the model to return one of the labels in style.choice_labels. Quantiles first compares the complete trimmed response with each configured label, case-insensitively. If the complete response does not match, Quantiles checks up to the last eight whitespace-separated tokens, ignoring non-alphanumeric punctuation around each token. If no configured label is found, the response is recorded as unparsed.

3. Run the evaluation

Run the configured evaluation by name:

qt run multiple_choice_nocode_eval

For more prompt template examples, see the custom no-code prompt templates  in the Quantiles GitHub repository.

See the sample custom_nocode configurations  for examples of common benchmarks implemented through configuration.

Dataset

Choose or publish a Hugging Face  dataset whose rows contain the fields needed to render your prompt and score the model’s response. As described above, the evaluation style determines which additional fields are required:

• style = { type = "exact_match", ... } evaluations need a golden-answer column
• style = { type = "multiple_choice", ... } evaluations need fields describing the choices and the correct answer

The Quantiles datasets on Hugging Face  include the datasets referenced by the custom no-code examples  on GitHub.

Loading a Hugging Face dataset may require network access, such as when downloading uncached data.

Metrics and results

For each dataset row processed by a custom no-code evaluation, Quantiles calculates and records the following sample-level metrics:

• is_correct indicates whether the parsed response matches the expected answer.
• response_parsed indicates whether Quantiles could parse the response. Exact-match responses are always considered parsed; for multiple-choice evaluations, this value is 0 when the response cannot be parsed as a configured choice label.
• latency_ms records the sample’s execution latency, including any LLM provider latency, in milliseconds.

Each run persists the following aggregate metrics:

• accuracy - the fraction of evaluated examples assigned the correct class label.
• mean_latency_ms, median_latency_ms, p95_latency_ms, p99_latency_ms, min_latency_ms, and max_latency_ms - statistics describing the distribution of latency_ms values for each sample in the run

Optional metrics for multiple_choice evaluations

Multiple-choice evaluations can compute additional metrics from their recorded step outputs. Configure the metrics array in the benchmark table alongside fields such as dataset, model, and prompt_template_file, rather than the nested [benchmarks.<eval_name>.style] table. These metrics are computed when command output is rendered and are not persisted with the default aggregate metrics.

F1

The following example adds F1 metrics to the evaluation:

[benchmarks.multiple_choice_nocode_eval]
# Other benchmark-level fields...
metrics = ["f1"]
 
[benchmarks.multiple_choice_nocode_eval.style]
# Multiple-choice style fields...

The F1 metric family provides:

• f1_label_N: one-vs-rest F1 for the label at index N in style.choice_labels.
• macro_f1: the arithmetic mean of the per-label F1 values.
• weighted_f1: the average of the per-label F1 values weighted by each label’s number of expected samples.

F1 is reported as 0 when its denominator is zero.

Confusion Matrix

To enable a confusion matrix, use the following value in the same benchmark-level location:

metrics = ["confusion"]

The confusion-matrix metric family provides:

• confusion_matrix_G_P: the number of samples whose expected label has index G and whose parsed prediction has index P.
• confusion_matrix_G_unparsed: the number of samples whose expected label has index G but whose response could not be parsed as a configured label.

In these metric names, N, G, and P are zero-based positions in style.choice_labels.

Expected labels form the matrix rows. Parsed labels and an additional unparsed bucket form the columns.

Metric display formats

F1 and confusion matrix values are shown only in JSON output by default. To include a metric family in both human-readable and JSON output, set show = "all" in the same benchmark-level location, for example:

metrics = [{ name = "confusion", show = "all" }]

Runtime overrides

Use --input to apply one-time overrides, such as model to use a different AI model or limit to evaluate a smaller number of samples, without modifying quantiles.toml. All other settings will continue to come from the configuration file, and subsequent runs will use just the saved configuration unchanged. This is useful for smoke tests, model comparisons, and temporary prompt experiments. For example:

qt run <eval_name> --input '{"model":"<provider>:<model>","limit":<count>}'

--input overrides apply only to the current run, or when that run is resumed. The next time you run the same evaluation without --input, Quantiles uses the values defined in the configuration file.

Validation and common failures

Quantiles validates the configuration and reports row-specific errors when it cannot prepare or score the dataset. Use the following fixes for common failures:

Use coding agents to create custom no-code evaluations

Use the Quantiles agent skill to have a coding agent build, run, and analyze custom no-code evaluations with the qt CLI. Before proceeding, install the skill. Then customize one of the prompts below based on the custom no-code evaluation style you want to create.

Exact-match agent prompt

Create and run a custom no-code exact-match evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <answer_column> as the golden-answer column, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.

Multiple-choice agent prompt

Create and run a custom no-code multiple-choice evaluation named <evaluation_name> using the Hugging Face dataset <dataset_id>. Use <prompt_column> as the prompt column, <choice_source> as the choice source, <choice_labels> as the choice labels, <answer_source> as the correct-answer source, and the following Jinja prompt template: <jinja_prompt_template>. Inspect the run and summarize the results.

See Agents Overview for more detail on using agents with Quantiles.