How to build Custom No-code AI Evaluations

AI benchmarks and evaluations often vary in their datasets and prompts while using the same underlying execution logic. They load a dataset, render a prompt, call a model, parse the response, and compare it with a known answer. In most cases, maintaining multiple custom implementations of the same evaluation pattern increases complexity and maintenance overhead.

Quantiles' custom no-code evaluations provide a configuration-based way to build dataset-backed evaluations without writing any code yourself. You define the dataset, prompt, model, and scoring behavior in a configuration file, then use the high-performance qt CLI to run the evaluation.

How to build a custom no-code evaluation

A custom no-code evaluation is built and configured entirely in a quantiles.toml configuration file and uses one of two deterministic scoring styles: multiple choice for labeled answer options, or exact match for constrained outputs compared against a golden value.

Every custom no-code evaluation requires four top-level fields:

• type: selects the custom no-code runner and must be set to custom_nocode.
• style: defines how responses are parsed and scored, including the fields required by the selected scoring method.
• dataset: identifies the Hugging Face dataset to evaluate and must include its name.
• prompt_template_file: points to a Jinja template used to render the prompt for each sample.

Below is an example of a configuration file that defines an exact-match custom no-code evaluation:

# Example of a custom no-code evaluation called "my_custom_nocode_eval"[benchmarks.my_custom_nocode_eval]# Type must be set to "custom_nocode" or it will default to "built-in"type = "custom_nocode"# Style sets how responses are parsed and scoredstyle = { type = "exact_match", golden_column = "answer" }# Identifies the Hugging Face dataset to evaluatedataset = { name = "quantiles/my_custom_nocode_eval_data" }# This example uses the built-in demo model.# Specify a provider-backed model (such as OpenAI or Anthropic)# if you want to use a different model. Provider-backed models# require provider-specific credentials.model = "random"# Path to an existing Jinja prompt template.prompt_template_file = "prompts/qa.txt"

After defining the evaluation in the configuration file, run it with the same qt run command used for all other benchmarks and evaluations:

The full configuration reference, including all required and optional fields, is available in the custom no-code evaluations documentation.

Ready-to-use benchmark templates

The custom no-code example configuration on GitHub includes complete quantiles.toml configurations and Jinja prompt templates for SimpleQA Verified, MedQA, MedMCQA, MMLU-Pro, and more. Copy a configuration and its matching prompt template, then adapt the dataset, model, sample limit, or scoring fields for your evaluation.

Below is an example configuration for MMLU-Pro, a broad, general-purpose benchmark used in industry model reporting. The benchmark covers 14 subject areas with up to ten answer options per question.

You can run MMLU-Pro by copying the matching MMLU-Pro prompt template to prompts/mmlu-pro.txt, then add this configuration to quantiles.toml:

# MMLU-Pro as a custom no-code evaluation[benchmarks.mmlu-pro]type = "custom_nocode"dataset = { name = "quantiles/MMLU-Pro", config_name = "default", split = "test" }model = "random"prompt_template_file = "prompts/mmlu-pro.txt"limit = 1000[benchmarks.mmlu-pro.style]type = "multiple_choice"choices = { column = "options" }choice_labels = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]answer = { label_column = "answer" }

This example uses Quantiles' built-in demo model so you can validate the setup without incurring inference costs. Simply run it with qt run mmlu-pro. When you're ready, replace the model value in the configuration with the provider-backed model you want to evaluate. See the model configuration guide for supported providers and model identifiers.

Metrics and results

A custom no-code evaluation records three sample-level metrics for each dataset row:

• is_correct reports whether the parsed response matches the expected answer.
• response_parsed reports whether a multiple-choice response could be mapped to a configured label. exact_match responses are always considered parsed.
• latency_ms records execution latency, including provider latency, in milliseconds.

Quantiles reports correctness and parsing separately, making it easier to distinguish a wrong answer from one that did not follow the expected response format.

Each run persists accuracy along with mean, median, p95, p99, minimum, and maximum latency. Together, the sample-level and aggregate metrics show whether degraded results are associated with incorrect answers, invalid output formats, or slow inference.

Additional metrics for multiple-choice evaluations

Multiple-choice evaluations provide optional class-level metrics, including F1 and a confusion matrix. The F1 family includes one-vs-rest F1 for each label, macro F1, and weighted F1. The confusion matrix records every expected and predicted label pair and adds an unparsed column, so formatting failures are not silently folded into another class. These optional metrics are computed when command output is rendered and are not persisted with the default aggregate metrics.

Custom no-code vs. custom code evaluations

Custom no-code evaluations remove much of the implementation work from evaluation design. Their fixed structure makes it straightforward to run a dataset through a prompt and model, then score the response with a specific method. Not every evaluation fits that structure. Evaluations involving agents, complex retrieval pipelines, tool calls, custom metrics, judge rubrics, or specialized scoring logic need the flexibility of a custom code evaluation. The table below provides a side-by-side comparison to help you choose between them.

Build custom no-code evaluations with agents

A coding agent, such as Codex or Claude Code, can handle most of the work required to create a custom no-code evaluation. Given a dataset and prompt template, it can inspect the available fields, configure the appropriate exact-match or multiple-choice scorer, and run the evaluation. The Quantiles agent skill provides the workflow and safeguards needed to complete those steps reliably.

Install the Quantiles agent skill, then choose the exact-match or multiple-choice prompt that matches the evaluation's scoring style. The example below is an exact-match prompt with placeholders for the dataset and evaluation settings:

Documentation reference

• Custom no-code evaluation guide: Configure datasets, prompt templates, models, scoring styles, and metrics without writing evaluation code.
• Custom no-code examples on GitHub: Browse complete quantiles.toml configurations and prompt templates for different dataset schemas.
• Quantiles evaluation quickstart: Install the CLI, run a sample benchmark, and inspect the resulting run record.

Keep reading

View all

Quantiles: Open-Source, Local-First Infrastructure for AI Evaluation Workflows

ProductJune 30, 2026

Evaluating Emergency Recognition in Patient-Facing AI Systems

EvaluationsApril 23, 2026

Recovery, Escalation, and Workflow Reliability in Clinical AI Agents

EvaluationsApril 16, 2026