Skip to Content
Get Started

Agent Quickstart

Use the Quantiles agent skill to help coding agents run repository-based evaluations with the qt CLI. It supports Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other agents that use reusable skills or instruction files.

The SKILL.md  standardizes local setup, benchmark runs, custom no-code evaluation configuration, custom code evaluation runs, sample-level inspection, evaluation comparisons, resume behavior, and analysis of benchmark and evaluation results. Agents can also find a quantiles.io/llms.txt file that provides a concise, LLM-readable overview of Quantiles and its key usage guidance.

Install the skill

Use the prompt below to set up your coding agent with the Quantiles CLI and agent skill:

Please install the Quantiles skill at github.com/quantiles-evals/skill

Alternatively, copy SKILL.md  into your agent’s skill directory.

Run your first benchmark

After your agent completes the install, have it run its first benchmark using the following prompt:

Run the SimpleQA Verified benchmark and summarize the results.

This prompt uses a demo model which generates random text, does not use any hosted LLM provider and does not incur any inference cost. This demo model run is useful for validating the workflow. Do not treat it as a real model-quality benchmark.

Customize a built-in benchmark

To customize the simpleqa-verified benchmark from above, ask your coding agent to use a hosted LLM provider of your choice and a subset of the samples in the benchmark, all with the following prompt:

Configure the `simpleqa-verified` benchmark in a Quantiles config file to use 10 samples and the <your model here> model, then run the benchmark and summarize the results.

See the Quantiles configuration documentation  for more details.

Configure and run a custom no-code evaluation

For dataset-backed QA checks with exact-match scoring, ask your coding agent to configure and run a custom no-code evaluation instead of writing and maintaining custom evaluation code:

Configure a custom no-code evaluation called <your custom eval name> in `quantiles.toml` using the <your HuggingFace dataset> dataset, the <prompt column> prompt column, the <answer column> golden answer column, and a prompt template file. Run 10 samples and summarize the exact-match accuracy.

See Custom no-code evaluations for the required configuration fields.

Build and run custom code evaluations

For custom evaluations that use your own datasets, models, and specialized measurement techniques, you can build evaluations with the Quantiles Python SDK, while still benefiting from the resilience, efficiency, and observability features built into the Quantiles platform that built-in and custom no-code evaluations use.

To have your coding agent build and run a custom evaluation, customize the below prompt template to your needs:

Write a Quantiles custom code evaluation using the Python SDK that uses the <your dataset> dataset, run samples through the <your model> model, and measures the output using the following metrics:

<list your metrics here>.

Call the evaluation <name>, and make sure to include it in the `quantiles.toml` config file. When you're done, run the new eval and summarize the results.

See custom evaluations documentation for details on how to write custom evaluations with your own code, using the Quantiles SDKs and tooling.

Agent documentation

The following resources provide more detailed guidance on running Quantiles evaluations using coding agents:

Last updated on