Evals
A small, reusable set of test tasks with known good answers that you run an AI setup against, so you can tell whether it works and whether a change made it better or worse.
Also called: evaluations · test cases · benchmarks · AI testing · golden examples
Connections
Understand first
Ideas this one builds on.
Related
Often used together or easily confused.
Skills that use it
Blueprints that practise it
Resources that teach it
- How to Improve Interview Feedback Consistency with a Custom GPT (garden)
- An Executive's Guide to Implementing AI (garden)
- An AI glossary (garden)
- Beyond vibe checks: A PM's complete guide to evals (garden)
- Giving your AI a Job Interview (garden)
- Making AI Work: Leadership, Lab, and Crowd (garden)
- How to be more ambitious: A tutorial for humans with models (garden)
- Evaluating AI for high-stakes work (garden)
Competency
Find evals on the commons map ↗
You may know it from
- operations quality control and acceptance testing Check samples against a standard before the batch ships, and after any change to the process.
- teaching a grading rubric Writing down what good looks like before marking makes judgments consistent.
In plain words
An eval is everyday quality control for AI. If you rely on an AI setup repeatedly, such as a saved prompt, a skill, a custom assistant or an automation, collect five to ten real examples where you know what a good result looks like. Each time you change the setup, or the tool updates its model, run those examples again and compare.
Without a test set, you judge by whatever result you saw last. With one, you can say “the new prompt fixed the tone problem but started missing deadlines in two cases out of eight.”
Good test cases include the tricky ones: the email with no clear ask, the document with a buried exception, the request that should be refused. Write down what “good” means for each before you look at any AI output.
Example
A support team uses an AI assistant to draft replies. They keep eight real (anonymised) customer emails with ideal replies written by their best agent. After editing the assistant’s instructions, they rerun all eight and catch a new habit of promising refunds it shouldn’t.
How it connects
- Verification, made repeatable and comparable.
- Needed for skills, custom assistants and automation that others rely on.
- The best way to settle model choice for your own work.
Try it
Pick a prompt you reuse. Write down three test inputs and, for each, one sentence on what a good answer must include. Run the prompt on all three and score them yourself.
Read this card as plain Markdown: /concepts/con-evals.md. Know a parallel from your own field, or a product that implements this? Add it.