Score your models and agents on what your business needs.
Metrics decide how every answer is scored in the Evaluations module of Bud AI Foundry. Start from 64 built-in metrics in 12 categories, create your own without code, and test models and agents against your own data and requirements.
From the metric library to a custom metric, in one walkthrough.
Browse the built-in metrics, create and calibrate a custom one, then use it in a new evaluation.
A test is only as useful as the way you score it.
Public evaluation datasets are a useful start, but they rarely reflect your business or how your agents will be used. Before anything reaches production, you need to know how it performs on your own data and requirements, and whether it is safe, with evidence. Metrics let you define what good looks like, test against it, and keep evaluating as models and agents meet real-world data.
64 metrics, across everything your AI does.
The built-in library spans 12 categories, from answer quality and retrieval to agents, safety and business behaviour. The highlights are below, and every metric is listed in the catalogue. Custom metrics your team adds sit beside them and are shared across the workspace, so once someone defines what good looks like, every evaluation can use it.
Answer quality · 4
Answer relevancy, hallucination, whether the response follows the prompt's instructions, and how well a summary matches its source.
Retrieval (RAG) · 9
Check whether a RAG system found the right information and actually used it: contextual precision, recall and relevancy, faithfulness and citation faithfulness, single-turn or per turn.
Agents · 14
Most agent failures don't show in the final answer. Tool use, MCP, and planning and execution metrics read the agent's full trace to catch the wrong tool, a step repeated in a loop, or ten actions where three would do.
Business behaviour · 13
Did it hand over to a human when it should have? Make a promise it wasn't authorised to make, or mention a competitor? Include the disclosure your industry requires? The classifiers measure this on every test case, instead of spot checks by hand.
Safety · 6
Bias, toxicity, PII leakage, misuse outside the model's declared domain, advice in restricted areas such as finance, medicine or law, and responses that break the assigned role.
Rule-based, no judge · 10
Most metrics use an AI judge. Ten are fully rule-based, such as exact match, pattern match, BLEU and ROUGE, or confirming that an agent used only the tools it was allowed to. They give the same result every time, so they are easy to audit.
All 64 built-in metrics.
Grouped into 12 categories. Open a category to see each metric, whether it scores one response or a whole conversation, and what it measures. The 10 metrics marked “no judge” are rule-based and give the same result every time.
Answer quality4
| Metric | Scope | What it measures |
|---|---|---|
| Answer Relevancy | Single turn | How relevant the answer is to the question |
| Hallucination | Single turn | Content in the answer that the context doesn’t support |
| Prompt Alignment | Single turn | Whether the response follows the prompt’s instructions |
| Summarization | Single turn | How well a summary matches its source document |
Retrieval (RAG)9
| Metric | Scope | What it measures |
|---|---|---|
| Citation Faithfulness | Single turn | Whether the retrieved passages back up the answer’s citations |
| Contextual Precision | Single turn | How much of the retrieved context is relevant |
| Contextual Recall | Single turn | How much of the needed information was retrieved |
| Contextual Relevancy | Single turn | How relevant the context is to the question |
| Faithfulness | Single turn | Whether the answer is factually consistent with the context |
| Turn Contextual Precision | Multi-turn | Ranking of relevant context, scored per turn |
| Turn Contextual Recall | Multi-turn | How much of the expected outcome the context covers, per turn |
| Turn Contextual Relevancy | Multi-turn | Relevance of the context, per turn |
| Turn Faithfulness | Multi-turn | Factual consistency with the context, per turn |
Tool use4
| Metric | Scope | What it measures |
|---|---|---|
| Argument Correctness | Single turn | Whether each tool call’s arguments fit the input |
| Tool Correctness | Single turn · no judge | Whether the expected tools were called with the expected arguments |
| Tool Permission | Single turn · no judge | Whether every tool called is allowed and none is denied |
| Tool Use | Multi-turn | Whether the agent picked the right tools and arguments |
MCP3
| Metric | Scope | What it measures |
|---|---|---|
| MCP Task Completion | Multi-turn | Whether MCP-backed agents complete the task |
| MCP Use | Single turn | Whether MCP tools and resources are used appropriately |
| Multi-Turn MCP Use | Multi-turn | MCP use across a whole conversation |
Planning & execution7
| Metric | Scope | What it measures |
|---|---|---|
| Agent Loop Detection | Single turn · no judge | Repeated calls, stalled reasoning or cycles in the agent’s trace |
| Goal Accuracy | Multi-turn | Whether the agent reached the goal, and how well it planned |
| Plan Adherence | Single turn | Whether the agent followed its own plan |
| Plan Quality | Single turn | Whether the plan suited the task |
| Step Efficiency | Single turn | Whether the agent avoided unnecessary steps |
| Task Completion | Single turn | Whether the task was completed end to end |
| Topic Adherence | Multi-turn | Whether the conversation stayed on the relevant topics |
Conversation4
| Metric | Scope | What it measures |
|---|---|---|
| Conversation Completeness | Multi-turn | Whether every user request was fulfilled |
| Knowledge Retention | Multi-turn | Whether facts from earlier in the conversation are remembered and reused |
| Role Adherence | Multi-turn | Whether the chatbot stayed within its assigned role |
| Turn Relevancy | Multi-turn | Answer relevancy, scored per turn |
Safety6
| Metric | Scope | What it measures |
|---|---|---|
| Bias | Single turn | Biased content in the answer |
| Misuse | Single turn | Use outside the model’s declared domain (lower is better) |
| Non-Advice | Single turn | Advice given in restricted areas such as finance, medicine or law (lower is better) |
| PII Leakage | Single turn | Personal data leaked in the response (lower is better) |
| Role Violation | Single turn | Responses that break the assigned role (lower is better) |
| Toxicity | Single turn | Toxic content in the answer |
Format3
| Metric | Scope | What it measures |
|---|---|---|
| Exact Match | Single turn · no judge | Whether the answer is exactly the same string as the expected output |
| JSON Correctness | Single turn | Whether the JSON is valid against the expected schema |
| Pattern Match | Single turn · no judge | Whether the answer matches a regular expression |
Text similarity5
| Metric | Scope | What it measures |
|---|---|---|
| Accepted Answer Match | Single turn · no judge | Normalised exact match against any of the accepted answers |
| BLEU | Single turn · no judge | Sentence BLEU score against the expected output |
| ROUGE-1 | Single turn · no judge | Single-word (unigram) overlap F1 |
| ROUGE-2 | Single turn · no judge | Word-pair (bigram) overlap F1 |
| ROUGE-L | Single turn · no judge | Longest common subsequence F1 |
Classifiers13
| Metric | Scope | What it measures |
|---|---|---|
| Abstention | Single & multi-turn | Whether the assistant abstained, answered or made something up |
| Clarification | Single & multi-turn | Whether it asked for clarification, answered directly or guessed |
| Data Leakage | Single & multi-turn | Leaks of the system prompt, personal data or secrets |
| Escalation | Single & multi-turn | Whether it handed off to a human, offered to, or handled it alone |
| Forbidden Commitments | Single & multi-turn | Unauthorised commitments, mentions of competitors, or disparagement |
| Instruction Completeness | Single & multi-turn | Whether it followed all instructions, some, or none |
| Prompt Injection | Single & multi-turn | Whether it resisted, partly followed or followed an injected instruction |
| Refusal | Single & multi-turn | Whether it complied, refused or partly refused |
| Required Disclosure | Single & multi-turn | Whether the required disclosures were made |
| Resolution | Single & multi-turn | Whether the issue was resolved, left unresolved or handed over |
| Response Language | Single & multi-turn | Whether the answer is in the user’s language |
| Scope Adherence | Single & multi-turn | Whether it stayed in scope, deflected, or answered an out-of-scope request |
| Tone Adherence | Single & multi-turn | Whether the required tone was kept |
Row rules1
| Metric | Scope | What it measures |
|---|---|---|
| Row rules | Single & multi-turn | Each row’s own “Must” and “Must not” conditions, scored by the share met |
Images5
| Metric | Scope | What it measures |
|---|---|---|
| Image Coherence | Single turn | Whether the image is coherent with the input text and instructions |
| Image Editing | Single turn | Quality of an image edit against its instruction |
| Image Helpfulness | Single turn | Whether a returned image helps answer the query |
| Image Reference | Single turn | Whether the response refers correctly to what’s in the input image |
| Text-to-Image | Single turn | How faithfully a generated image matches its text prompt |
See how every metric works, and make it yours.
No metric is a black box, and none has to be used exactly as shipped.
Open any metric
See what it measures, which data it needs and how it is set up, including the pass mark, the judge model and whether it explains its scores.
Duplicate and calibrate
If a built-in metric is close but not quite right, duplicate it as your own and calibrate it against your team's judgement, so its scores reflect how your experts would grade.
Five steps, no code.
Describe what a good answer looks like, decide how the judge should reach its score, and set the line between pass and fail.
-
01
Choose how to score
Score the output with a rule, a decision tree or a label.
-
02
Choose the scope
A single-turn metric scores one input and its response. A multi-turn metric evaluates an entire conversation.
-
03
Describe a good answer
Select the fields the judge should consider, name the metric and describe what a good answer looks like.
-
04
Define the judging
Let the judging model decide from your description, or give it a fixed set of steps it must follow.
-
05
Set the threshold
Set the score that separates a pass from a fail, and save the metric to your library. Calibrate it with sample data against your expected results.
To use it, open the Evaluations hub and click New Evaluation. Give the evaluation a name and description, select its scope, upload the dataset you want to test against, then add metrics from your library to assess your model or agent.
Where Evaluation Metrics fits.
Metrics are part of the Evaluations module in Bud AI Foundry, where experiments and evaluations test models and agents side by side.
Product
Bud AI Foundry
Where models and agents are evaluated
Inference, guardrails, observability, evaluations and cost control on one serving plane, with agents built and governed in the same place.
Explore Bud AI FoundryLibrary
Product demos
More walkthroughs of Bud AI Foundry
Videos of each module and feature, including the Evaluations module and the experiments that run models and agents side by side.
Browse the demosEvaluate your agents on your own data.
Bring a dataset and the rules your business cares about. We will build the metrics with you and run them against your models and agents.