Home/Products/Bud AI Foundry/Evaluation Metrics
Bud AI Foundry · Evaluations · Metrics

Score your models and agents on what your business needs.

Metrics decide how every answer is scored in the Evaluations module of Bud AI Foundry. Start from 64 built-in metrics in 12 categories, create your own without code, and test models and agents against your own data and requirements.

Demo

From the metric library to a custom metric, in one walkthrough.

Browse the built-in metrics, create and calibrate a custom one, then use it in a new evaluation.

Why metrics

A test is only as useful as the way you score it.

Public evaluation datasets are a useful start, but they rarely reflect your business or how your agents will be used. Before anything reaches production, you need to know how it performs on your own data and requirements, and whether it is safe, with evidence. Metrics let you define what good looks like, test against it, and keep evaluating as models and agents meet real-world data.

Built-in metrics64, in 12 categories
Custom metricsNo code · shared across the workspace
ScoringAI judge, or rule-based for 10
ScopeOne response or a whole conversation
CalibrationAgainst your experts' judgement
The metric library

64 metrics, across everything your AI does.

The built-in library spans 12 categories, from answer quality and retrieval to agents, safety and business behaviour. The highlights are below, and every metric is listed in the catalogue. Custom metrics your team adds sit beside them and are shared across the workspace, so once someone defines what good looks like, every evaluation can use it.

01

Answer quality · 4

Answer relevancy, hallucination, whether the response follows the prompt's instructions, and how well a summary matches its source.

02

Retrieval (RAG) · 9

Check whether a RAG system found the right information and actually used it: contextual precision, recall and relevancy, faithfulness and citation faithfulness, single-turn or per turn.

03

Agents · 14

Most agent failures don't show in the final answer. Tool use, MCP, and planning and execution metrics read the agent's full trace to catch the wrong tool, a step repeated in a loop, or ten actions where three would do.

04

Business behaviour · 13

Did it hand over to a human when it should have? Make a promise it wasn't authorised to make, or mention a competitor? Include the disclosure your industry requires? The classifiers measure this on every test case, instead of spot checks by hand.

05

Safety · 6

Bias, toxicity, PII leakage, misuse outside the model's declared domain, advice in restricted areas such as finance, medicine or law, and responses that break the assigned role.

06

Rule-based, no judge · 10

Most metrics use an AI judge. Ten are fully rule-based, such as exact match, pattern match, BLEU and ROUGE, or confirming that an agent used only the tools it was allowed to. They give the same result every time, so they are easy to audit.

The full catalogue

All 64 built-in metrics.

Grouped into 12 categories. Open a category to see each metric, whether it scores one response or a whole conversation, and what it measures. The 10 metrics marked “no judge” are rule-based and give the same result every time.

Answer quality4
MetricScopeWhat it measures
Answer RelevancySingle turnHow relevant the answer is to the question
HallucinationSingle turnContent in the answer that the context doesn’t support
Prompt AlignmentSingle turnWhether the response follows the prompt’s instructions
SummarizationSingle turnHow well a summary matches its source document
Retrieval (RAG)9
MetricScopeWhat it measures
Citation FaithfulnessSingle turnWhether the retrieved passages back up the answer’s citations
Contextual PrecisionSingle turnHow much of the retrieved context is relevant
Contextual RecallSingle turnHow much of the needed information was retrieved
Contextual RelevancySingle turnHow relevant the context is to the question
FaithfulnessSingle turnWhether the answer is factually consistent with the context
Turn Contextual PrecisionMulti-turnRanking of relevant context, scored per turn
Turn Contextual RecallMulti-turnHow much of the expected outcome the context covers, per turn
Turn Contextual RelevancyMulti-turnRelevance of the context, per turn
Turn FaithfulnessMulti-turnFactual consistency with the context, per turn
Tool use4
MetricScopeWhat it measures
Argument CorrectnessSingle turnWhether each tool call’s arguments fit the input
Tool CorrectnessSingle turn · no judgeWhether the expected tools were called with the expected arguments
Tool PermissionSingle turn · no judgeWhether every tool called is allowed and none is denied
Tool UseMulti-turnWhether the agent picked the right tools and arguments
MCP3
MetricScopeWhat it measures
MCP Task CompletionMulti-turnWhether MCP-backed agents complete the task
MCP UseSingle turnWhether MCP tools and resources are used appropriately
Multi-Turn MCP UseMulti-turnMCP use across a whole conversation
Planning & execution7
MetricScopeWhat it measures
Agent Loop DetectionSingle turn · no judgeRepeated calls, stalled reasoning or cycles in the agent’s trace
Goal AccuracyMulti-turnWhether the agent reached the goal, and how well it planned
Plan AdherenceSingle turnWhether the agent followed its own plan
Plan QualitySingle turnWhether the plan suited the task
Step EfficiencySingle turnWhether the agent avoided unnecessary steps
Task CompletionSingle turnWhether the task was completed end to end
Topic AdherenceMulti-turnWhether the conversation stayed on the relevant topics
Conversation4
MetricScopeWhat it measures
Conversation CompletenessMulti-turnWhether every user request was fulfilled
Knowledge RetentionMulti-turnWhether facts from earlier in the conversation are remembered and reused
Role AdherenceMulti-turnWhether the chatbot stayed within its assigned role
Turn RelevancyMulti-turnAnswer relevancy, scored per turn
Safety6
MetricScopeWhat it measures
BiasSingle turnBiased content in the answer
MisuseSingle turnUse outside the model’s declared domain (lower is better)
Non-AdviceSingle turnAdvice given in restricted areas such as finance, medicine or law (lower is better)
PII LeakageSingle turnPersonal data leaked in the response (lower is better)
Role ViolationSingle turnResponses that break the assigned role (lower is better)
ToxicitySingle turnToxic content in the answer
Format3
MetricScopeWhat it measures
Exact MatchSingle turn · no judgeWhether the answer is exactly the same string as the expected output
JSON CorrectnessSingle turnWhether the JSON is valid against the expected schema
Pattern MatchSingle turn · no judgeWhether the answer matches a regular expression
Text similarity5
MetricScopeWhat it measures
Accepted Answer MatchSingle turn · no judgeNormalised exact match against any of the accepted answers
BLEUSingle turn · no judgeSentence BLEU score against the expected output
ROUGE-1Single turn · no judgeSingle-word (unigram) overlap F1
ROUGE-2Single turn · no judgeWord-pair (bigram) overlap F1
ROUGE-LSingle turn · no judgeLongest common subsequence F1
Classifiers13
MetricScopeWhat it measures
AbstentionSingle & multi-turnWhether the assistant abstained, answered or made something up
ClarificationSingle & multi-turnWhether it asked for clarification, answered directly or guessed
Data LeakageSingle & multi-turnLeaks of the system prompt, personal data or secrets
EscalationSingle & multi-turnWhether it handed off to a human, offered to, or handled it alone
Forbidden CommitmentsSingle & multi-turnUnauthorised commitments, mentions of competitors, or disparagement
Instruction CompletenessSingle & multi-turnWhether it followed all instructions, some, or none
Prompt InjectionSingle & multi-turnWhether it resisted, partly followed or followed an injected instruction
RefusalSingle & multi-turnWhether it complied, refused or partly refused
Required DisclosureSingle & multi-turnWhether the required disclosures were made
ResolutionSingle & multi-turnWhether the issue was resolved, left unresolved or handed over
Response LanguageSingle & multi-turnWhether the answer is in the user’s language
Scope AdherenceSingle & multi-turnWhether it stayed in scope, deflected, or answered an out-of-scope request
Tone AdherenceSingle & multi-turnWhether the required tone was kept
Row rules1
MetricScopeWhat it measures
Row rulesSingle & multi-turnEach row’s own “Must” and “Must not” conditions, scored by the share met
Images5
MetricScopeWhat it measures
Image CoherenceSingle turnWhether the image is coherent with the input text and instructions
Image EditingSingle turnQuality of an image edit against its instruction
Image HelpfulnessSingle turnWhether a returned image helps answer the query
Image ReferenceSingle turnWhether the response refers correctly to what’s in the input image
Text-to-ImageSingle turnHow faithfully a generated image matches its text prompt
Inspect and adapt

See how every metric works, and make it yours.

No metric is a black box, and none has to be used exactly as shipped.

Inspect

Open any metric

See what it measures, which data it needs and how it is set up, including the pass mark, the judge model and whether it explains its scores.

Adapt

Duplicate and calibrate

If a built-in metric is close but not quite right, duplicate it as your own and calibrate it against your team's judgement, so its scores reflect how your experts would grade.

Create a custom metric

Five steps, no code.

Describe what a good answer looks like, decide how the judge should reach its score, and set the line between pass and fail.

  1. 01

    Choose how to score

    Score the output with a rule, a decision tree or a label.

  2. 02

    Choose the scope

    A single-turn metric scores one input and its response. A multi-turn metric evaluates an entire conversation.

  3. 03

    Describe a good answer

    Select the fields the judge should consider, name the metric and describe what a good answer looks like.

  4. 04

    Define the judging

    Let the judging model decide from your description, or give it a fixed set of steps it must follow.

  5. 05

    Set the threshold

    Set the score that separates a pass from a fail, and save the metric to your library. Calibrate it with sample data against your expected results.

To use it, open the Evaluations hub and click New Evaluation. Give the evaluation a name and description, select its scope, upload the dataset you want to test against, then add metrics from your library to assess your model or agent.

Go deeper

Where Evaluation Metrics fits.

Metrics are part of the Evaluations module in Bud AI Foundry, where experiments and evaluations test models and agents side by side.

Get started with Bud AI Foundry

Evaluate your agents on your own data.

Bring a dataset and the rules your business cares about. We will build the metrics with you and run them against your models and agents.

01 Pick built-in metrics, or create your own without code.
02 Calibrate them against how your experts would grade.
03 Run evaluations on your own data, before and after launch.