Grader library
METIS graders assess a recorded session against one criterion. Each grader measures one
attribute of agent quality and gives a score. The grader's judgment form decides the form
of that score: every grader in the library today is binary, so it scores 0 or 1. When the
event that the criterion looks for does not happen in the session, the result is errored and
has no score.
To learn how the Wizard uses the library, which settings a grader takes, and how a grader turns a session into scores, see How graders work.
METIS adapts battle-tested graders from other evaluation packages. It copies the upstream prompt wording that fits the measure and keeps its own scoring rules. A grader marked METIS uses a rubric written for METIS. Each "Adapted from" link goes to the upstream file at the commit that METIS copied. For the licence of each upstream package, see EXTERNAL_SOURCES.md.
Select a grader to see its settings, an example criterion, how it is judged, and its limits.
| Grader | What it checks | Adapted from |
|---|---|---|
| Task and Output Quality | ||
TaskSource file | Whether the requested end goal was achieved under declared success conditions. | DeepEval Task Completion (verdict) |
OutputSource file | Whether an answer or deliverable is substantively correct. | MLflow Correctness |
FactualSource file | Whether factual claims agree with the designated authoritative ground truth. | Braintrust Factuality, OpenAI Evals fact |
RequiredSource file | Whether every explicitly requested component is present. | Phoenix completeness, MLflow Completeness |
TaskSource file | Whether explicit execution or presentation restrictions were respected. | MLflow Guidelines |
FormatSource file | Whether an output satisfies specified structural and syntactic requirements. | METIS |
ClaimSource file | Whether claims in the output are supported by the evidence in the session. | DeepEval Faithfulness, Braintrust Faithfulness |
EvidenceSource file | Whether evidence sources meet authority, relevance, and currency requirements. | METIS |
SpeechSource file | Whether recognized words accurately represent the spoken input. | DeepEval Transcription Accuracy |
ContentSource file | Whether a transformation preserves the meaning of its source. | METIS |
AttributionSource file | Whether citations correctly identify evidence that supports claims. | DeepEval Citation Faithfulness |
| Execution and Decision Quality | ||
PlanningSource file | Whether the proposed sequence of actions is feasible and sufficient. | DeepEval Plan Quality |
ActionSource file | Whether recorded intermediate choices are suitable. | METIS |
ToolSource file | Whether the selected tool is suitable for the task. | Phoenix tool_selection |
ToolSource file | Whether submitted tool parameters are valid and semantically aligned. | Phoenix tool_invocation |
StateSource file | Whether the agent maintains up-to-date state representation. | METIS |
UseSource file | Whether relevant earlier facts, instructions, or experiences are retrieved and applied appropriately. | MLflow KnowledgeRetention, DeepEval Knowledge Retention |
ErrorSource file | Whether the agent observably identifies failed or erroneous actions. | METIS |
ErrorSource file | Whether corrective action after a failure restores task progress. | METIS |
AdaptationSource file | Whether the agent appropriately revises its approach when conditions change. | METIS |
StoppingSource file | Whether the agent stops, continues, or hands over at the appropriate point. | METIS |
AvoidingSource file | Whether the agent avoids repeating equivalent actions without progress. | METIS |
AppropriateSource file | Whether choices to seek information, abstain, or escalate are appropriate. | METIS |
OutcomeSource file | Whether the agent checks relevant results before claiming success. | METIS |
ReasoningSource file | Whether recorded conclusions follow from stated premises and evidence. | METIS |
CoordinationSource file | Whether agents appropriately delegate, share information, and reconcile. | METIS |
| Reliability and Robustness | ||
RobustnessSource file | Whether the required result or an acceptable fallback holds when a tool fails. | Phoenix tool_response_handling, for the error-handling rules only |
LongSource file | Whether performance holds as the number of dependent steps increases. | METIS |
RobustnessSource file | Whether performance holds when the context is long, noisy, incomplete, or stale. | METIS |
| Interaction Quality | ||
RelevanceSource file | Whether the assistant's replies address what the user needs. | DeepEval Turn Relevancy |
CommunicationSource file | Whether the assistant's replies are clear, well organized and free of avoidable ambiguity. | Phoenix conciseness, for the padding check only |
CrossSource file | Whether the conversation stays connected from one recorded turn to the next. | METIS |
ClarificationSource file | Whether the assistant asks the clarifying questions it needs, and no others. | METIS |
UserSource file | Whether the assistant accepts and applies the user's corrections and overrides. | Phoenix user_friction |
ActionSource file | Whether the user can see what the assistant did, what is still pending and what failed. | METIS |
ExplanationSource file | Whether the assistant's explanations help the user understand a result or decision. | METIS |
ToneSource file | Whether the assistant's tone suits the user, the situation and the task. | METIS |
OutputSource file | Whether the assistant's replies read as fluent, idiomatic text in their language. | METIS |
| Fallback judge | ||
CriterionSource file | The recorded session against a criterion description with a binary score. | METIS |