Skip to main content

Grader library

METIS graders assess a recorded session against one criterion. Each grader measures one attribute of agent quality and gives a score. The grader's judgment form decides the form of that score: every grader in the library today is binary, so it scores 0 or 1. When the event that the criterion looks for does not happen in the session, the result is errored and has no score.

To learn how the Wizard uses the library, which settings a grader takes, and how a grader turns a session into scores, see How graders work.

METIS adapts battle-tested graders from other evaluation packages. It copies the upstream prompt wording that fits the measure and keeps its own scoring rules. A grader marked METIS uses a rubric written for METIS. Each "Adapted from" link goes to the upstream file at the commit that METIS copied. For the licence of each upstream package, see EXTERNAL_SOURCES.md.

Select a grader to see its settings, an example criterion, how it is judged, and its limits.

GraderWhat it checksAdapted from
Task and Output Quality
TaskSuccessGraderSource fileWhether the requested end goal was achieved under declared success conditions.DeepEval Task Completion (verdict)
OutputCorrectnessGraderSource fileWhether an answer or deliverable is substantively correct.MLflow Correctness
FactualAccuracyGraderSource fileWhether factual claims agree with the designated authoritative ground truth.Braintrust Factuality, OpenAI Evals fact
RequiredComponentCompletenessGraderSource fileWhether every explicitly requested component is present.Phoenix completeness, MLflow Completeness
TaskConstraintAdherenceGraderSource fileWhether explicit execution or presentation restrictions were respected.MLflow Guidelines
FormatSchemaComplianceGraderSource fileWhether an output satisfies specified structural and syntactic requirements.METIS
ClaimSupportByEvidenceGraderSource fileWhether claims in the output are supported by the evidence in the session.DeepEval Faithfulness, Braintrust Faithfulness
EvidenceSourceSuitabilityGraderSource fileWhether evidence sources meet authority, relevance, and currency requirements.METIS
SpeechRecognitionAccuracyGraderSource fileWhether recognized words accurately represent the spoken input.DeepEval Transcription Accuracy
ContentFidelityGraderSource fileWhether a transformation preserves the meaning of its source.METIS
AttributionAccuracyGraderSource fileWhether citations correctly identify evidence that supports claims.DeepEval Citation Faithfulness
Execution and Decision Quality
PlanningQualityGraderSource fileWhether the proposed sequence of actions is feasible and sufficient.DeepEval Plan Quality
ActionChoiceAppropriatenessGraderSource fileWhether recorded intermediate choices are suitable.METIS
ToolChoiceAppropriatenessGraderSource fileWhether the selected tool is suitable for the task.Phoenix tool_selection
ToolArgumentCorrectnessGraderSource fileWhether submitted tool parameters are valid and semantically aligned.Phoenix tool_invocation
StateTrackingGraderSource fileWhether the agent maintains up-to-date state representation.METIS
UseOfPriorInformationGraderSource fileWhether relevant earlier facts, instructions, or experiences are retrieved and applied appropriately.MLflow KnowledgeRetention, DeepEval Knowledge Retention
ErrorDetectionGraderSource fileWhether the agent observably identifies failed or erroneous actions.METIS
ErrorRecoveryGraderSource fileWhether corrective action after a failure restores task progress.METIS
AdaptationToChangedConditionsGraderSource fileWhether the agent appropriately revises its approach when conditions change.METIS
StoppingAppropriatenessGraderSource fileWhether the agent stops, continues, or hands over at the appropriate point.METIS
AvoidingUnproductiveRepetitionGraderSource fileWhether the agent avoids repeating equivalent actions without progress.METIS
AppropriateHandlingOfCapabilityLimitsGraderSource fileWhether choices to seek information, abstain, or escalate are appropriate.METIS
OutcomeVerificationGraderSource fileWhether the agent checks relevant results before claiming success.METIS
ReasoningValidityGraderSource fileWhether recorded conclusions follow from stated premises and evidence.METIS
CoordinationQualityGraderSource fileWhether agents appropriately delegate, share information, and reconcile.METIS
Reliability and Robustness
RobustnessToToolFailuresGraderSource fileWhether the required result or an acceptable fallback holds when a tool fails.Phoenix tool_response_handling, for the error-handling rules only
LongTaskPerformanceStabilityGraderSource fileWhether performance holds as the number of dependent steps increases.METIS
RobustnessToContextDegradationGraderSource fileWhether performance holds when the context is long, noisy, incomplete, or stale.METIS
Interaction Quality
RelevanceGraderSource fileWhether the assistant's replies address what the user needs.DeepEval Turn Relevancy
CommunicationClarityGraderSource fileWhether the assistant's replies are clear, well organized and free of avoidable ambiguity.Phoenix conciseness, for the padding check only
CrossTurnCoherenceGraderSource fileWhether the conversation stays connected from one recorded turn to the next.METIS
ClarificationAppropriatenessGraderSource fileWhether the assistant asks the clarifying questions it needs, and no others.METIS
UserCorrectionSupportGraderSource fileWhether the assistant accepts and applies the user's corrections and overrides.Phoenix user_friction
ActionStatusTransparencyGraderSource fileWhether the user can see what the assistant did, what is still pending and what failed.METIS
ExplanationUsefulnessGraderSource fileWhether the assistant's explanations help the user understand a result or decision.METIS
ToneAppropriatenessGraderSource fileWhether the assistant's tone suits the user, the situation and the task.METIS
OutputNaturalnessGraderSource fileWhether the assistant's replies read as fluent, idiomatic text in their language.METIS
Fallback judge
CriterionJudgeGraderSource fileThe recorded session against a criterion description with a binary score.METIS