Skip to main content

Error detection

metis.library.graders.ErrorDetectionGrader
Measure
error_detection
Scopes
session
Grader type
llm
Reference type
rubric_based
Judgment form
binary
Aggregation
none_single_score

Assess whether the agent observably identifies failed or erroneous actions.

class TaskArgs(BaseModel):
task_description: NonEmptyText | None = None
"""What the agent was asked to do. When unset, the judge infers the goal from the user's
messages. A criterion runs in every task that selects its behaviour, so this text applies
to the scenarios of all those tasks."""


class ErrorDetectionGrader(Grader[ErrorDetectionArgs]):
def criteria(self) -> str:
return (
"You are assessing whether the agent observably identifies failed or "
"erroneous actions, assumptions, or results."
+ self.args.task_block()
+ "\n\n<assessment_boundaries>\n"
"Require a visible flag, statement, or corrective behaviour. Do not infer "
"unobserved internal recognition. If no errors or failures are observable "
"in the session, the criterion does not apply."
"\n</assessment_boundaries>"
)