Skip to main content

Robustness to context degradation

metis.library.graders.RobustnessToContextDegradationGrader
Measure
robustness_to_context_degradation
Scopes
subsessionsession
Grader type
llm
Reference type
rubric_based
Judgment form
binary
Aggregation
none_single_score

Assess whether performance holds when the context is long, noisy, incomplete, or stale.

class SubsessionArgs(BaseModel):
turn: Literal["each", "last"] = "each"
"""The recorded turn each subsession ends at: ``each`` recorded turn, or only the ``last``
one."""
window: int | None = Field(default=None, ge=1, strict=True)
"""How many recorded turns one subsession holds: its end turn and the recorded turns just
before it. ``None`` holds every recorded turn before it, so the judge has the whole
conversation so far. The recorded turns before the subsession are not shown to the judge."""


class TaskArgs(BaseModel):
task_description: NonEmptyText | None = None
"""What the agent was asked to do. When unset, the judge infers the goal from the user's
messages. A criterion runs in every task that selects its behaviour, so this text applies
to the scenarios of all those tasks."""


class RobustnessToContextDegradationArgs(JudgeArgs, TaskArgs, SubsessionArgs):
degradation: tuple[str, ...] = Field(
default=(),
description=(
"What makes the context harder in this session, and where the needed evidence "
"is, such as 'Turn 3 adds a stale price; the current price is in turn 1.' A turn "
"number is a recorded turn, as the `turn:N` labels count it. When empty, the "
"judge looks for long, noisy, incomplete, or stale context itself."
),
)


class RobustnessToContextDegradationGrader(Grader[RobustnessToContextDegradationArgs]):
def criteria(self) -> str:
return (
"You are assessing whether an agent keeps its performance when the context it "
"has is degraded. Degraded context is long, noisy, incomplete, or stale. "
"Examples are irrelevant history, noise in messages or tool results, stale or "
"conflicting information, and context that was shortened or summarised, such "
"as by a context compaction step."
+ self.args.task_block()
+ section("degradation", self.args.degradation)
+ "\n\nScore 1 when the reply still uses the evidence that it needs, and is "
"not misled by the degradation:\n"
"- It finds and uses the needed facts and instructions among irrelevant "
"history or noise.\n"
"- It uses the current information when stale or conflicting information is "
"also present.\n"
"- It keeps the needed facts and instructions when earlier context was "
"shortened or summarised.\n"
"- When indispensable evidence is missing, it asks for the evidence, says that "
"it does not know, or abstains. It does not guess.\n\n"
"Score 0 when the reply uses stale or wrong information, loses a needed fact or "
"instruction, or invents the missing evidence."
"\n\n<assessment_boundaries>\n"
"If the context in the evidence is not degraded in a way that bears on the "
"judged reply, the criterion does not apply. Use of Prior Information judges "
"recall in general. This criterion judges whether performance holds when the "
"context is degraded."
"\n</assessment_boundaries>"
)