Skip to main content

Clarification appropriateness

metis.library.graders.ClarificationAppropriatenessGrader
Measure
clarification_appropriateness
Scopes
session
Grader type
llm
Reference type
rubric_based
Judgment form
binary
Aggregation
none_single_score

Whether the assistant asks the clarifying questions it needs, and no others.

class ClarificationAppropriatenessArgs(JudgeArgs):
confirm_before: NonEmptyText | None = Field(
default=None,
description=(
"Actions that the assistant must confirm with the user before it does them, "
'such as "payments, deletions, and messages sent for the user".'
),
)
question_limit: NonEmptyText | None = Field(
default=None,
description=(
"The most questions that the product accepts before the assistant must act, "
'such as "at most one question before the first answer".'
),
)


_CRITERIA = (
"You are grading how an assistant asks clarifying questions over a whole conversation. "
"First, list every question the assistant asked the user, including forms or option "
"lists it set up for the user to fill in, and every point where it proceeded on an "
"assumption. A question is necessary only when information that changes how the task "
"should be done is missing or ambiguous. For each question, check that the user had not "
"already given the answer, that the assistant could not work it out from the "
"conversation, and that it was not one of a run of questions asked turn after turn when "
"the assistant could have proceeded. For each assumption, check that a question was not "
"needed. Score 1 when every question was necessary and no necessary question was "
"missed, including when none was needed and none was asked. Score 0 when the assistant "
"asked a redundant or unnecessary question, asked again for something already given, or "
"proceeded on a guess where a question was needed. Do not judge a question that is a "
"response to being unable to proceed, such as a missing permission."
)

class ClarificationAppropriatenessGrader(Grader[ClarificationAppropriatenessArgs]):
def criteria(self) -> str:
rules = []
if self.args.confirm_before is not None:
rules.append(
"The assistant must confirm these actions with the user before it does "
f"them: {self.args.confirm_before} A confirmation question for such an "
"action is necessary. Doing such an action without one is a missed question."
)
if self.args.question_limit is not None:
rules.append(
f"The product sets this question limit: {self.args.question_limit} "
"Asking more questions than the limit allows scores 0, even when each "
"question was necessary."
)
return " ".join([_CRITERIA, *rules])