Skip to main content

Tool choice appropriateness

metis.library.graders.ToolChoiceAppropriatenessGrader
Measure
tool_choice_appropriateness
Scopes
session
Grader type
llm
Reference type
rubric_based
Judgment form
binary
Aggregation
none_single_score

Assess whether the selected tool is suitable for the task.

Adapted from Phoenix tool_selection.

class ToolListArgs(BaseModel):
available_tools: tuple[NonEmptyText, ...] = Field(
default=(),
description=(
'Tool descriptions, such as "search(query): Searches the index", for when the '
"person writing the specification or the wizard knows them. When empty, the "
"grader reads the tools that the session offered."
),
)


class TaskArgs(BaseModel):
task_description: NonEmptyText | None = None
"""What the agent was asked to do. When unset, the judge infers the goal from the user's
messages. A criterion runs in every task that selects its behaviour, so this text applies
to the scenarios of all those tasks."""


class ToolChoiceAppropriatenessGrader(Grader[ToolChoiceAppropriatenessArgs]):
def criteria(self, tools: tuple[str, ...]) -> str:
return (
"You are an impartial judge evaluating an agent's tool-calling behavior, "
"specifically whether the agent selected the most appropriate tool or tools "
"for the task."
+ self.args.task_block()
+ section("available_tools", tools)
+ "\n\nYour task: Determine whether the agent's tool selection was correct or "
"incorrect based on:\n"
"- The conversation context in the session evidence\n"
"- The available tools in the available_tools list above\n"
"- The agent's output and tool invocation(s) in the session evidence\n\n"
"Score 1 (correct) only when ALL of the following are true:\n"
"- The agent chose the best available tool for the user query OR correctly "
"avoided tools if none were needed.\n"
"- The tool name exists in the available tools list.\n"
"- The agent selected the correct number of tools for the task.\n\n"
"Score 0 (incorrect) if ANY of the following are true:\n"
"- The agent used a hallucinated or nonexistent tool.\n"
"- The agent selected a tool when none was needed.\n"
"- The agent did not use a tool when one was required.\n"
"- The agent chose a suboptimal or irrelevant tool.\n"
"- The tool name does not appear in the available tools list.\n\n"
"Before providing your final judgment, explain your reasoning and consider:\n"
"- What does the input context require?\n"
"- Can this be answered without tools, or is a tool necessary?\n"
"- If a tool was selected, does it exist in the available tools?\n"
"- Does the selected tool's description match the user's needs?\n"
"- Is there a better tool available that should have been chosen instead?"
"\n\n<assessment_boundaries>\n"
"The available_tools list above is the complete set the agent could "
"choose from. Do not assume the trace shows every available tool — tools "
"that were available but unused may still be the better choice. Arguments "
"are assessed separately."
"\n</assessment_boundaries>"
)