Tool argument correctness
metis.library.graders.ToolArgumentCorrectnessGraderAssess whether submitted tool parameters are valid and semantically aligned.
Adapted from Phoenix tool_invocation.
class ToolListArgs(BaseModel):
available_tools: tuple[NonEmptyText, ...] = Field(
default=(),
description=(
'Tool descriptions, such as "search(query): Searches the index", for when the '
"person writing the specification or the wizard knows them. When empty, the "
"grader reads the tools that the session offered."
),
)
class TaskArgs(BaseModel):
task_description: NonEmptyText | None = None
"""What the agent was asked to do. When unset, the judge infers the goal from the user's
messages. A criterion runs in every task that selects its behaviour, so this text applies
to the scenarios of all those tasks."""
class ToolArgumentCorrectnessArgs(JudgeArgs, TaskArgs, ToolListArgs):
argument_requirements: tuple[str, ...] = Field(
default=(),
description=(
"Specific syntactic or semantic requirements for tool parameters. "
"When empty, the judge assesses general argument correctness."
),
)
class ToolArgumentCorrectnessGrader(Grader[ToolArgumentCorrectnessArgs]):
def criteria(self, tools: tuple[str, ...]) -> str:
return (
"You are an impartial judge evaluating an agent's tool-calling behavior, "
"specifically whether the agent invoked a tool (or tools) correctly with "
"valid arguments and proper formatting."
+ self.args.task_block()
+ section("argument_requirements", self.args.argument_requirements)
+ available_tools_block(
tools,
"Check each submitted argument against the tool's description in "
"available_tools: the tool name, the parameter names, and the types "
"where the description gives them.",
)
+ (
""
if tools
else " The rules below that mention available_tools then apply only "
"where the evidence itself shows a tool's parameters."
)
+ "\n\nYour task: Determine whether the agent's tool invocation(s) were "
"correct or incorrect based on:\n"
"- The full conversation context (including all previous turns, not just "
"the most recent message)\n"
"- The tool descriptions in available_tools\n"
"- The agent's tool invocation(s) with arguments\n\n"
"IMPORTANT - Tool Invocation vs. Tool Selection:\n"
"- You are ONLY evaluating the tool invocation, not the tool selection.\n"
"- If the tool selection is incorrect or not relevant to the user's query, "
"but the tool invocation is correct, score 1.\n"
"- If the tool selection is correct but the tool invocation is incorrect, "
"score 0.\n\n"
"IMPORTANT - Multi-Tool Invocations:\n"
"- The agent may invoke MULTIPLE tools in a single response. This is valid "
"and expected for complex requests.\n"
"- When multiple tools are invoked, evaluate EACH tool invocation "
"independently.\n"
"- Score 1 only if ALL tool invocations are correct.\n"
"- Score 0 if ANY tool invocation has an error.\n\n"
"IMPORTANT - Conversation Context:\n"
"- Read the entire conversation history carefully, not just the final user "
"message.\n"
"- Argument values may need to be extracted from EARLIER turns in the "
"conversation (e.g., user mentions a location, date, or quantity in a "
"previous message).\n"
"- The agent should use context from the full conversation to populate "
"argument values correctly.\n\n"
"Score 1 (correct) only when ALL of the following are true for EVERY tool "
"invocation:\n"
"- JSON is properly structured (if applicable).\n"
"- All required fields/parameters are present.\n"
"- No hallucinated or nonexistent fields (all fields exist in the tool "
"descriptions in available_tools).\n"
"- Argument values match the user's intent from the conversation context "
"(correct types, realistic values).\n\n"
"Score 0 (incorrect) if ANY of the following are true for ANY tool "
"invocation:\n"
"- The invocation contains hallucinated or nonexistent fields not in the "
"tool descriptions in available_tools.\n"
"- Required fields/parameters are missing.\n"
"- JSON is improperly formatted or malformed.\n"
"- Argument values are incorrect, hallucinated, or do not match user intent "
"from the conversation.\n\n"
"Before providing your final judgment, explain your reasoning and consider:\n"
"- How many tools were invoked? Evaluate each one.\n"
"- Does each tool invocation match the tool descriptions in available_tools "
"for that tool?\n"
"- Are all required parameters provided with appropriate values for each "
"invocation?\n"
"- Are there any extra fields that don't exist in the tool descriptions in "
"available_tools?\n"
"- Looking at the FULL conversation: do the argument values accurately "
"reflect what the user requested across all messages?\n"
"- Check that you are not evaluating the tool selection, only the tool "
"invocation."
"\n\n<assessment_boundaries>\n"
"Assess syntactic validity and semantic appropriateness of submitted "
"parameters. Tool selection is a separate criterion."
"\n</assessment_boundaries>"
)