Run the graders
metis engine evaluate reads each trial's conversation from Langfuse, runs the graders from your
specification and writes the results. It needs the LANGFUSE_* and OPENAI_API_KEY values in
.env, and an application that records its traces as described in
Set up observability.
What the graders judge
The graders read each conversation from your application's traces in Langfuse: the messages, the tool calls and their results. A grader can also judge a file that your application made, such as an image, a chart or a PDF.
Each grader has settings that choose what it judges, such as one turn, the whole session or one file. See How graders work for these settings, and the Grader library for each grader.
Run it
Pass the output folder that simulate created:
1metis engine evaluate --experiment-dir experiments/execute/<your-run>
simulate prints this command with your path. The folder must hold the frozen inputs/ copy
that simulate made. Running evaluate again on the same folder overwrites the earlier results.
When traces are missing
Langfuse can take time to show a trace after your application sends it. METIS waits for
--langfuse-timeout seconds, 90 by default:
1metis engine evaluate --experiment-dir experiments/execute/<your-run> --langfuse-timeout 300
Traces from older Langfuse Client SDKs can arrive several minutes late. If you see missing
traces, upgrade the Langfuse Client SDK in your
application first (Python langfuse 4.7.0 or later, JS/TS @langfuse/* 5.4.0 or later), and
only then raise the timeout. A longer timeout hides the delay but does not remove it. Also
check the keys in .env and the troubleshooting table.
metis engine run takes the same --langfuse-timeout option.
Outputs
evaluate writes outputs/reports/results.json in the experiment folder, and the log of the
latest evaluation to logs/evaluate.log. A trial that cannot be graded is marked failed with the
reason, and the other trials are still graded.
Next step
Turn your results into a report.