Set up observability
METIS does not read your application's answers from the simulation itself. It reads the traces that your application sends to Langfuse, so if a trace is missing, or is recorded in an old format, the graders have nothing to grade. Three things have to be right: the Langfuse Server that stores the traces, the Langfuse Client SDK that sends them, and the way you instrument your code.
Langfuse Server
Use Langfuse v4, self-hosted or Cloud. This is the version of the server, not of the Langfuse Client SDK.
A self-hosted Langfuse v4 server runs in events_only mode. In that mode, the server rejects the
legacy ingestion API and the /api/public/traces, /spans, /generations and /events
endpoints. It also no longer builds trace-level input, output and attributes for you.
Langfuse Client SDK
The Langfuse Client SDK is the library inside your application that sends its traces to the
Langfuse server. Use the Python langfuse package at 4.7.0 or later, or the JS/TS @langfuse/*
packages at 5.4.0 or later. These versions send traces over OpenTelemetry, which a v4 server
needs.
To check, run pip show langfuse (Python) or npm ls @langfuse/tracing (JS/TS) in your
application's environment, and upgrade if the version is lower.
Follow instrumentation best practices
Langfuse describes how to instrument an application in its best-practices page. Three of its practices matter most to METIS, because they decide whether the graders can read the conversation:
The propagate_attributes() call that sets the attributes comes from Langfuse's
instrumentation guide.
1. Root observation
Wrap each turn in one enclosing root observation, of type agent or span. Put the user message
in its input, and put the final answer in its output. Instrument fully and do not trim: any
grader may read any of it.
To check, search your code for langfuse.trace(, update_trace( and set_current_trace_io(.
A v4 server no longer supports these calls, and they are how legacy code recorded the input and
output of a trace. If you find any, replace them as shown below.
Before, the legacy pattern. It fails on a server in events_only mode:
# Legacy. Do not use.
trace = langfuse.trace(
name="support-agent",
user_id=user_id,
session_id=session_id,
input=user_message,
)
span = trace.span(name="retrieval")
trace.update(output=final_answer)
After, the v4 pattern:
from langfuse import get_client, propagate_attributes
langfuse = get_client()
def handle_turn(user_id: str, session_id: str, user_message: str) -> str:
# Attributes set here are inherited by everything started inside the block.
with propagate_attributes(
user_id=user_id,
session_id=session_id,
tags=["metis"],
trace_name="support-agent",
):
# One enclosing root observation per turn.
with langfuse.start_as_current_observation(
name="support-agent",
as_type="agent",
input={"message": user_message},
) as root:
# Your code. Its steps nest under root.
final_answer = run_agent(user_message)
root.update(output={"answer": final_answer})
return final_answer
If you use the @observe decorator, put @observe(as_type="agent") on the function that handles
the whole turn, and make sure it records the user message as its input and the answer as its
output. The examples on this page are Python. For JS/TS, follow the same steps with
Langfuse's own guide.
2. Attributes
Propagate user_id, session_id and tags with propagate_attributes(), so that every child
step inherits them. The example above does this: propagate_attributes() wraps the root
observation, so the attributes are set before any child step starts.
To check, look for the right userId and sessionId on the root observation and on its child
steps. See Verify your setup yourself.
3. Session ID
The runner copies the session ID from your application's response, and METIS fetches every trace that carries that ID. So your application must stamp the same ID on every trace of one trial, and it must return that ID to the caller. If each turn gets a new ID, METIS finds one turn instead of the conversation.
Verify your setup yourself (optional)
This step is optional, and you do it by hand: METIS never runs it for you. It is worth doing
before your first evaluation, because without it a mistake shows up only after a full simulation,
when metis engine evaluate cannot find the traces.
The check tests all three Langfuse parts at once: the server, the Langfuse Client SDK and the instrumentation.
-
Make sure
.envholds the keys of the Langfuse project that your application sends its traces to. METIS reads the traces with the same keys. See Installation and Setup. -
Send one test conversation to your application.
-
Ask Langfuse for its root observations. This loads
.envinto your shell first:Terminal1set -a; source .env; set +a2curl -s -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \3 "$LANGFUSE_BASE_URL/api/public/v2/observations?isRootObservation=true&limit=5"
The test conversation should appear with:
isRootObservationset totrue,- a populated
inputthat holds the user message and a populatedoutputthat holds the answer, - the right
userIdandsessionId.
You can also open the trace in the Langfuse web page and check the same three things. If they all hold, your application records traces that METIS can read.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
The root input or output is empty | The turn is not wrapped in a root observation, or output is never set on it | Wrap the turn as shown above and call update(output=...) before the block ends. Replace update_trace() and set_current_trace_io() |
| Child steps show up as separate roots | They run outside the root observation, for example in a thread or task that lost the context | Start the work inside the with block, or pass the context to the worker |
userId, sessionId or tags are missing | The attributes were set with a removed call, or set after the children started | Use propagate_attributes() around the root observation |
UnsupportedTraceError, "recorded before this Langfuse server's v4 upgrade" | The trace was recorded before the server moved to v4 | Run that trial again. METIS still grades your other sessions |
| The session has fewer traces than turns | A framework overwrote the session ID on some of its spans | Compare the sessionId of the root and the child steps in the Langfuse web page. Set the session ID after the framework starts its spans, or turn off the framework's own session handling |
| Traces arrive minutes late | An older Langfuse Client SDK | Upgrade the Langfuse Client SDK first. See Run the graders |