Skip to main content

Set up observability

METIS does not read your application's answers from the simulation itself. It reads the traces that your application sends to Langfuse, so if a trace is missing, or is recorded in an old format, the graders have nothing to grade. Three things have to be right: the Langfuse Server that stores the traces, the Langfuse Client SDK that sends them, and the way you instrument your code.

Langfuse Server​

Use Langfuse v4, self-hosted or Cloud. This is the version of the server, not of the Langfuse Client SDK.

A self-hosted Langfuse v4 server runs in events_only mode. In that mode, the server rejects the legacy ingestion API and the /api/public/traces, /spans, /generations and /events endpoints. It also no longer builds trace-level input, output and attributes for you.

Langfuse Client SDK​

The Langfuse Client SDK is the library inside your application that sends its traces to the Langfuse server. Use the Python langfuse package at 4.7.0 or later, or the JS/TS @langfuse/* packages at 5.4.0 or later. These versions send traces over OpenTelemetry, which a v4 server needs.

To check, run pip show langfuse (Python) or npm ls @langfuse/tracing (JS/TS) in your application's environment, and upgrade if the version is lower.

Follow instrumentation best practices​

Langfuse describes how to instrument an application in its best-practices page. Three of its practices matter most to METIS, because they decide whether the graders can read the conversation:

  1. A root observation for each turn.
  2. Attributes on every step.
  3. One session ID for the whole trial.

The propagate_attributes() call that sets the attributes comes from Langfuse's instrumentation guide.

1. Root observation​

Wrap each turn in one enclosing root observation, of type agent or span. Put the user message in its input, and put the final answer in its output. Instrument fully and do not trim: any grader may read any of it.

To check, search your code for langfuse.trace(, update_trace( and set_current_trace_io(. A v4 server no longer supports these calls, and they are how legacy code recorded the input and output of a trace. If you find any, replace them as shown below.

Before, the legacy pattern. It fails on a server in events_only mode:

# Legacy. Do not use.
trace = langfuse.trace(
name="support-agent",
user_id=user_id,
session_id=session_id,
input=user_message,
)
span = trace.span(name="retrieval")
trace.update(output=final_answer)

After, the v4 pattern:

from langfuse import get_client, propagate_attributes

langfuse = get_client()


def handle_turn(user_id: str, session_id: str, user_message: str) -> str:
# Attributes set here are inherited by everything started inside the block.
with propagate_attributes(
user_id=user_id,
session_id=session_id,
tags=["metis"],
trace_name="support-agent",
):
# One enclosing root observation per turn.
with langfuse.start_as_current_observation(
name="support-agent",
as_type="agent",
input={"message": user_message},
) as root:
# Your code. Its steps nest under root.
final_answer = run_agent(user_message)
root.update(output={"answer": final_answer})
return final_answer

If you use the @observe decorator, put @observe(as_type="agent") on the function that handles the whole turn, and make sure it records the user message as its input and the answer as its output. The examples on this page are Python. For JS/TS, follow the same steps with Langfuse's own guide.

2. Attributes​

Propagate user_id, session_id and tags with propagate_attributes(), so that every child step inherits them. The example above does this: propagate_attributes() wraps the root observation, so the attributes are set before any child step starts.

To check, look for the right userId and sessionId on the root observation and on its child steps. See Verify your setup yourself.

3. Session ID​

The runner copies the session ID from your application's response, and METIS fetches every trace that carries that ID. So your application must stamp the same ID on every trace of one trial, and it must return that ID to the caller. If each turn gets a new ID, METIS finds one turn instead of the conversation.

Verify your setup yourself (optional)​

This step is optional, and you do it by hand: METIS never runs it for you. It is worth doing before your first evaluation, because without it a mistake shows up only after a full simulation, when metis engine evaluate cannot find the traces.

The check tests all three Langfuse parts at once: the server, the Langfuse Client SDK and the instrumentation.

  1. Make sure .env holds the keys of the Langfuse project that your application sends its traces to. METIS reads the traces with the same keys. See Installation and Setup.

  2. Send one test conversation to your application.

  3. Ask Langfuse for its root observations. This loads .env into your shell first:

    Terminal
    1set -a; source .env; set +a
    2curl -s -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
    3 "$LANGFUSE_BASE_URL/api/public/v2/observations?isRootObservation=true&limit=5"

The test conversation should appear with:

  • isRootObservation set to true,
  • a populated input that holds the user message and a populated output that holds the answer,
  • the right userId and sessionId.

You can also open the trace in the Langfuse web page and check the same three things. If they all hold, your application records traces that METIS can read.

Troubleshooting​

SymptomLikely causeFix
The root input or output is emptyThe turn is not wrapped in a root observation, or output is never set on itWrap the turn as shown above and call update(output=...) before the block ends. Replace update_trace() and set_current_trace_io()
Child steps show up as separate rootsThey run outside the root observation, for example in a thread or task that lost the contextStart the work inside the with block, or pass the context to the worker
userId, sessionId or tags are missingThe attributes were set with a removed call, or set after the children startedUse propagate_attributes() around the root observation
UnsupportedTraceError, "recorded before this Langfuse server's v4 upgrade"The trace was recorded before the server moved to v4Run that trial again. METIS still grades your other sessions
The session has fewer traces than turnsA framework overwrote the session ID on some of its spansCompare the sessionId of the root and the child steps in the Langfuse web page. Set the session ID after the framework starts its spans, or turn off the framework's own session handling
Traces arrive minutes lateAn older Langfuse Client SDKUpgrade the Langfuse Client SDK first. See Run the graders

Next step​

Check your environment.