Skip to content

Defining the Evaluation Task

The EvalTask is the central configuration that defines what to evaluate and how to parse responses. It's the most important component to configure correctly as it determines the structure of your entire evaluation.

Overview

EvalTask supports two main evaluation scenarios:

  1. Judge LLM Responses: Evaluate responses from another LLM (prompt + response evaluation)
  2. Judge Text Content: Evaluate arbitrary text content (response-only evaluation)

Quick Setup

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from meta_evaluator.eval_task import EvalTask

# Evaluate chatbot responses for safety and helpfulness
task = EvalTask(
    task_schemas={
        "safety": ["safe", "unsafe"],              # Classification task
        "helpfulness": ["helpful", "not_helpful"], # Classification task  
        "explanation": None                        # Free-form text
    },
    prompt_columns=["user_prompt"],                # Original prompt to the LLM
    response_columns=["chatbot_response"],         # LLM response to evaluate
    answering_method="structured",                 # Use structured output parsing
    structured_outputs_fallback=True               # Fallback to instructor or XML
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Evaluate text summaries for quality
task = EvalTask(
    task_schemas={
        "accuracy": ["accurate", "inaccurate"],
        "coherence": ["coherent", "incoherent"],
        "summary": None  # Free-form numeric or text score
    },
    prompt_columns=None,                   # No prompt context needed
    response_columns=["summary_text"],     # Just evaluate the summary
    answering_method="structured"
)

Arguments

Define columns (prompt_columns and response_columns)

Control which columns judges see during evaluation:

1
2
3
4
5
6
7
# Scenario 1: Judge sees both prompt and response
prompt_columns=["user_input", "system_instruction"]  # Context
response_columns=["llm_output"]                      # What to evaluate

# Scenario 2: Judge sees only the content to evaluate
prompt_columns=None                    # No context
response_columns=["text_to_evaluate"]  # Direct evaluation

Template Variable System:

MetaEvaluator uses a template-based system where your prompt.md files can include placeholders like {column_name} that get automatically replaced with actual data. The available variables correspond to your prompt_columns and response_columns.

CSV Data: 

user_input,system_instruction,llm_output
"What is 2+2?","Be helpful","The answer is 4"

Configuration: 

1
2
prompt_columns=["user_input", "system_instruction"]
response_columns=["llm_output"]

Example prompt.md: 

## Instructions:
Evaluate the LLM response for helpfulness.

## Context:
User Input: {user_input}
System Instruction: {system_instruction}

## Response to Evaluate:
{llm_output}

Formatted prompt given to Judge: 

## Instructions:
Evaluate the LLM response for helpfulness.

## Context:
User Input: What is 2+2?
System Instruction: Be helpful

## Response to Evaluate:
The answer is 4

CSV Data: 

text_to_evaluate
"This is a summary of the research paper findings."

Configuration: 

1
2
prompt_columns=None
response_columns=["text_to_evaluate"]

Example prompt.md: 

## Instructions:
Evaluate the quality of this summary.

## Text to Evaluate:
{text_to_evaluate}

Formatted prompt given to Judge: 

## Instructions:
Evaluate the quality of this summary.

## Text to Evaluate:
This is a summary of the research paper findings.

Task Schemas (task_schemas)

The task_schemas dictionary maps task names to their allowed outcomes:

1
2
3
4
5
6
7
8
9
task_schemas = {
    # Classification tasks (predefined options)
    "toxicity": ["toxic", "non_toxic"],
    "relevance": ["relevant", "irrelevant", "partially_relevant"],

    # Free-form tasks (open-ended responses)
    "explanation": None,
    "how_confident": None
}

Classification Tasks: Use a list of strings for predefined outcomes

  • Minimum 2 outcomes required
  • Judges must choose from these exact options
  • Examples: sentiment, safety, relevance ratings

Free-form Tasks: Use None for open-ended responses

  • No restrictions on judge responses
  • Examples: explanations, detailed feedback

Add Free-form Tasks for Context and Explainability

Include explanation fields to understand judge reasoning: 

1
2
3
4
task_schemas = {
    "is_helpful": ["helpful", "not_helpful"],
    "explanation": None  # Why this classification?
}

Multi-label Tasks (pick several)

Some tasks require selecting several labels at once for a single field (e.g. a response can be both hateful and insults). Wrap the outcomes in a MultiLabelSchema to declare a multi-label task:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from meta_evaluator.eval_task import EvalTask, MultiLabelSchema

task = EvalTask(
    task_schemas={
        "harm_types": MultiLabelSchema(
            outcomes=["hateful", "insults", "sexual", "violent"]
        ),
        "sentiment": ["positive", "negative"],  # still single-select
    },
    prompt_columns=["user_prompt"],
    response_columns=["chatbot_response"],
    answering_method="structured",  # or "instructor"
)

A multi-label task's value is a fixed-length ordered vector with one slot per declared outcome. Each slot holds either that outcome's own name (selected) or the reserved sentinel "FALSE" (not selected). For the example above, a response that is hateful and sexual (but not insulting or violent) is stored as:

["hateful", "FALSE", "sexual", "FALSE"]

Key rules:

  • Outcome order is load-bearing — it defines slot order and is preserved through serialization, judging, storage, and export.
  • outcomes must contain at least 2 values, and the exact string "FALSE" is reserved (it marks a not-selected slot) and cannot be declared as an outcome.
  • "Nothing applies" is fully defined — an all-"FALSE" vector, never an empty/missing value.
  • Answering methods: multi-label supports only structured and instructor. Direct answering_method="xml" raises an error, and XML is excluded from the fallback sequence (its scalar-only path cannot carry the ordered vector).
  • In the annotation interface, a multi-label task renders as a checkbox group; the number-key shortcut toggles a slot instead of replacing the selection.

Scoring multi-label tasks (see the Scoring guide for details):

  • Score with ClassificationScorer via task_strategy="single". The vector is binarized positionally (slot i → 1 if it holds outcome i's name, else 0) into a per-slot indicator vector.
  • For F1, precision, and recall, ClassificationScorer.average must be "macro" (per-label, recommended) or "samples" (per-item overlap). The global default "binary" is rejected. Accuracy ignores average and works with its default. "macro" is required for a mixed single-class + multi-label multitask configuration using an averaged metric.
  • AltTestScorer scores the native name vector unchanged (routes to jaccard).
  • CohensKappaScorer does not support multi-label tasks and raises a clear error — κ has no valid averaging axis over a sparse multi-label vector. Use ClassificationScorer or AltTestScorer instead.

Required Tasks (required_tasks)

The required_tasks parameter controls which tasks must be completed for a valid annotation or judge response.

Default Behavior (when required_tasks is not specified):

  • All classification tasks (non-None schemas) are required
  • All free-form tasks (None schemas) are not required
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Default behavior example
task = EvalTask(
    task_schemas={
        "safety": ["safe", "unsafe"],        # Required by default
        "helpfulness": ["helpful", "not_helpful"],  # Required by default
        "explanation": None,                 # NOT required by default (free-form)
        "notes": None                        # NOT required by default (free-form)
    },
    # required_tasks not specified - uses default behavior
    prompt_columns=["user_prompt"],
    response_columns=["chatbot_response"],
    answering_method="structured"
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Custom behavior example
task = EvalTask(
    task_schemas={
        "safety": ["safe", "unsafe"],
        "helpfulness": ["helpful", "not_helpful"],
        "explanation": None,  # Free-form
    },
    required_tasks=["safety"],  # Only `safety` required
    prompt_columns=["user_prompt"],
    response_columns=["chatbot_response"],
    answering_method="structured"
)

Impact on Annotation Interface:

In the annotation interface, required fields are marked with a red asterisk (*) and must be filled before the annotation is auto-saved.

Impact on Judge Results:

For judge evaluations, only the required tasks need to be successfully parsed for a result to be marked as successful.

Answer Parsing Methods (answering_method)

Three parsing methods with different trade-offs:

1
2
answering_method="structured"
structured_outputs_fallback=True  # Fallback to other methods if unsupported

Pros: Most reliable, cleanest parsing, best model support
Cons: Newer feature, not supported by all models
Best for: Production use with modern models

Fallback sequence (when structured_outputs_fallback=True): structured → instructor → xml

1
2
answering_method="instructor"
structured_outputs_fallback=True  # Fallback to other methods if unsupported

Pros: Good compatibility, structured validation
Cons: Additional dependency, model-specific implementation
Best for: When you need structured outputs with older models

Fallback sequence (when structured_outputs_fallback=True): instructor → structured → xml

1
2
answering_method="xml"
structured_outputs_fallback=True  # Fallback to other methods if unsupported

Pros: Universal compatibility, works with any model
Cons: More prone to parsing errors, verbose output
Best for: Maximum compatibility, legacy models

Fallback sequence: None

Enable Fallback for Production

Always enable fallback to maximise Judge completion.

Skip Function to Filter Data Rows (skip_function)

1
2
3
4
5
6
7
def skip_empty_responses(row):
    return len(row.get("llm_response", "").strip()) == 0

task = EvalTask(
    # ... other config ...
    skip_function=skip_empty_responses
)

Skip Function Serialization

Currently, skip functions are not saved when EvalTask is serialized/deserialized. When loading a saved project:

1
2
3
4
5
# Load existing project
evaluator = MetaEvaluator(project_dir="my_project", load=True)

# Skip function resets to default. You must reassign it after loading:
evaluator.eval_task.skip_function = skip_empty_responses

Annotation Prompt for Human Interface (annotation_prompt)

Customize the prompt shown to human annotators:

1
2
3
4
task = EvalTask(
    # ... other config ...
    annotation_prompt="Please evaluate this response for toxicity and helpfulness. Consider both content and tone."
)

Real-World Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
moderation_task = EvalTask(
    task_schemas={
        "toxicity": ["toxic", "borderline", "safe"],
        "harassment": ["harassment", "no_harassment"], 
        "violence": ["violent", "non_violent"],
        "explanation": None
    },
    prompt_columns=["user_message"],      # Original user input
    response_columns=["content_to_check"], # Content that might violate policy
    answering_method="structured",
    structured_outputs_fallback=True,
    annotation_prompt="Evaluate this content for policy violations."
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
conversation_task = EvalTask(
    task_schemas={
        "coherence": ["coherent", "somewhat_coherent", "incoherent"],
        "helpfulness": ["very_helpful", "helpful", "not_helpful"],
        "factuality": ["factual", "mostly_factual", "inaccurate"],
        "improvement_suggestions": None
    },
    prompt_columns=["conversation_history", "user_query"],
    response_columns=["assistant_response"],
    answering_method="structured",
    structured_outputs_fallback=True
)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
research_task = EvalTask(
    task_schemas={
        "methodology_quality": ["excellent", "good", "fair", "poor"],
        "novelty": ["highly_novel", "somewhat_novel", "incremental", "not_novel"],
        "clarity": ["very_clear", "clear", "unclear", "very_unclear"],
        "detailed_feedback": None,
    },
    prompt_columns=None,  # No prompt needed
    response_columns=["paper_abstract", "methodology_section"],
    answering_method="structured",
    structured_outputs_fallback=True,
    annotation_prompt="Please evaluate this research paper on methodology, novelty, and clarity. Provide detailed feedback."
)