Skip to main content

Write your own specification

specification.yaml is the file that the Engine runs. The guided interview writes it for you. If you already know the tests and the graders to run, write it in the manual form in the browser, or write the YAML yourself. This page shows both ways, and lists every field of the file.

Fill in the manual form​

The manual form is in the browser only. To open it, start the Wizard and select Skip the interview and write it yourself.

A Test Case in the manual form, with one grader selected

To connect a target, select Build Spec and connect target at the top of the form. See Target connection.

Download specification gives a ZIP file, for example 20261016-223045-manual.zip. It holds outputs/specification.yaml, the files under outputs/fixtures/, and outputs/target-binding.yaml when you connect a target. The form writes each Test Case as one Task, with one Behaviour that holds a Criterion for each grader.

Write the file yourself​

Make a folder with this layout:

experiments/design/research-assistant/outputs/
├── specification.yaml
├── target-binding.yaml (only for the HTTP or browser runner)
└── fixtures/
└── leave-policy.pdf (the files that the Tasks send)

This specification.yaml tests one Behaviour of the research assistant with two graders:

schema_version: 1
id: research-assistant
runner_factory_ref: metis.engine.simulation.runners.http.HttpRunnerFactory
target_binding: target-binding.yaml
target:
name: Policy research assistant
users: Policy staff
description: Answers questions about internal policy documents and cites the source of each answer.
features:
- id: F-1
name: Policy research
description: Answer policy questions from the shared policy documents.
behaviours:
- id: EB-1
name: Cite a source in each answer
description: Each answer cites the policy document that supports it.
measured: true
criteria:
- id: AC-1-1
name: Citations support the answer
description: Each citation points to a passage that supports its claim.
grader_fqn: metis.library.graders.AttributionAccuracyGrader
grader_args: {}
- id: AC-1-2
name: Cites a source for each claim
description: The answer cites a source for each policy claim.
grader_fqn: metis.library.graders.CriterionJudgeGrader
grader_args:
threshold: 0.5
personas:
- id: policy-analyst
name: Policy analyst
description: A policy analyst who checks how a new rule applies to their team.
metadata:
role: analyst
scenarios:
- id: part-time-leave
name: Leave for part-time staff
description: Ask how the new leave policy applies to a part-time employee.
tasks:
- id: TC-001
scenario_ref: part-time-leave
persona_ref: policy-analyst
behaviour_refs: [EB-1]
num_trials: 3
fixtures: [leave-policy.pdf]

Run it with metis engine run --input-dir experiments/design/research-assistant/outputs/. Before the run starts, the Engine checks the file, the fixture names, and the settings of each grader. If a check fails, the Engine names the error and does not start the run.

The JSON schema of the file is packages/core/schemas/specification.schema.json. Your editor can use it to check the file as you type.

Field reference​

Top level​

FieldRequiredMeaning
schema_versionYesAlways 1.
idYesThe ID of the experiment. The Engine uses it in the name of the output folder.
runner_factory_refYesThe runner that talks to your application. See Choose the runner.
target_bindingNoThe target connection file, relative to specification.yaml.
defaultsNoThe models for the run. See Defaults.
targetYesThe application that you test.
featuresYes, to runThe Features, Behaviours, and Criteria to measure.
personasYes, to runThe simulated users.
scenariosYes, to runThe situations to test.
tasksYes, to runThe Test Cases.
fixture_content_descriptionsNoA short description of each fixture, by file name. The Engine writes one for each fixture that has none.

Choose the runner​

Your applicationrunner_factory_reftarget_binding
An HTTP APImetis.engine.simulation.runners.http.HttpRunnerFactoryRequired
A web pagemetis.engine.simulation.runners.browser.BrowserRunnerFactoryRequired

When runner_factory_ref is not set, it defaults to the HTTP runner. To write a runner for your own application, see the engine guide.

target​

FieldRequiredMeaning
nameYesThe name of the application.
usersYesWho uses the application.
descriptionYesWhat the application does, in one or two sentences.
detailNoMore context, such as rules that the application must follow.

features, behaviours, and criteria​

Each Feature holds Behaviours, and each Behaviour holds Criteria. Every item has an id, a name, and a description.

FieldOnMeaning
behavioursFeatureThe Behaviours of the Feature.
measuredBehaviourtrue to measure the Behaviour. Default false.
criteriaBehaviourThe checks for the Behaviour.
grader_fqnCriterionThe full Python name of the grader, for example metis.library.graders.CriterionJudgeGrader.
grader_argsCriterionThe settings of the grader, such as threshold. {} keeps the defaults.

The description of a Criterion is the instruction for the criterion judge.

personas and scenarios​

FieldRequiredMeaning
idYesThe ID that the Tasks use.
nameYesThe display name.
descriptionYesFor a Persona, how the user acts. For a Scenario, the situation.
metadataNoExtra details as names and text values, for example role: analyst.

tasks​

FieldDefaultMeaning
idRequiredThe ID of the Test Case.
persona_refRequiredThe id of a Persona.
scenario_refRequiredThe id of a Scenario.
behaviour_refs[]The id of each Behaviour to check. Each Behaviour supplies all its Criteria.
num_trials3How many times to run the Test Case.
fixtures[]The files to send, by name, from fixtures/.
max_turns20The most turns in one Trial. The Trial stops at this number.
timeout_minutes15The time limit for one Trial.
response_wait_ms300000How long to wait for each reply, in milliseconds.
max_run_retriesNoneHow many times to retry a Trial that fails to run.
user_agent_modelNoneThe model of the simulated user for this Test Case.

defaults​

FieldMeaning
user_agent_modelThe model of the simulated user, for every Test Case.
judge_modelThe model of every LLM judge.
runner_agent_modelThe model that drives the HTTP or browser runner.

A field that you leave out uses the value in .env. See the model guide.

Next step​

To run the tests against your application, set up the target connection.