Skip to main content

Run the test suite

metis engine simulate runs your test cases against your application. For each trial, a simulated user talks to the application through the runner, and METIS records the result. It does not grade anything. METIS must first be able to reach your application. See Expose an API or web UI.

Start a run​

Terminal
1metis engine simulate --input-dir my-evaluation

--input-dir is the folder with specification.yaml, the target connection that target_binding: names, and fixtures/. See Run your evaluations.

When it finishes, the command prints the output folder, the evaluate command for it, and the --resume command. Copy them from the output instead of typing the timestamp.

Run trials in parallel​

Terminal
1metis engine simulate --input-dir my-evaluation --concurrency 4

--concurrency (short -c) sets how many trials run at once. The default is 1. Raise it only as far as your application can handle, and check any rate limits in your target connection.

Resume a stopped run​

If a run stops before all trials finish, continue the same output folder:

Terminal
1metis engine simulate --resume --experiment-dir experiments/execute/<your-run>

Give both flags together. Resume reads the specification from the frozen inputs/ folder in that output folder, not from your source folder, and it retries only the trials that are missing or failed with a system error. Do not combine --resume with --input-dir.

Simulate and evaluate together​

metis engine run does both steps, and also takes --resume --experiment-dir:

Terminal
1metis engine run --input-dir my-evaluation
2metis engine run --resume --experiment-dir experiments/execute/<your-run>

Other options​

OptionMeaning
--no-tuiPrint plain log lines instead of the progress screen
--verbose, -vShow more log detail. Repeat it for more
--quietShow errors only
--log-formattext (default) or json for the log files
--log-dirWrite the logs to another folder

Outputs​

experiments/execute/<your-run>/
├── inputs/
│ ├── specification.yaml
│ └── fixtures/
├── outputs/
│ └── trials/
│ └── <trial>/
│ ├── record.json
│ ├── evidence/
│ └── artifacts/
└── logs/
└── simulate.log

Next step​

Run the graders.