Skip to main content

Measurement and Evaluation of Trustworthy Intelligent Systems

METIS

Specification-driven evaluation. Evaluate an application's performance against the system requirements that matter most.

Quick start​

  1. Install METISInstallation and Setup
  2. Prepare the target connection and LangfusePre-requisites
  3. Create a specificationUse the Wizard or configure manually
  4. Run the test suite, then run the gradersSimulation or Grading
  5. Analyse your resultsInsights

Run all commands from the repository root. To use your own evaluation tools, export the specification.

Components​

Wizard, Engine, and Insights are the three main tools. Web and API give access to them. Library and Core hold the shared parts.

WebCreate and export specifications in a browser.apps/web
APIConnect the web client and other clients to METIS.apps/api
Create a specificationDescribe your application in a guided Interview, or write the specification yourself.
Run an evaluationSimulate conversations with your target, then grade each trial.
Create a reportRead pass rates, grader reasons, and trial evidence.
LibraryProvide shared graders and reusable components.packages/library
CoreDefine shared models, schemas, and interfaces.packages/core
TestbedCheck METIS components against reference cases.packages/testbed

The map groups the components by purpose. For package dependencies, read the architecture specification in the repo.