Evaluation

We test AI where conventional benchmarks stop.

We build environments, rubrics and training systems for measuring how AI behaves when it has to actually do things: investigate a problem, decide with incomplete information, act, and sometimes recognise that it should stop.

Test AI by putting it somewhere it has to actually do something.

See an evaluation Test your AI

The platform

Four parts, one loop.

What we test Rubric Catalog Real-world situations and the expert-authored criteria they are graded against — written by people who were there when it went wrong. Rubrics → Where we test Simulation Raccoon Ridge, a simulated town with businesses, roles and consequences, so a decision happens somewhere instead of in a vacuum. Simulation → How we develop Training Matrix Controlled training where every change is tied back to an evaluation that can show whether it helped. Training → What we discover next Hive Mind A sensor on AI-to-AI interaction that flags new behaviours and failure modes as candidates for a rubric — reviewed by a person first. Hive Mind →

Why this matters

A correct answer is not a correct decision.

Static benchmarks measure what a model knows when it is asked. Deployed agents fail somewhere else: in the gap between knowing a fact and noticing that it applies, in information that arrives late or contradicts itself, and in the pressure to close the ticket.

Conventional benchmark

  1. Ask a question
  2. Measure the answer
  3. Repeat

SGAIL evaluation

  1. Give the AI a situation and an environment
  2. Let it act
  3. Inject incomplete, conflicting, misleading or changing information
  4. Observe the decisions
  5. Measure the consequences
  6. Grade against an expert-authored rubric
  7. Record new failure modes
  8. Feed validated findings into controlled training

This is not a solved problem. What we build is the infrastructure and the method for this kind of evaluation — and the honest status of each piece is marked on the page that describes it.

How it works

The loop.

Everything on this site sits somewhere on one cycle. A failure in the real world becomes a rubric; the rubric runs in the simulation; the run is evaluated and observed; what is learned goes into controlled training; the changed behaviour is tested again — and anything new it does becomes the next rubric.

The SGAIL loop: real world, rubric, simulation, evaluation, observation, training, new behaviour, new rubric, and back to the real world. REAL WORLDwhere failures happen RUBRICRubric Catalog SIMULATIONRaccoon Ridge EVALUATIONthe run, graded OBSERVATIONHive Mind TRAININGTraining Matrix NEW BEHAVIORre-tested, not assumed NEW RUBRICback into the catalog Each pass can end in a new test.
  1. Real world — a failure someone actually stood in front of.
  2. Rubric — written up as a situation with expert criteria. Rubric Catalog
  3. Simulation — run inside an environment with context and consequences. Raccoon Ridge
  4. Evaluation — decisions graded against the rubric.
  5. Observation — unexpected behaviour flagged for review. Hive Mind
  6. Training — validated findings enter controlled training. Training Matrix
  7. New behavior — re-tested, not assumed.
  8. New rubric — what it does next becomes the next test.
Worked through once, step by step: see an evaluation →

Who uses it

Two kinds of team, one question: will it hold up?

For AI developers

You are shipping an agent and want to know where it breaks before your users find out.

  • Agent evaluation — scenario runs against situations your model has not seen
  • Failure discovery — a written account of where it went, and where it should have
  • Regression testing — re-runs draw fresh variations, so a fix has to be real
  • Red teaming — adversarial pressure drawn from people who do it for a living

Starts with one free run. Paid audits and drill packs are priced on the evaluation page.

Evaluate your AI

For enterprises & government

You are deploying, buying or regulating AI and need evidence of operational behaviour, not a leaderboard position.

  • Operational AI assessment — evaluated against the disciplines your deployment touches
  • Deployment readiness — does it stop, escalate and ask when it should
  • Continuous evaluation — scheduled or per-build runs API in development
  • Procurement evidence — results tied to a tamper-evident log by arrangement

Training

If a behavior matters, we should be able to test it.

The Training Matrix is how evaluation results become development: controlled material, a hard wall between what trains and what tests, and every change measured against the evaluation that motivated it. Parts are built; the training runs themselves are under development — the page says which is which.

Training Matrix →Facilitated training slots →

The record underneath

An evaluation is only as good as its record.

Runs are written to the same tamper-evident witness log we build for farms, evidence rooms and AI agent machines, so a result can be checked rather than asserted. Research that is still open has its own page.

The Witness →Research →

For contributors

Know the work? Trade it in.

The hard material comes from people who were there when it went wrong — electricians, plumbers, landlords, red teamers. If that is you, there is a door for it.

Trade in →See who's in →Not a writer? →