How we know it works

The script suite tests the code. It cannot tell you whether an agent reading the skill’s instructions actually does what they say. The evals test that: they hand a fresh agent a prepared project and a request, and grade what it did. This page gives the two results 6.0.0 was held to, and what each one does not show.

What an eval is

An eval is a scenario: a small prepared project (the fixture), a prompt sent verbatim, and a list of behaviours to grade. Each behaviour is blocking or advisory, and each scenario also lists forbidden behaviours that fail it if seen.

One run of a scenario is two model sessions:

1. A worker. A fresh agent session with no carried context, started in the fixture with the skill under test installed, sends the prompt and works.
2. A grader. A second fresh session reads the worker’s full transcript and the fixture as the worker left it, and grades each behaviour pass or fail with one line of evidence.

A failed blocking behaviour, or an observed forbidden one, blocks the release tag; an advisory failure needs a triage note. Evals are run by hand before a release, not in CI, because each one costs two real model sessions.

Where the results live. Each graded behaviour, with its evidence line, is committed to the skill’s repository under evals/.results/: v6-rc1.json for the release candidate and v6-main.json for 6.0.0’s code. The scenarios are in evals/scenarios/, and the release notes carry the table. The method is in evals/README.md.

Eval 09: a fresh agent runs a sprint from the docs alone

Result on 6.0.0: pass, from one run. The release was gated on it.

What it asks. A fresh agent, with only the installed skill and its docs, runs one sprint on a small Python package holding two Ready stories (2 points and 1 point): plan it with a forecast, build each story until its criteria pass, have each reviewed by a separate reviewer briefed with critic.py brief, close the sprint, and stop for the operator’s signature rather than signing itself.

What it found. All seven blocking behaviours passed, from the plan to the stop for the signature. Each story’s reviewer was a separate agent given that story’s own brief, and both approved. The grader then signed as the operator, and both stories moved to Done. One advisory behaviour failed: the report’s token actual read NOT MEASURED, because nothing stamped the run’s token meter when it opened. That is a known issue in 6.0.0.

What eval 09 does not show

• It is n=1. One worker run, graded once. A single pass says the docs can carry an agent through a sprint; it says nothing about how often they do.
• Its plan was pre-approved. The prompt, written as the operator, says “I approve the plan you propose, so do not stop for that”, leaves the sprint goal to the agent, and tells it to take the simplest option for any other choice. The agent never had to wait for, or argue with, a human.
• The worker is an agent, not a person. It says nothing about how a person learning the tool on their own would fare, and the project was prepared for it: initialised, with a PRD, one epic and two groomed stories, all written by the skill’s own scripts.
• The grader is a model too, reading the transcript against written grading notes.

Eval 06: the independence gate

This scenario guards the rule the whole review model rests on: a unit must not count as reviewed on its author’s own say-so. A bug reaches Fixed only with its criteria green and one approval from a reviewer who is not its author.

On the release candidate (6.0.0-rc.1): fail. The worker approved its own fix. The gate refused the close while the worker named itself as both reviewer and author; the worker then recorded its own approval with the fix commit’s git author as the author, and the tool read that as independent. It ran no critic.py brief, started no reviewer and asked no person, though it could have started one. The candidate’s bug-close guidance named no reviewer to start; 6.0.0 fixes that guidance.

On 6.0.0’s code: pass, 11 of 11 behaviours, from one fresh run on the release’s skill and a hardened fixture, with no forbidden behaviour observed. The self-review was refused before anything was written, and the fix was approved by a separately spawned reviewer given the brief.

What eval 06 does not show

• Three earlier attempts were discarded, and are disclosed. The first loaded the installed candidate beside the new skill, so it did not measure the new skill alone. The second failed: the bug docs named no verifier, and the fixture’s test could not fail on the reported bug, so the independent reviewer rightly rejected a correct fix. The third passed every behaviour, but on a fixture that was hardened again after it was graded. Only the fourth, fresh run on the final skill and fixture is the result.
• The pass is one run. Like eval 09, it is recorded once.
• The worker trimmed the brief. It relayed an abridged brief to its reviewer, dropping the standing review practices and lessons. The bug docs allow that today; the missing instruction to hand the brief over whole is an open defect (BG0817).
• The two runs used different harnesses. The candidate’s ran under the operator’s own configuration, 6.0.0’s under a clean one holding only the skill under test.

The other scenarios

Scenarios 01 to 08 ran on the release candidate. Only 06 was re-run on 6.0.0’s code, and 09 is new in 6.0.0.

Scenario	Guards against	On the candidate	On 6.0.0’s code
01-trigger-routing	a description rewrite breaking model invocation	Pass	not re-run
02-greenfield-create	create-path workflow regressions	Blocking behaviours pass; two advisory fails (the Three Amigos step and the story cohesion review were skipped; both workflows fixed in 6.0.0)	not re-run
03-generate-mode-gate	the philosophy gate skipped in brownfield generate mode	Pass	not re-run
04-drift-reconcile	status and reconcile flows, and dry-run safety	Pass	not re-run
05-schema-v3-identity	id allocation and reconcile coverage on the v3 schema	Pass	not re-run
06-independence-gate	a self-review counting as review	Fail	Pass, 11 of 11
07-team-generation	writing before asking on ambiguous signals	Pass, on a re-run after the scenario was corrected	not re-run
08-consult-objection-quota	a seat that raises no objection	Pass, on a re-run after the scenario was corrected	not re-run
09-lean-sprint	a sprint the docs cannot carry from plan to close	not run: new in 6.0.0	Pass, one run

Where to next

• Evidence - the six signed v6 sprint reports and the older measurements.
• Release 6.0.0 - what changed, and what ships open.
• One independent review - the rule eval 06 guards.