Evidence

Everything on this page is in the skill’s repository at the v6.0.0 tag, including the findings that do not flatter. Each figure names the signed report (sdlc-studio/reports/RPTxxxx) or the file it was read from, so you can check it rather than take it.

The six v6 runs, 23 to 29 September 2026

Each row is one sprint of this repository on the v6 loop, read from its signed sprint report. The verdict records whether the goal was met, judged by the batch units’ own acceptance criteria and recorded once at the close as the goal verdict (D0253, D0254); no check judges the goal sentence itself, though a plan refuses a goal over 20 words. Hours are the wall-clock span of the run, so waiting counts. Tokens are the run’s actual spend over the plan’s forecast.

Report	Sprint goal	Verdict	Points	Hours	Tokens vs forecast	Persona rulings	Operator rulings
RPT0006	A sprint runs start to finish on its own, learns from itself, and hands you one page.	Judged achieved	58	8.7h	1.14x	4	0
RPT0007	Commits clear in about a minute, and the sprint learns from repeated mistakes without being told.	Judged partial	42	5.4h	0.5x	2	0
RPT0008	A commit clears in ninety seconds, and every commit lane left standing shows what it caught.	Judged partial	32	7.2h	1.03x	3	0
RPT0009	Maya's units are reviewed once against their files' history, and the review paperwork nobody reads is deleted.	Judged partial	111	7.0h	0.51x	4	0
RPT0010	Maya runs the lean loop on a fresh v6 project, and the release candidate ships.	Judged achieved	102	27.3h	0.53x	2	0
RPT0011	v6.0.0 ships: Maya and Jonah install, upgrade and learn it from docs and notes that match the code.	Judged achieved	131	29.5h	0.5x	0	8

What the table says, and what it does not:

Who answered the questions. In the first five runs every question was ruled by a persona seat, 15 in all, and none went to the operator. The release run, RPT0011, was the reverse: no persona ruling and 8 operator rulings. A v6 run still comes to you when a question is yours to answer.
The token forecast ran high. Four of the six runs spent about half of what the plan forecast; the other two came in at 1.03x and 1.14x. A forecast that misses by half in the safe direction is still a miss. The minute forecast, new in these runs, missed by between 0.59x and 2.51x (RPT0007-RPT0011; RPT0006 recorded no minute forecast).
What the token actual counts. Each report’s actual is the main-thread meter plus the totals its delegated agents reported: 21 agents in RPT0006, 100 in RPT0011. Spend that was never tagged to the run is not in it.
Points are delivered points. RPT0008 and RPT0009 were judged partial and dropped units: 5 of 21 planned and 3 of 30 planned, most of them carried at the review cap.

RPT0011: the release run’s review rounds

RPT0011 delivered 55 units, each reviewed by one reviewer who did not write it. 31 were approved in the first round. 24 reached a second round, and one of those, BG0818, went to a third round the operator granted past the two-round cap. BG0818 was first carried at the cap, as the cap requires, with its round-two findings filed as BG0840; the operator then granted one bounded third round for that unit alone (D0285), the same reviewer approved the repair, and BG0818 rejoined the batch and closed (D0286). The tool had no path for a granted extra round, so it landed by force; BG0850 closed that gap in 6.1. BG0840 stays open for its non-blocking residuals.

Across all six runs, counted from the Review rounds column of each report’s Delivered to plan table (a unit dropped before review is not counted):

Report	Units reviewed	Went to a second round	Went to a third round	Rejected at the cap	Stayed carried
RPT0006	11	9	0	0	0
RPT0007	11	4	0	0	0
RPT0008	20	12	0	4	4
RPT0009	37	14	0	2	2
RPT0010	39	19	0	2	2
RPT0011	55	24	1	1	0

That is 173 units, 82 of them reviewed at least twice and one three times. 9 were still rejected at the cap and had their findings filed as bugs; 8 of them stayed carried out of their run, and the ninth, BG0818, landed on the granted third round. Each run went on to its close. Review has not got easier: roughly half of all units still need a second round. What the cap changed is the tail, and the one unit that went past it did so on the operator’s say-so, recorded as a decision.

Before v6: the older measurements

These were taken before the v6 loop existed. Each heading carries the month it was measured; read them as the history that argued for v6, not as figures about it.

The review baseline under v5’s ceremony, July 2026

Under the per-unit ceremony v5 carried, review rounds on this project ran at 62% REJECT, and one run spent 33.76 hours of which 25.21 - 75% of the run - was review, across 18 rounds. It was published in the change request that argued the ceremony was disproportionate to blast radius (CR0510, filed 2026-07-31), and it is part of why v6 caps review at two rounds per unit.

This counts rounds; the v6 table above counts units. The two are not like for like, and together they do not show review getting cheaper per unit.

Where the review runs, July 2026

One measured run (CR0500, filed 2026-07-29) delivered in five hours and took six hours thirty-five minutes to close. Gate and test time across the whole close was about 70 minutes - roughly 18%. The rest was repair generated by a review that ran at the close rather than inside delivery. A review of those repairs then found nine more surviving mutants, because the repairs had been written fast, late and self-reviewed.

That measurement is why the review now runs inside delivery, one unit at a time as each lands.

The v4 benchmark, 10 July 2026

The rerun of the frozen fixtures against the v4 release candidate, across three model tiers, with a mandated-arm addendum (report).

The trap. The headline fixture is a notification service whose specification contains a rule: nothing non-urgent is delivered during a user’s quiet hours, and a suppressed delivery is deferred, never dropped. The ticket asks for an everyday feature - digest mode - and never mentions quiet hours. A digest is still a delivery, so the existing rule applies to the new feature, and the obvious implementation flushes a batch at 23:00 and wakes the user. The benchmark asks one question: does anything force the agent to re-read the spec it was given?

The engagement floor. Three arms, same fixture, same models. The middle arm had the full pipeline installed and available, plus judgement about when to use it.

Defects escaped	Judgement-gated	No process	Mandated planning
Mid-tier model	5/5	2/5	1/5
Premium model	3/3	3/3	0/5
Frontier model	0/5	0/5	not run

The judgement-gated arm behaved exactly like having no process at all. Under effort pressure it judged the ticket too small for ceremony, implemented the ticket text directly, went green on its own tests, and shipped the same defect. One premium run named the dangerous interaction “the one genuinely ambiguous edge” in its own delivery summary and shipped without resolving it. That result is why this toolchain gates rather than advises:

A process step gated on judgement is the step that gets skipped.

An earlier study on the previous model generation (8 July 2026, n=5) found 10 of 10 unstructured runs shipping the same defect, with an auditability score of 0.88 for the governed arm against 0.60 for the baseline.

What the discipline cost.

Approach	Tokens	Cost per ticket	Escapes	Audit trail
Frontier, no process	54.6k	$0.98	0/5	none
Frontier, full pipeline	77.2k	$1.39	0/5	full
Premium, no process	54.0k	$0.49	3/3	none
Premium, mandated	57.9k	$0.52	0/5	full
Mid-tier, no process	67.1k	$0.24-0.36	2/5	none
Mid-tier, mandated	79.2k	$0.29-0.43	1/5	full

Mandated process cost 1.07 to 1.18x the baseline tokens - a premium of three to seven cents a ticket. On the premium model that took the escape rate from 3/3 to 0/5 for three cents. Enforced process on a mid-tier model undercut frontier prompt-and-hope by 47 to 70% while producing an evidence trail the frontier run does not produce at any price. The worst value measured was a premium model with no process: it cost more than the governed run beneath it and shipped the defect every time.

A story point is a unit of cost, July 2026

Twenty-one delivered units from four earlier sprints of this repository, re-estimated blind by three independent estimators with no access to the outcomes; nineteen had measured actuals (RFC0038, 2026-07-14). The actuals are per-unit build costs, with no review rounds or close in them. The table gives each points band’s tokens per point; individual units scatter round it.

Band	Tokens per point
2 points	22,370
3 points	26,153
5 points	27,396
8 points	25,171
13 points	14,144 - 1.9x cheaper per point

Flat from 2 to 8, and then it breaks. The thirteens were systematically over-estimated, and all three estimators returned them with low confidence and the unprompted words “should be split”. That is the measurement behind the 8-point split threshold. The model it gave is cost = points x ~25,000 tokens, flat: with no fitting at all, 12 of the 19 units fell within 0.75x-1.25x of it, and adding a fixed base term to the fit made it slightly worse (11 of 19).

Story points predicted cost; a computed metric did not. Points scored r = +0.68 pooled against measured cost, and +0.78 on units of 8 points or fewer. A code-complexity metric scored r = 0.03 and was dropped.

Decomposition improved the estimate, not only the tracking. Four large requests sized in one shot came in at 0.56x their forecast with a 2.3x spread. Decomposed first, their summed points against the same unchanged rate predicted the batch to 1.00x (lesson LL0038).

A run has a fixed cost, July 2026

Fitting two of this project’s sprints, measured whole, as fixed plus marginal times points gave a marginal of about 13,100 tokens per point and a fixed of about 3.88 million per sprint (CR0391, filed 2026-07-22). An 18-point run measured at least 228,884 tokens per point; a 100-point run at least 51,945. That is 5.5 times the work for 1.26 times the tokens.

Under-forecasting is worst at small batch sizes, which is the opposite of the usual intuition. The model misleads most exactly when a team is being careful and keeping batches small.

The audit’s own cost, July 2026

The pre-flight estimator is calibrated against a measured reference run: 7 lenses produced 57 candidates across 192 agents, about 6.9M tokens and about 29 minutes. It is an order-of-magnitude guide, not a promise. The run of 27 July 2026 estimated 209 agents and 7.5M tokens and spent 282 agents and 14.26M - a 1.9x overrun, because 112 candidates turned up against an estimate of 64. Both numbers are in the committed ledger (sdlc-studio/retros/evidence/audit-cost-2026-07-27.jsonl), which is the only reason the seeds improve.

What none of this proves

The v6 runs are one project’s own

All six reports are this repository building its own tool, signed by its own operator. They are not a controlled comparison, they have no baseline arm, and they say nothing yet about a project that is not this one. The evals are the closest thing to an outside check, and each of them is one run.

The benchmark is small

Five runs per cell, three for the premium model, and two fixtures carry all the signal. The 8 July study’s headline 5-of-5 against 2-of-5 has a one-sided Fisher exact p of 0.083 - the direction is consistent, the result is below conventional significance. The rubric is post-hoc and imperfectly blinded. Two escapes in the published results table turned out to be harness artefacts; they were disclosed and left in, because the oracle is the oracle.

The token actuals were lower bounds before v6

Before 6.0.0 the meter was the session transcript, and it recorded no subagent usage at all: one sprint published a figure that understated its known cost by 64% (RETRO0065). The v6 reports add the totals each delegated agent reported, as the table above says; spend never tagged to a run is still missing from it, and on a fresh project the token actual can read NOT MEASURED (a known issue in 6.0.0).

The forecast rate, before and after 6.0.0

Measured tokens per point across v5-era sprints spanned 21,905 to 151,701 - a 6.9x range - while the planner forecast with a single constant at the bottom of it (CR0510). Since 6.0.0 the rate follows the newest runs, and estimate.tokens_per_point in .config.yaml sets it by hand; the plan names the override and the measured rate it replaced. The six v6 runs came in at 0.5x to 1.14x of their forecast, so the error now runs the other way. Rates cannot be pooled across models, and several recent sprints were delivered by more than one, so they carry no rate at all.

Grooming is not priced

Grooming a freshly refined story is real work, and story points do not price it.

There is no recommended batch size

The fixed cost pulls towards larger batches; review convergence cost pulls towards smaller ones. Both directions are visible in the measured history, the sample is small and noisy, and inventing a number would repeat a mistake this project has twice had to undo - writing a rate down as a constant until it hardened into an article of faith.

The field results are uncontrolled

A maintenance deliverable estimated at twelve months - thirty production websites - delivered in under seven days. Features estimated at a team of five for twenty weeks, delivered in under one. These are operator-reported, not benchmarked. Treat them as testimony.

And the finding that cuts against the pitch

On the frontier model in the July benchmark the traps did not bite in any arm - 30 clean runs out of 30, and the overhead for the full pipeline was around 1.4x rather than the 3.1x the 8 July study measured. If your whole fleet is frontier and your tickets look like this fixture, the discipline buys you the audit trail rather than the defect reduction.

The honest reading of the whole benchmark is narrower and more useful than the headline: judgement-gated process converges on the model’s unaided behaviour exactly when the model most needs the process.

Where to next

How we know it works - the eval scenarios run against 6.0.0, and what each one does not show.
Release 6.0.0 - what changed, and the 35 open defects it ships with.
The white paper - the full study, its protocol and its claims register.
How it compares - where the tool sits in the category.