Evidence
This page exists because a paper about verifiable delivery that cannot show its own working would be a poor start. Everything here is published in the repository, including the findings that do not flatter.
The trap
Section titled “The trap”The headline fixture is a notification service whose specification contains a rule: nothing non-urgent is delivered during a user’s quiet hours, and a suppressed delivery is deferred, never dropped.
The ticket asks for an everyday feature - digest mode - and never mentions quiet hours. That silence is the trap. A digest is still a delivery, so the existing rule applies to the new feature, and the obvious implementation flushes a batch at 23:00 and wakes the user.
This is the classic shape of a production regression: not a bug in the new code, but an interaction between a new requirement and an existing one nobody re-read. So the benchmark asks exactly one question: does anything force the agent to re-read the spec it was given?
The engagement floor
Section titled “The engagement floor”Three arms, same fixture, same models. The middle arm had the full pipeline installed and available, plus judgement about when to use it.
| Defects escaped | Judgement-gated | No process | Mandated planning |
|---|---|---|---|
| Mid-tier model | 5/5 | 2/5 | 1/5 |
| Premium model | 3/3 | 3/3 | 0/5 |
| Frontier model | 0/5 | 0/5 | not run |
The judgement-gated arm behaved exactly like having no process at all. Under effort pressure it judged the ticket too small for ceremony, implemented the ticket text directly, went green on its own tests, and shipped the same defect. One premium run named the dangerous interaction “the one genuinely ambiguous edge” in its own delivery summary and shipped without resolving it.
That result is why this toolchain gates rather than advises:
A process step gated on judgement is the step that gets skipped.
An earlier study at n=5 found 10 of 10 unstructured runs shipping the same defect, with an auditability score of 0.88 for the governed arm against 0.60 for the baseline.
What the discipline costs
Section titled “What the discipline costs”| Approach | Tokens | Cost per ticket | Escapes | Audit trail |
|---|---|---|---|---|
| Frontier, no process | 54.6k | $0.98 | 0/5 | none |
| Frontier, full pipeline | 77.2k | $1.39 | 0/5 | full |
| Premium, no process | 54.0k | $0.49 | 3/3 | none |
| Premium, mandated | 57.9k | $0.52 | 0/5 | full |
| Mid-tier, no process | 67.1k | $0.24 | 2/5 | none |
| Mid-tier, mandated | 79.2k | $0.29 | 1/5 | full |
Mandated process costs 1.07 to 1.18x the baseline tokens - a premium of three to seven cents a ticket. On the premium model that took the escape rate from 3/3 to 0/5 for three cents.
Enforced process on a mid-tier model undercuts frontier prompt-and-hope by 47 to 70% while producing an evidence trail the frontier run does not produce at any price. The worst value measured was a premium model with no process: it costs more than the governed run beneath it and shipped the defect every time.
A story point is a unit of cost
Section titled “A story point is a unit of cost”Twenty-one delivered units, re-estimated blind by three independent estimators with no access to the outcomes. Nineteen had measured actuals.
| Band | Tokens per point |
|---|---|
| 2 points | 22,370 |
| 3 points | 26,153 |
| 5 points | 27,396 |
| 8 points | 25,171 |
| 13 points | 14,144 - 1.9x cheaper per point |
Flat from 2 to 8, and then it breaks. The thirteens are systematically over-estimated, and all three estimators returned them with low confidence and the unprompted words “should be split”. That is the measurement behind the 8-point split threshold.
The model is cost = points x ~25,000 tokens, flat. Adding a fixed base term to the fit made it slightly worse.
Story points predict cost; a computed metric does not. Points scored r = +0.68 pooled against measured cost, and +0.78 on units of 8 points or fewer. A code-complexity metric built over two days scored r = 0.03 and was dropped.
Decomposition improves the estimate, not only the tracking. Four large requests sized in one shot came in at 0.56x their forecast with a 2.3x spread. Decomposed first, their summed points against the same unchanged rate predicted the batch to 1.00x.
A run has a fixed cost
Section titled “A run has a fixed cost”Fitting this project’s own measured sprints as fixed plus marginal times points gives a marginal of about 13,100 tokens per point and a fixed of about 3.88 million per sprint.
An 18-point run measured at least 228,884 tokens per point. A 100-point run measured at least 51,945. That is 5.5 times the work for 1.26 times the tokens.
Under-forecasting is worst at small batch sizes, which is the opposite of the usual intuition. The model misleads most exactly when a team is being careful and keeping batches small.
Where the review runs
Section titled “Where the review runs”One measured run delivered in five hours and took six hours thirty-five minutes to close. Gate and test time across the whole close was about 70 minutes - roughly 18%. The rest was repair generated by a review that ran at the close rather than at the batch boundary.
A review of those repairs then found nine more surviving mutants, because the repairs had been written fast, late and self-reviewed.
That single measurement is why the adversarial review moved to the batch boundary.
The audit’s own cost
Section titled “The audit’s own cost”The pre-flight estimator is calibrated against a measured reference run: 7 lenses produced 57 candidates across 192 agents, about 6.9M tokens and about 29 minutes.
It is an order-of-magnitude guide, not a promise. One recorded run estimated 209 agents and 7.5M tokens and spent 282 agents and 14.26M - a 1.9x overrun, because 112 candidates turned up against an estimate of 64. Both numbers are in the committed ledger, which is the only reason the seeds improve.
What none of this proves
Section titled “What none of this proves”The benchmark is small
Section titled “The benchmark is small”Five runs per cell, three for the premium model, and two fixtures carry all the signal. The headline 5-of-5 against 2-of-5 has a one-sided Fisher exact p of 0.083 - the direction is consistent, the result is below conventional significance. The rubric is post-hoc and imperfectly blinded. Two escapes in the published results table turned out to be harness artefacts; they were disclosed and left in, because the oracle is the oracle.
The token actuals are lower bounds
Section titled “The token actuals are lower bounds”The meter is the session transcript, and it records no subagent usage at all. One sprint published a figure that understated its true cost by 64%. Any sprint that delegated work to agents cost more than its row says.
The 25,000 rate systematically under-prices
Section titled “The 25,000 rate systematically under-prices”Measured tokens per point across real sprints spans 21,905 to 151,701 - a 6.9x range - while the planner forecasts with a single constant at the bottom of it. Nothing auto-recalibrates; a human reads the trend and decides. Rates cannot be pooled across models, and several recent sprints were delivered by more than one, so they carry no rate at all.
The process has a real cost
Section titled “The process has a real cost”Review rounds on this project run at 62% REJECT. One run spent 33.7 hours of which 25.2 - 75% of the run - was review, across 18 rounds. Grooming a freshly refined story is real work and story points do not price it. There is an open change request arguing that the ceremony is disproportionate to blast radius, and it is in the public backlog rather than resolved quietly.
There is no recommended batch size
Section titled “There is no recommended batch size”The fixed cost pulls towards larger batches; review convergence cost pulls towards smaller ones. Both directions are visible in the measured history, the sample is small and noisy, and inventing a number would repeat a mistake this project has twice had to undo - writing a rate down as a constant until it hardened into an article of faith.
The field results are uncontrolled
Section titled “The field results are uncontrolled”A maintenance deliverable estimated at twelve months - thirty production websites - delivered in under seven days. Features estimated at a team of five for twenty weeks, delivered in under one. These are operator-reported, not benchmarked. Treat them as testimony.
And the finding that cuts against the pitch
Section titled “And the finding that cuts against the pitch”On the current frontier model the traps do not bite in any arm - 30 clean runs out of 30, and the overhead for the full pipeline is around 1.4x rather than 3.1x. If your whole fleet is frontier and your tickets look like this fixture, the discipline buys you the audit trail rather than the defect reduction.
The honest reading of the whole table is narrower and more useful than the headline: judgement-gated process converges on the model’s unaided behaviour exactly when the model most needs the process.
Where to next
Section titled “Where to next”- The white paper - the full study, its protocol and its claims register.
- A spec per run, or a sprint? - what these numbers argue for.
- How it compares - where the tool sits in the category.