Proof F — the build-off: what's the same build cost, each way?
Claim tested: "The harness way is cheaper than the ordinary way — measured, not asserted."
Project: deploydiff — a brand-new six-module Python tool that diffs two deploy manifests and flags unlisted file drops (built for this proof; now installed and in use). Complete specification, exact output formats, exact exit codes. The acceptance checks were written and frozen before any build, and proven against a hand-written reference first.
Model: DeepSeek deepseek-flash — the same weak model in both arms. Every call metered from the API's own usage record and priced at DeepSeek's published rates (off-peak: $0.15 per million input tokens, $0.60 per million output). Two independent runs per arm.
Verdict: MEASURED — the harness median bill is ~15% lower, and far more predictable. The runaway repair loop we expected to price never happened, and we say so.
What happened
| Ordinary way (one session, whole project) | Harness way (sealed parts) | |
|---|---|---|
| Finished to all checks | 2 / 2 | 2 / 2 |
| Calls per build | 1 and 1 | 7 and 7 |
| Repairs needed | 0 | 1 per build (same slip, inside one part) |
| Regressions (a fix breaking something) | 0 | 0 |
| Bill, run 1 | $0.010260 | $0.006950 |
| Bill, run 2 | $0.006248 | $0.007092 |
| Median bill | $0.008254 | $0.007021 |
What this does and doesn't prove
It proves the contrast is real but modest at this size: the harness median is 14.9% lower, and the two harness bills landed within 2% of each other while the ordinary bills swung 1.64× apart — the ordinary way costs whatever the model's long think happens to cost that day.
It does not prove the ordinary way wastes money on fix-loops for work like this — on a completely specified six-module build, the model finished in one shot both times, so there was no loop to price. Anyone quoting this proof for a bigger claim is quoting it wrong; the numbers, the per-call logs, and the frozen rules are all in the record for anyone to re-check.