The Splitter test: we tried to prove splitting beats one-shot. The baseline wouldn't lose.

Why this test exists

The claim on the table was ours: split a job into sealed parts, let free or weak models build each part blind, recombine — and the weak model does work it couldn't do in one shot. That's a strong claim. Strong claims get frozen protocols here, not anecdotes.

The protocol — frozen before any run

One build: "Pixel Ledger," four interlocking Python modules (money, storage, reports, command line) with exact parsing rules, exact output formats, and integration checks that run the finished tool end-to-end. Same weak model in both arms (deepseek-flash). Same deterministic checker grading every attempt cold.

• Baseline: the whole spec in one competent prompt, one shot,

five attempts. No help, no repair.

• Split: each module built from a sealed brief — its own spec

plus the other modules' interface contracts, never their internals. Parts recombined mechanically. Three full runs.

Pre-registered bar: split passes ≥2 of 3 and baseline passes ≤1 of 5.

The result

Verdict: NOT PROVEN — the claim's bar is not met, because the baseline never failed. Given the complete spec in a single prompt, the current weak model wrote a correct four-module tool five times out of five. The split arm matched it perfectly — every sealed part, built blind, fit its neighbours every time, integration checks included. That consistency is real, and it is all this test establishes. There was no gap for the method to close, so the method's advantage cannot be quoted from this test, and won't be.

Where the advantage would have to be proven: a job big enough that one-shot actually breaks — a dozen interlocking parts, more contract than one prompt of attention can hold. That test is specified in the summary of this slate. It has not been run, and nothing here pretends it has.

Disclosure — our checker had a bug (again)

Two attempts originally failed on a summary-output mismatch where the expected and actual text printed identically. Byte inspection showed both end without a trailing newline — their command-line module writes the summary with a bare write instead of print(). Our spec says the command "prints exactly render_summary(...)" and defines that text as having NO trailing newline: the models had followed the spec's exact wording, and the checker had encoded our own reference's habit instead. We fixed the one comparison, re-graded every attempt from the saved files (zero new model calls), and preserved the original verdicts alongside. The correction moved the baseline from 4/5 to 5/5 — the re-grade made our own claim harder to prove, and it is published for exactly that reason. Full story: RE-GRADE.md.

Evidence — re-grade it yourself

← All proofs How the harness is tested