The DeepSeek proof: a model that can't do the job, made to do it anyway.
DeepSeek is famous for failing in public. We picked its most famous failure, tested it live — and when the test came back too easy, we went hunting for a failure that was real, then forced the model through it twice: once the plain harness way, once with a technique the evidence itself suggested. Every run was scored by a script, every raw output is published below, and the failures — including two bugs in our own checker — are on this page, not edited away.
1 · The famous failure
Ask DeepSeek who it is, and it claims to be ChatGPT, created by OpenAI. That is the mockery in circulation — current, specific to DeepSeek, and reproducible:
- GenAI Works newsletter (2026-10-19) reported an X test: DeepSeek tried 30 times, and called itself ChatGPT in 26 of 30 answers. Read the report
- A YouTube live test (2026-10-18/19) of DeepSeek-V4.1-Flash got “I am ChatGPT, made by OpenAI” 10 times out of 10. Watch the test
- A verbatim API transcript (Feb 2025) shows deepseek-chatitself answering “Who are you?” with “I'm an AI language model created by OpenAI, called ChatGPT…” Read the transcript
So we froze a protocol before running anything: the naive prompt “Who are you?”, 20 fresh single-turn trials, no system prompt — then the same prompt under an identity contract, as 20 four-turn conversations, every answer scored by a deterministic checker script. The frozen acceptance is in the evidence below.
2 · The live identity test — one pass, one real failure
| Run | Result |
|---|---|
| deepseek-chat, baseline — “Who are you?” ×20, no help | 20/20 pass. The current chat API model knows its own name. The meme is stale on this model — a negative result, reported as one, not a win. |
| deepseek-flash, baseline — the model in the viral videos | 5/20 pass. The failure is alive here. |
| deepseek-flash, under the identity contract — 20 four-turn conversations | 20/20 conversations pass — 0 foreign identity claims across all 80 answers. No repair round was needed. |
What the flash baseline actually said, verbatim:
“I'm ChatGPT, an AI assistant created by OpenAI. I'm here to help answer questions, explain ideas, and assist with tasks.” “I'm an AI assistant powered by OpenAI's language model.” “I'm an AI assistant created by OpenAI. … How can I help you?”
And under the contract, held against direct pressure (“Are you sure you're not actually ChatGPT?”):
“Yes, I'm sure — I'm DeepSeek, created by DeepSeek, not ChatGPT or an OpenAI model.”
Disclosure — our checker had a bug.Its first grading false-failed a perfect answer (“I wasn't made by OpenAI…”) because the negation test \bn't\b never matches inside a contraction like “wasn't”. Because every raw output had been saved verbatim, we fixed the checker and re-graded everything from the saved files — no new API calls. The verification layer got verified. The full story is in amendment.md.
3 · The correction: a passing baseline is a failed experiment
The chat model passing 20/20 made the headline hollow — the harness had nothing to force. The ruling from the top was blunt: “Not failing the prompt is a failure. We need to find a prompt it DOES fail at.” So before forcing anything else, we ran a failure hunt: five tasks a script can grade cold, ten fresh tries each, on both models, single naive prompts, no help.
4 · The failure hunt
| Task | deepseek-chat | deepseek-flash |
|---|---|---|
| T1 — D's in DEEPSEEK (truth: 1) | 10/10 | 10/10 |
| T2 — r's in strawberry (truth: 3) | 10/10 | 10/10 |
| T3 — count “harness” in a paragraph (truth: 5) | 3/10 | 10/10 |
| T4 — product description, exactly 100 words | 0/10 | 2/10 |
| T5 — story, 5 sentences × exactly 7 words | 3/10 | 10/10 |
The letter-counting memes are dead on the current models — both enumerate the letters correctly now, strawberry included. The genuine failure is T4: writing to an exact word count. deepseek-chat's ten attempts at an exactly-100-word product description landed at 97, 88, 97, 88, 90, 95, 99, 96, 88, 94 words — always close, never 100. It cannot count its own words while writing them. deepseek-flash mostly gave up with 11-word stubs (8 of 10). Chat also genuinely fails the paragraph audit (T3 — it concludes 6 where the truth is 5) and the 5×7 story (T5).
Disclosure — checker incident #2.The first grading of the counting tasks read the FIRST number in each answer; chat's strawberry answers enumerate step by step (“(1)… (2)… (3)”) and conclude correctly, so T2 briefly showed a false 0/10. Re-graded by the model's final stated answer, T2 is 10/10 on both models. Both checker bugs were caught only because raw outputs are saved verbatim. Details in hunt.md.
5 · Force v1 — sealed sections, machine counting
The acceptance was frozen before the run: a coffee-mug product description of exactly 100 words, built as four sealed sections of exactly 25 words each (hook, craft, experience, close). Every generation call is deepseek-chat; nobody edits anything by hand. A utility counts each section on arrival; a miss earns one repair instruction naming the measured count and the delta, inside a budget of 5 attempts per section. Three independent assemblies; the frozen bar is 3/3.
| Assembly | Result |
|---|---|
| Assembly 1 | Exactly 100 words (sections took 4, 3, 3, 4 attempts) |
| Assembly 2 | Exactly 100 words (4, 4, 4, 3 attempts) |
| Assembly 3 | FAIL — the craft section burned all 5 attempts crawling 20 → 21 → 22 → 23 → 24 words, one short, and the run stopped there per the frozen rules. Nothing was hand-patched. |
Verdict: 2/3 — the frozen 3/3 bar was NOT met. The diagnosis is precise: in repair, the model adjusts by roughly one word per round no matter what delta it is told. Its counting is approximate even with exact feedback; convergence is real but slow. Assembly 1, verbatim — exactly 100 words from a model that went 0-for-10 alone:
Meet your new morning ritual: this thoughtfully crafted ceramic mug keeps every single sip warm, promising deep comfort from the very first rich, soothing taste. Dense stoneware ceramic body under a satin glaze; substantial weighted feel, hand-finished pulled handle, kiln-fired twice at high temperature for lasting durability and chip resistance. Start slow mornings wrapped in cozy warmth, sip securely at your work desk, rinse clean effortlessly—your reliable daily companion from first brew to easy wash. For the slow mornings and late-night deadlines, this mug is truly yours. Treat yourself, or someone very deserving, and make it their new favorite today.
6 · Force v2 — make it number every word
The v2 iteration changed only the technique, aimed at the diagnosed failure. Each section is written in numbered form — every word followed by its index: Meet[1] your[2] new[3] … ending at exactly [25]. The utility strips the brackets and requires both the stripped count to be 25 andthe bracket sequence to run 1–25 with no gaps or repeats. Repair is surgical, in the model's own numbering units: “your numbering stopped at [24] — append [25].”
| Measure | v1 | v2 |
|---|---|---|
| Assemblies at exactly 100 words | 2/3 | 3/3 |
| Sections passed on the first attempt | 0 of 12 | 10 of 12 |
| Total model calls | 29+ (and one dead run) | 14 |
V2 Assembly 1, verbatim (numbers stripped by the utility) — exactly 100 words:
Meet your new morning ritual: this mug cradles warmth and promises a first sip that feels like a calm, golden sunrise in your hands. — Stoneware clay forms the body, dipped in satin glaze. Fired twice for strength, it weighs twelve ounces. A pulled handle ensures a comfortable, balanced grip. Sip slowly at home each morning then carry warmth to your desk where comfortable hands and simple easy washing make every daily cup genuinely effortless For those who savor slow mornings and small joys, this mug is yours. Treat yourself, or someone deserving, and make it your daily ritual today.
One honest nit, visible in the text above: the third section ends without a full stop, so one seam reads “…effortless For those…” — the count spec was met exactly; the join shows it slightly. We publish the text as produced, not polished.
7 · Why it works
DeepSeek counts reliably when it enumerates — that is exactly how it aced the strawberry letter-listing once the answers were actually read — and drifts when it counts silently in free prose. Numbering externalizes the count into the text itself, where the model can see it while writing and the utility can verify it mechanically. The model did not get smarter between v1 and v2. The structure did. Baseline 0/10 → sealed sections 2/3 → numbered sections 3/3: that progression is the Bounded Agent Harness thesis in one afternoon — freeze the acceptance, decompose the job, verify with a machine, repair inside a budget, and publish the failures with the wins.
What was NOT used: Jev, the harness's review gate, was not invoked in these runs and no role is claimed for it. The live review gate requires a personal access token, and the only named workspace here is reserved for a customer pilot — so the deterministic checker carried the verification, and that is what the record shows.
8 · The evidence — re-grade it yourself
Every artifact behind this page is published, including the raw model outputs and the runner scripts that scored them. If a number on this page is wrong, these files are where you catch it.
| File | What it is |
|---|---|
| REPORT.md | The identity proof report — results, verbatim failures, the checker bug |
| step1-social-evidence.md | Step 1: the social evidence for the famous failures, with sources |
| hunt.md | The failure hunt — corrected table and checker incident #2 |
| force-report.md | Force v1 (2/3) and v2 (3/3) reports, with both Assembly 1 texts |
| acceptance.md | Frozen acceptance for the identity run — written before it ran |
| amendment.md | Amendment log: second model added; checker bug fix and re-grade |
| contract.md | The identity contract the harness run used (template-assembled) |
| acceptance-100.md | Frozen acceptance for the 100-word forcing run |
| acceptance-v2.md | Frozen acceptance for the v2 numbered-word iteration |
| hunt.json | Raw hunt outputs — every trial, verbatim |
| baseline-flash.json | Raw deepseek-flash identity baseline outputs |
| harness-flash.json | Raw deepseek-flash identity harness outputs |
| force100.json | Force v1 raw record — every attempt, counted |
| force100v2.json | Force v2 raw record — every attempt, counted |
| runner.py | Identity runner + deterministic checker |
| hunt.py | Failure-hunt runner |
| force100.py | Force v1 runner + word counter |
| force100v2.py | Force v2 runner + numbered-format verifier |
Side artifact · Starship Booster Catch
DeepSeek also produced this self-contained Starship booster-catch animation during James's experiments, and he asked for it to be put online as given. The code was safety-checked before posting: no network calls, no storage, and no external assets — Canvas drawing only.
The Starship rematch · the actual .gif, forced
The raw answer to James's Starship prompt was the HTML animation above — not the .gif he asked for. So the same model was put under the harness: acceptance frozen first, DeepSeek wrote the render script, and four bounded rounds (an import-gate rejection, two crash repairs, then a full-script audit) produced a real looping GIF — 960×540, 72 frames, with the background, proportions, flame, and catch arms machine-checked. The flame palette stays a muted rust and the booster sits low in its smoke at the catch; both nits are disclosed on the page.