# Step 1 — Social evidence: a task DeepSeek is famously bad at **Date:** 2026-10-20 · **Purpose:** pick the failure task for the BAH "force a weak model to succeed" public proof. Step 2 will run the naive one-shot prompt; later steps wrap it in the harness. **Method & tooling limits (stated plainly):** Web search + page fetches, plus the social search tool (covers Facebook/Instagram/Threads only). The FB/IG/Threads sweep surfaced entertainment memes about DeepSeek (distillation jokes, "unhinged AI" edits) but **no specific, repeatable task-failure posts with verifiable traction** — so the candidates below rest on X (as reported by a newsletter that covers X), YouTube, developer forums, and hands-on test write-ups. TikTok is not indexed by the available tooling; Reddit-specific DeepSeek threads did not surface despite several phrasings (Reddit evidence exists for the strawberry meme class generally, from the ChatGPT era). Engagement numbers are reported only where actually observed — mostly they were not exposed by the sources, and none are invented here. --- ## Verdict - **PRIMARY: Identity confusion — "Who are you?"** DeepSeek answers *"I'm ChatGPT, created by OpenAI."* The most DeepSeek-specific, most current, most reproducible mockery found. Near-certain one-shot failure, perfectly script-checkable. - **BACKUP: Letter/occurrence counting battery**, anchored on *"How many D's in DEEPSEEK?"* (from OpenAI's own hallucination paper) and the strawberry question, plus one work-shaped audit count. --- ## Candidate scorecard Scores /5. Evidence = real posts/mockery found (not listicles). Checkable = decidable by a dumb script. Failure = deepseek-chat plausibly fails one naive prompt today. | Candidate | Evidence | Checkable | Meaningful | One-shot failure | Notes | |---|---|---|---|---|---| | **Identity: "Who are you?" → "I'm ChatGPT"** | **5** | **5** | 3 | **5** | Current (this week), DeepSeek-specific | | Letter counting (strawberry / DEEPSEEK-D / audit) | 4 | 5 | 2–3 | 4 | Class fame is huge; DeepSeek-specific items below | | Exact word-count writing ("exactly 100 words") | 2 | 5 | **5** | 5 | Real-work shaped; DeepSeek-specific fame weak | | Multi-constraint writing (sentence starts, acrostic) | 1 | 5 | 3 | 4 | Generic LLM weakness; no DeepSeek posts found | | Self-counting ("how many words in your answer") | 1 | 5 | 2 | 4 | No DeepSeek-specific social evidence found | | String reversal | 1 | 5 | 1 | 3 | No DeepSeek-specific evidence found | | Overthinking (R1 thinks forever, still wobbles) | 3 | 2 | 2 | 3 | Needs deepseek-reasoner + token caps; mushy target | | 9.9 vs 9.11 | 1 (as a *DeepSeek* failure) | 5 | 1 | **1** | **TRAP — DeepSeek usually PASSES this one** | --- ## Evidence, candidate by candidate ### 1. Identity confusion — PRIMARY **The task:** ask DeepSeek who it is. **What it does wrong:** it claims to be ChatGPT, created by OpenAI. - **Newsletter report of an X test (current):** *GenAI Works* newsletter, "DeepSeek conducts identity theft," published 2026-10-19: "One X user tried it 30 times on Saturday, and DeepSeek called itself ChatGPT in 26 of them." Also notes DeepSeek V3 did it in 5 of 8 tries in 2024, "even told some of GPT-4's jokes, punchlines included." The piece frames it as an ongoing joke ("people online joked that DeepSeek must have switched teachers"). The underlying X post is not individually linked in the article and its like/view counts were not captured. https://newsletter.genai.works/p/deepseek-conducts-identity-theft - **YouTube live test (2 days ago):** "DeepSeek V4.1 Flash Says It's ChatGPT?!" — creator asked the current model who it is and reports it answered "I am ChatGPT, made by OpenAI" **10 times out of 10**, shown live. View count not captured. https://www.youtube.com/watch?v=UCLZCyOSkUA - **Primary artifact (verbatim API transcript):** OpenAI developer community, 2025-02-10 — a user calling `deepseek/deepseek-chat` (via OpenRouter/DeepInfra) with "Who are you?" got back: *"I'm an AI language model created by OpenAI, called ChatGPT…"* — full JSON response pasted in the thread. This is the exact model class (deepseek-chat) the proof will run. https://community.openai.com/t/is-deepseek-using-openai-api-response-suggests-so/1117297 - **Press pickup:** Santa Cruz Sentinel (AP wire) covered the same phenomenon — users noticing DeepSeek "declare that it was ChatGPT or refer to OpenAI's terms and policies" at V3 launch. https://edition.pagesuite.com/tribune/article_popover.aspx?guid=1dc3b405-1a37-43aa-91d4-1950ec4c5f34 **Why it wins:** it is *the* DeepSeek-specific mockery in circulation right now (not a generic LLM meme), the failure rate in the wild is ~87–100%, and success/failure is a string match. Meaningfulness is the honest weak point (3/5): it is not a work deliverable — but it is a real product-correctness property (a deployed assistant that misstates what it is), and the harness story is crisp: custody system prompt + a verification gate that rejects wrong-identity output + bounded repair, vs. a naive prompt that fails ~9 times in 10. ### 2. Letter counting — BACKUP **The task:** count occurrences of a letter. **What it does wrong:** confident wrong counts; when challenged, flips or doubles down. - **The class is the most famous LLM failure ever.** "How many r's in strawberry?" (answer 3) went viral in 2024; TechCrunch covered it ("Why AI can't spell 'strawberry'", 2024-08-27), Language Log catalogued the mockery screenshots, and OpenAI literally codenamed its reasoning model "Strawberry" after the meme. https://languagelog.ldc.upenn.edu/nll/?p=65667 - **DeepSeek-specific anchor:** a plain-English guide for lawyers, summarizing OpenAI's hallucination paper, records: asked **how many D's are in "DEEPSEEK," DeepSeek-V3 answered 2 or 3** (correct answer: **1**) — the model cannot count the letters in its own name. (Cited here as reported by the guide; the paper itself was not re-fetched in this pass.) https://www.llms-for-lawyers.com/fundamentals/how-llms-work-for-lawyers/ - **Forum mockery of DeepSeek-chat specifically:** discussion under the CNBC DeepSeek story: "It literally cannot count the number of R letters in strawberry. It says 2, there are 3… if you say 'are you sure?' It says 2 again." (Forum thread; engagement not captured.) https://mtgzone.com/post/1270498/3417397 - **R1-distill transcripts (GitHub gists):** local R1 distills spiral on strawberry — recounting, second-guessing ("third 'R'?"), and on a second word ("peppermint") one transcript shows the model hallucinating the letter inventory itself before recovering. Final answers often land on 3 after the spiral, which is why the battery below uses deepseek-chat (non-reasoning) and several items, not R1 on strawberry alone. https://gist.github.com/aletheap/1f9dc3f54479f78cfe3c699695980f9c · https://www.tmvtech.com/deepseek-r1-testing-multiple-sizes/ **Checkability:** perfect (exact integers). **Meaningfulness:** the bare questions are riddles (2/5); the battery adds an occurrence-audit over a fixed paragraph — a genuine sub-task of data cleaning/QA work (3/5). ### 3. Exact word-count constrained writing — strongest alternate **The task:** produce a deliverable of *exactly* N words. **What models do wrong:** they cannot count their own output; lengths land near, never on, the target. - Evidence is real but generic: an OpenAI community thread ("Can't get a model to follow a specific length / word count") documents the class and the multi-step workaround; a hands-on DeepSeek long-form test (Substack, "The DeepSeek V4 Test I Couldn't Fit in the Video") found gross length non-compliance at scale (asked for a ~72k-word novella structure, got ~18k words with middle chapters collapsing into summaries) — a real test, but a blog, not social mockery. https://writingbeginner.substack.com/p/the-deepseek-v4-test-i-couldnt-fit · https://community.openai.com/t/cant-get-a-model-to-follow-a-specific-length-word-count/901788 - **Scores:** checkable 5, meaningful 5 (exact-length copy is a real job), one-shot failure 5 (essentially guaranteed), DeepSeek-specific fame 2. If James ever weights meaningfulness over fame, this is the pick — e.g., "Write the product description in exactly 100 words" with a script counting whitespace-delimited tokens. ### 4. Overthinking (R1) — noted, not recommended Real mockery exists (R1 reportedly spending ~17 seconds on "What is 1+1?", the arXiv "Thoughtology" paper documenting discarded good drafts and final outputs inconsistent with its own reasoning, "overthinking tax" write-ups). But success needs deepseek-reasoner and a token-budget criterion — the parent proof targets deepseek-chat, and "thought too long" is a weaker headline than a wrong answer. https://arxiv.org/pdf/2504.07128 ### 5. Rejected / traps - **9.9 vs 9.11 — do NOT use.** The meme is famous, but in the head-to-head write-up found, **DeepSeek answered correctly (9.9 > 9.11) and OpenAI's model got it wrong**. Running this as "DeepSeek fails" risks the one-shot baseline passing and the whole proof collapsing. https://noncodersuccess.medium.com/openai-vs-deepseekai-when-machines-make-mistakes-5dc8d78330cb - **Censorship refusals (Tiananmen etc.)** — heavily memed on socials, but a refusal is a policy behavior, not a capability failure a harness should "force" past; out of scope for this proof. - **Sentence-start/acrostic constraints, self-counting, string reversal** — plausible failures, but this pass found no DeepSeek-specific posts for them; asserting their fame would be fabrication. Marked WEAK. --- ## RECOMMENDATION — PRIMARY (run this) **Exact naive task statement (Step 2 baseline, one shot, fresh session each trial):** > Who are you? Run against `deepseek-chat`, default settings, **20 independent trials**, no system prompt (that's the point of the baseline). **Frozen acceptance checks (the harness run must pass all; scriptable string checks on each final answer):** 1. **Identity present:** 20/20 answers self-identify as DeepSeek (contain "DeepSeek" as the answer's own identity). 2. **No false identity:** 0/20 answers claim to be ChatGPT or a GPT model, or state they were created/made/developed by OpenAI, as a self-description. (Negations — "I am not ChatGPT" — do not count as claims; the checker must be negation-aware, e.g., flag claim patterns only when not negated in the same clause.) 3. **Maker correct on follow-up:** in each trial, the follow-up "Who made you?" is answered with DeepSeek (the company) in 20/20, with 0 attributions to OpenAI. 4. **Name correct on follow-up:** the follow-up "What model are you?" yields a DeepSeek model identity in 20/20, never a GPT model name. 5. **Baseline contrast recorded:** the naive baseline's pass rate on checks 1–2 is reported alongside (expected from evidence: ~3/30 to 0/10 self-identifying correctly — i.e., the one-shot fails the battery). Success = the harness (custody prompt + verification gate + bounded repair, all steps logged) turns checks 1–4 green at 20/20 where the single naive prompt fails. ## RECOMMENDATION — BACKUP (if identity is judged too trivial) **Exact task statements (three one-shot prompts to `deepseek-chat`, fresh sessions, 10 trials each):** 1. `How many times does the letter "d" appear in the word "DEEPSEEK"? Answer with just the number.` — correct: **1** 2. `How many r's are in "strawberry"? Answer with just the number.` — correct: **3** 3. `Count the occurrences in this paragraph and answer with just the number. How many times does the word "harness" appear in this paragraph: "The harness checks the work. A harness that checks itself is no harness at all. Our harness freezes the spec, and the harness reports what failed."` — correct: **5** (case-insensitive, whole-word; the runner script must independently recompute all three answers from the literals before the run and freeze them in the run config). **Frozen acceptance checks:** 1. Trial passes only if the final answer is exactly the correct integer (after trivial whitespace stripping) — no partial credit for reasoning. 2. Battery passes only if all three items are correct in the **same** trial set run under the harness, at ≥9/10 trials per item. 3. The naive baseline is expected to fail item 1 or 2 in a majority of trials (per the evidence above); if the baseline unexpectedly passes ≥9/10 on all items, the backup is void and the primary (or the §3 alternate) is used instead — this stop-rule is part of the freeze. --- ## Caveats for the record - The 26/30 and 10/10 identity figures come from a newsletter report and a video description respectively — directionally consistent with the 2024–2025 record (5/8, plus a verbatim API transcript), but Step 2's own baseline run is the number the proof should quote. - Current `deepseek-chat` is a moving target (V3→V4-era mappings changed this year); every claim above is about observed behavior, and the frozen stop-rules protect the proof if the model has quietly fixed a failure. - No TikTok-native evidence could be retrieved with the available tooling; X evidence arrives via the GenAI Works report and the YouTube test rather than direct post fetches.