#Evals for ADHD — Phase 2

#We tried to break our own result. Here's what survived.

---

#The pitch, in one paragraph

v0.1 of ADHD showed a 5/6 win over single-shot prompting — judged by a model from the same family as the generator, on six hand-picked engineering problems. That's a demo, not evidence. So Phase 2 stress-tests the actual claim: swap the judge, leave engineering entirely, and — the sharpest test we could design — hand the harness only what was known before a real scientific discovery, strip every name and term that could leak the answer, and see if the candidate pool contains it. Every number below links straight to the JSON that produced it, and every JSON is backed by raw, append-only LLM call logs. Nothing here is asserted without a file you can open.

---

#💡 The Core Architectural Mechanism: Structured Divergence vs. Automated Convergence (#1 Winner Selection)

ADHD is NOT a naive "Best-of-30" or Pass@30 random sampling method.

• Naive Pass@N Sampling: Runs temperature sampling 30 times on the same prompt. All 30 samples suffer from mode collapse, clustering in the same high-frequency textbook region (e.g., 30 minor variations of standard retry loops).
• ADHD Structured Divergence: Spawns 5 orthogonally forced cognitive frames (Inversion, $0 Budget, Hardware Engineer, Biology, 3AM On-Call), guaranteeing coverage across 5 distinct topological design sectors.
• Automated Convergence Engine: The 30 candidate ideas represent internal test-time compute (similar to internal reasoning tokens in System 2 search like DeepSeek-R1 or OpenAI o1). The user or agent never reads 30 raw guesses. ADHD's automated Critic Engine clusters redundant angles, prunes operational traps, and converges down into 1 single non-obvious architecture recommendation (backed by a top-3 shortlist & trap list).

---

#Headline result: can it find things nobody told it?

Given only a genericized pre-discovery prompt — no paper titles, no drug names, no jargon — for 9 real published discoveries in systems engineering, medicine, and biology:

Small N, and we say so. This is 9 cases, not 900 — treat it as a promising early result on a genuinely new evaluation method, not a settled law. What earns it a place on this page is that it's the one study in this whole project with an answer that isn't another LLM's opinion — it's checkable against the literature, case by case, above.

The twist worth remembering: on several hits, the correct idea was sitting in the pool at rank 12–25 out of 30 — meaning the harness generated the right answer and its own critic pass nearly buried it. That's the actual engineering finding here, and it's going straight into v0.2.

Full case-by-case reasoning: findings/study3_finding_reproduction.md · Raw data: findings/finding_reproduction.json

---

#The shape of the win, everywhere we looked

Same pattern, every study, every judge, every domain: ADHD dominates on breadth/novelty/trap-detection and loses on immediate actionability. That's not noise — it's the architecture's actual trade-off, and it held under a judge swap and three unrelated domains.

#Study 1 — Cross-model judging (12 engineering problems, 2 judges)

Win rate: 2/12 under Judge A, 4/12 under Judge B — modest, and lower than v0.1's headline number. We're reporting it exactly as measured. What's notable is the dimension deltas barely moved when we swapped judges — that's the part that looks structural rather than judge-flattery.

Caveat on record: Judge B (gemini-3.1-flash-lite) is a smaller model from the same vendor as the generator, not an independent lab — call this cross-variant, not cross-family. A non-Gemini judge is the natural next test.

→ findings/study1_cross_model.md · findings/cross_model.json

#Study 2 — Cross-domain generalization (18 problems, 3 tiers, 6 each)

Reads like win rate rises the further you get from engineering — at N=6 per tier that's 1–2 results moving the number, so we're stating the trend, not claiming it's proven. What holds regardless of win/loss: novelty and trap-detection gains transfer cleanly outside code.

→ findings/study2_cross_domain.md · findings/cross_domain.json

#Study 4 — Frame ablation (51 runs, 15 frames)

Top survival drivers are pragmatic, risk-hunting operational frames (ops-3am, adversary, speedrunner, inversion), while speculative frames (extreme-infinite, extreme-zero) drive extreme novelty at the cost of survival.

→ findings/study4_frame_ablation.md · findings/frame_ablation.json

→ findings/study4_frame_ablation.md · findings/frame_ablation.json

---

#How it works

graph TD
    subgraph SingleShot ["Single-shot generation"]
        P1["Problem"] --> Greedy["Greedy decoding P(y_t | y_<t)"]
        Greedy --> Conventional["Conventional, first-plausible answer"]
    end

    subgraph ADHDLoop ["ADHD: isolated divergent branches → critic"]
        P2["Problem"] --> Frames["N cognitive frames, zero shared context"]
        Frames --> B1["inversion"]
        Frames --> B2["$0 budget"]
        Frames --> B3["biology"]
        Frames --> B4["3am on-call"]

        B1 --> Pool["Candidate pool"]
        B2 --> Pool
        B3 --> Pool
        B4 --> Pool

        Pool --> Critic["Critic: score, cluster, flag traps"]
        Critic --> Shortlist["Top-K deepened picks"]
    end

    style SingleShot fill:#1f1f2e,stroke:#4a4a6a,color:#ffffff
    style ADHDLoop fill:#112233,stroke:#2b6cb0,color:#ffffff

Isolated branches can't anchor on each other — that's the actual mechanical difference from Chain-of-Thought or Tree-of-Thought, both of which share context across the search. Full formalism: findings/adhd_phase2_research_paper.md.

---

#What breaks next (v0.2 roadmap, based on this data)

• Dual-output mode — stop forcing a choice between wide idea space and ship-today code. Generate both from the same divergent pool.
• Critic recalibration — Study 3's rank-25 hits show the critic sometimes buries the right answer under conventional-looking ones. Fix the scoring bias, not the generator.
• A real cross-family judge — Study 1's next iteration should use a judge that isn't from the same vendor as the generator.
• Bigger N — cross-domain and finding-reproduction trends are directionally interesting at N=6–9; the next phase should push toward N=20+ before calling any trend proven.

---

#Reproducing this

research/
  problems/    # exact prompts used — nothing hidden
  cases/       # the 9 discovery cases: pre-discovery prompts + ground truth
  logs/        # raw, append-only LLM call logs — one file per run
  results/     # aggregated, machine-readable scoring output
  reports/     # full human-readable writeup per study

Model versions, temperatures, and run IDs are logged per call. Full methodology and per-study reports: findings/phase2_summary.md.

Want to check a number yourself, or re-run a study from scratch (install steps, API keys, caching behavior, and an honest note on LLM nondeterminism included)? See REPRODUCE.md.

---

#Citation

@article{akhouri2026evalsforadhd,
  title={Evals for ADHD: Cross-Model, Cross-Domain, and Novel-Finding-Reproduction Evaluation of Divergent Cognitive Branching},
  author={Akhouri, Udit},
  journal={Divergent Labs Research Benchmark Series},
  year={2026},
  url={https://github.com/DivergentLab/evals-for-adhd}
}

Maintained by Divergent Labs · Research lead: Udit Akhouri · researchudit@gmail.com