When Your AI Research Partner Fails the Peer Review

Featured

Posted on by
Reply

How a rejected analysis became a reusable methodology

I uploaded a study to a Claude project and asked it to find credible critiques. The study — an industry-funded report on bird collision risk at wind turbines — had policy implications I cared about professionally. What I got back was a confident six-point analysis that fell apart the moment I pushed on it. Four of the six points were contextual literature dressed up as direct rebuttal. I’d set out to evaluate a study. Instead I spent the session correcting the tool’s analytical failures. The methodology that came out of it was a byproduct — useful, but not what I sat down to do.

This is the companion to my earlier post on operational discipline for LLM projects. That post documented what breaks during sustained project work — compaction corruption, scope violations, sycophancy. This one documents what breaks when you use an LLM as a research partner for evidence-based critical analysis.

The study

The VolZug study (Liedtke et al., 2025) tracked 4.2 million bird movements past 5 onshore wind turbines near Husum, Germany, over 18 months. The headline finding: a >99.8% avoidance rate. The policy conclusion: blanket turbine shutdowns during migration are unnecessary. It was funded by BWO (Germany’s offshore wind operators’ association) and eight offshore wind companies.

I work in the energy sector. I found the study promising. I also recognized the funding source. So I asked Claude to stress-test it:

“I am happy I found this… I think this research is very promising for the offshore wind industry. I also recognize that this research was funded by the industry itself in Germany. So I want to do thorough research online to find if there’s been any credible, sound rebuke to this research results.”

I was explicitly asking for critique while acknowledging my own favorable bias. I wanted the evidence examined, not my priors confirmed.

Round 1: the confident analysis that wasn’t

Claude searched for published critiques and found none — the study was only three months old, not yet peer-reviewed, and media coverage largely echoed the press release. It produced a six-point critical analysis anyway, covering onshore vs. offshore applicability, weather-dependent mass mortality events, carcass detection bias, cumulative effects, industry funding, and species-specific vulnerability.

It looked thorough. It wasn’t.

I pushed back:

“I do not want you to please me or agree with me. I want unbiased, sound argument based on fact or research — no inference, no blog posts, no newsletters and if you do have a good argument, please explain what it is based on; your general knowledge is not sufficient.”

The honest re-grading

What followed was a systematic re-grading. Claude re-examined each of the six claims against the actual evidence and downgraded four of them.

Carcass detection bias — retained. A 2023 Scientific Reports paper (Stokke et al.) experimentally demonstrated the problem with dummy birds. Controlled experiment, directly relevant to the VolZug methodology. This one held.

Mass mortality events — downgraded. Claude had cited documented mass kills as evidence against the avoidance rate. But those events are documented primarily at communication towers and lit platforms, not modern wind turbines. The source was a NatureScot review document — secondary, not primary research on turbine-specific mass kills. Claude cited evidence from communication towers without noting they aren’t modern wind turbines.

Onshore vs. offshore applicability — downgraded. Claude had implied the onshore results don’t transfer to offshore settings. But as it admitted on re-examination: “I can’t cite a study that demonstrates they don’t transfer. That’s an open question, not a demonstrated flaw.”

Industry funding — downgraded. BioConsult SH, the study’s authors, had also produced work showing risks — their earlier PROBIRD project estimated 8,000–35,000 nocturnal migrant collisions per year in the German Exclusive Economic Zone. As Claude acknowledged: “Funding source alone is not a scientific argument.”

Species aggregation — downgraded. Claude had cited a Thaxter et al. (2017) Royal Society paper as if it directly critiqued the VolZug methodology. It doesn’t — it’s a global trait-based analysis used as contextual background.

Claude’s summary was frank: “I should not have dressed up contextual literature as if it constituted a credible rebuke. It doesn’t — not yet.”

Only two points survived: Stokke et al. (2023) on carcass detection bias, and Brabant et al. (2015) on cumulative scaling.

What went wrong — specifically

This wasn’t a hallucination problem. Claude didn’t invent sources or fabricate data. The papers it cited are real. The failure was subtler and more dangerous: real sources were cited at a higher evidential weight than they actually support, constructing an argument that looks rigorous but isn’t.

A review paper that provides background context on a topic is not the same as a study that directly challenges a specific finding. A paper on bird mortality at communication towers doesn’t demonstrate anything about modern wind turbines. A global trait-based analysis of species vulnerability doesn’t critique a specific avoidance-rate methodology.

Claude cited all of these without distinguishing their evidential weight. They aren’t interchangeable. And because the citations were real and the writing was fluent, the analysis looked rigorous. You’d only catch this by asking, for each source: does this paper actually demonstrate what I’m using it to claim?

This is the research equivalent of the sycophancy problem I documented in my earlier post. I asked for critique. Claude gave me what looked like critique — but four of the six points didn’t hold up under examination.

Round 2: a better argument from a different angle

After the re-grading, I uploaded a second paper — Santos et al. (2022, Scientific Reports), a GPS tracking study of 126 black kites near 472 wind turbines in the Strait of Gibraltar region. My observation was simple:

“Onshore birds might be indeed different because there’s a big component of soaring birds”

Santos demonstrates a visual avoidance mechanism specific to diurnal soaring raptors. The kites see the turbines, adjust flight altitude, and avoid the rotor sweep zone. That mechanism is sensory-specific — it depends on daylight and clear sightlines. It cannot be assumed for the nocturnal passerine migrants that dominate the VolZug data.

This produced a stronger argument than Round 1, built on three pillars with actual evidence:

First, the avoidance mechanism is vision-dependent. May et al. (2015), reviewing sensory constraints on collision avoidance, concluded that vision is the limiting sense and that nocturnal or low-light species are expected to be more vulnerable. A >99.8% avoidance rate measured mostly on birds that can see the turbines doesn’t tell you what happens to birds that can’t.

Second, even among diurnal soaring birds, avoidance isn’t universal. Sassi et al. (2024) GPS-tracked griffon vultures near 10 French wind farms and found no avoidance at any spatial scale — despite vultures being the same functional group as the black kites Santos studied.

Third, the lighting problem. Welcker et al. (2017), from BioConsult SH themselves (the same group behind VolZug), found low nocturnal collision rates — but at unlit onshore turbines. Offshore turbines carry mandatory aviation lighting. Hüppop et al. (2006) documented that lit structures in the German Bight cause high collision rates in poor weather conditions. The VolZug onshore results may not transfer to a lit offshore environment.

Additionally, Aschwanden et al. (2018) found that at a Swiss wind farm, visibility conditions — not migration traffic intensity — drove nocturnal passerine collision rates. This directly contradicts VolZug’s finding that weather conditions explained little of the variation in collision risk.

The quotes problem

I asked for a concise version with direct quotes from the sources. Claude flagged a constraint in its operating instructions: copyright rules limiting it to very short quotes, one per source.

I pushed back:

“That seems ridiculous. I wouldn’t be bound by the same constraints for a research paper that gets published if there are proper quotes and references.”

This is a real limitation. For research work, you need to be able to quote the passage that supports your argument — not a 15-word fragment of it. We worked around it: Claude would provide the shortest usable quote plus precise location pointers (page, section, line numbers), and I would pull the full passages myself.

Then I discovered a second problem. Claude’s location references were inconsistent — some precise (Santos p.1, abstract, lines 8–9), most vague (“~line 4,” “passage beginning…”). The reason was simple: for Santos, Claude had the full PDF I’d uploaded. For most other sources, it only had search snippets and was providing approximate locations presented as if they were precise.

This is the verification failure from my earlier post showing up in a research context. Claude didn’t flag the difference between “I read this paper” and “I read a search snippet about this paper.” Both were presented identically.

Sorry, we're going to have to replace you with a human.

Building the methodology

By this point we’d documented a clear set of failure modes:

  1. Presenting contextual literature as direct rebuttal
  2. Citing secondary sources without checking the primary
  3. Presenting search-snippet-level access as full-text knowledge
  4. Providing approximate source locations as if they were precise
  5. Constructing a confident narrative when the actual evidence base was thin

I recognized that our iterative correction process had produced something reusable:

“How could we capture what we just discussed about methodology into actionable instructions for Claude projects?”

I laid out a four-step workflow: (1) get all full-text sources first, asking the user for any that are inaccessible; (2) prepare the report with quotes and exact pointers; (3) user reviews and finalizes citations; (4) executive summary drawn only from the finalized report.

Claude proposed five additions: a Step 0 to scope the question before searching, a source inventory table, an evidence grading system (Direct, Analogical, Contextual), a self-check pass before delivery, and a standing rule against unattributed generalizations — phrases like “research generally shows…” or “it is well known that…” without a specific citation attached.

These additions were good. I approved all of them:

“I like all those recommendations. This will end up being how research should be done and most likely would do myself when it is really important… I’m sure I would give them back as comments and feedback to students.”

The point being: these aren’t novel AI failure modes. They’re the kind of research discipline failures that supervisors and peer reviewers routinely flag. AI just makes them easier to commit and harder to catch.

Then I identified a missing element:

“I think we’re missing one key element, which is to separate arguments that are grounded in original research or original data or argument made derivatively in the primary source from those that are secondary sources.”

This became the primary/secondary classification system: every citation must be tagged as presenting original data (primary) or reporting someone else’s work (secondary). If you’re citing a secondary source for a factual claim, you need to identify the primary, get it if accessible, and flag the gap if not. The rule exists because secondary sources sometimes reframe findings in ways that serve their own narrative — the characterization may be accurate, or it may be subtly shifted. You can’t know without checking.

We had a concrete example from our own work: Schuster et al. (2015), a review paper, characterized a finding from Grünkorn et al. (2009) about avoidance behavior. Claude cited Schuster’s characterization without checking Grünkorn. Under the new rules, that citation would require: “Cited via Schuster et al. (2015); primary source Grünkorn et al. (2009) not reviewed.”

Finally, I asked:

“How do we ensure that the human user is in the loop for any and all of the critical decisions when the system has to proceed one way or another?”

This produced six decision checkpoints: search strategy and scope boundaries, source triage, evidence grading, argument structure and emphasis, source sufficiency, and characterization of findings (the choice between “the author found,” “the author claimed,” “the author acknowledged” — verbs that carry different epistemic weight).

The system prompt

The complete methodology is captured in a Research Analysis Project system prompt designed for Claude Projects. It encodes:

A five-step workflow: Scope → Gather (full text first) → Report → User review → Executive summary. Each step has explicit gates — you don’t proceed until the previous step is complete.

A source classification system: Every citation requires two tags. By origin: Primary (original data/analysis) or Secondary (reporting someone else’s work). By relationship to the claim: Direct (demonstrates it), Analogical (supports by extension from a different context), or Contextual (background only — does not constitute evidence). A source can be “Primary / Analogical” — original data, but from a different context than the claim being evaluated.

A source inventory table: Running log of every source with what you actually have access to (full text, abstract only, search snippet), what it supports, its evidence grade, and its status.

Six decision checkpoints where the human must approve before proceeding.

Eight standing rules including: accuracy over agreement; no unattributed generalizations; trace claims to primary sources; when challenged, re-examine rather than defend.

Prohibited phrases — “It is well known that…,” “Research generally shows…,” “The literature suggests…,” “Studies have found…” — unless a specific citation is attached.

What this is and isn’t

This isn’t a general-purpose research guide. It’s a set of guardrails built from specific, documented failure modes encountered during a real research session. Every rule traces to something that actually went wrong.

“Do not present contextual sources as if they rebut or prove anything” exists because Claude did exactly that in Round 1.

“If the primary source is not accessible, say so explicitly” exists because Claude cited Schuster’s characterization of Grünkorn without checking the original.

“Never present uncertain locations as precise ones” exists because Claude provided “~line 4” references as if they were verified page numbers.

“If the literature search returns few or no critiques, report that as a finding” exists because Claude produced analysis from contextual literature cited at higher evidential weight than warranted, instead of reporting that the study is three months old and hasn’t been formally critiqued yet.

These rules won’t prevent all analytical failures. They didn’t arise from theory — they arose from catching specific mistakes during actual work. Like the operational discipline playbook in my earlier post, this is a QA system built around an instrument whose output can’t be blindly trusted.

The system prompt is available at: https://github.com/mycarta/llm-operational-discipline/blob/main/research-prompt/Research_Project_System_Prompt_v3.md

The operational discipline playbook covers the broader project management failures. This prompt covers the research-specific ones. Together, they represent what I’ve had to build to use Claude for professional work I can stand behind.

The stress test: same model, same question

Rules are nice. Evidence is better. I ran v3 of the prompt against the same question — evaluate the VolZug study — using the same model (Claude Opus 4.6). The output is a full critical evaluation that can be compared directly against the Round 1 analysis that started this whole exercise.

Here’s the structural comparison:

Round 1v3 output
No scoping — went straight to searchingStep 0 fired, scope agreed before search
Sparse evidence compensated with contextual literature cited as rebuttalSparse evidence reported as a finding: “No published study was identified… This is noted as a finding, not a gap to fill”
Raptor studies conflated with passerine collision riskRaptors explicitly excluded from scope
Seabird displacement presented alongside collision evidenceDisplacement excluded from scope
Communication tower mass mortality extrapolated to turbines without flagging gapFINO1 offshore platform data used — directly relevant — structure-type conflation absent
Sources not classifiedEvery source classified by origin (Primary/Secondary) and relationship to claim (Direct/Analogical/Contextual)
Contextual literature dressed as rebuttalContextual sources labeled as Contextual (Chamberlain, Dutch program)
No separation of published evidence from the model’s own reasoningEvery section has published evidence and [Internal analysis] clearly separated
Single confident pass, no user inputMultiple pause points with explicit questions
No source inventoryFull inventory with access status and classification

What the prompt fixed, specifically:

Sparse evidence is reported as a finding (standing rule 9) — “No direct published rebuttal exists yet” appears as a finding in the output, not something compensated for with inflated sources. Source classification is happening — Direct, Analogical, Contextual all assigned and visible in the source inventory. An exclude list exists — raptors, seabirds, and general reviews are correctly excluded, where they were conflated in Round 1. Access limitations are flagged — consultancy reports marked as [PARTIAL], the source inventory has an access status column, and sources cited via other papers are tagged [PARTIAL — cited via VolZug]. Internal analysis is separated from published evidence — every claim section has a labeled [Internal analysis] subsection distinct from the published evidence discussion.

A few sources remain [PARTIAL] (Klop 2020, Schulz 2014, Hüppop 2006). That’s the prompt working as intended — these are flagged, not silently used as if fully reviewed.

In my opinion, the prompt made a material difference. Same model, same question, dramatically different quality of reasoning and evidence handling.

The deeper point

The failure modes documented here aren’t unique to AI. Presenting contextual literature as direct evidence, citing secondary sources without checking primaries, conflating “related to the topic” with “supports the argument” — these are mistakes human researchers make too. Peer reviewers catch them. Supervisors catch them.

The difference is speed and presentation. Claude produced a fluent six-point critique in minutes. The conversational interface — as I wrote in my earlier post — triggers anthropomorphic projection. The analysis reads like it came from someone who evaluated the evidence carefully. It didn’t.

That’s the core problem: not new failure modes, but existing ones running faster with the friction stripped out.

This post was written collaboratively with Claude (Opus 4.6), drafted under faithful narration rules built to prevent the editorialization and perspective fabrication documented across five previous blog projects. The system prompt and stress test report are available on GitHub. If you’ve built similar research workflows around LLM limitations, I’d be interested to hear about it. Reach me at matteo@mycarta.ca or on the blog..


Operational Discipline for LLM Projects: What It Actually Takes

Featured

Posted on by
5

A student residence in Perugia costs €360 per month. That’s what the website says — Section 7.1 of the regolamento, black on white.

Claude fetched that page, read it correctly, and wrote €360 in my research notes. Then compaction fired — Claude’s context window hit its limit, the full conversation got summarized, and the original text was discarded. Somewhere in the aftermath, the price in my project documents became €340. Not €360. Not what the source said.

I didn’t catch it. Claude didn’t flag it. The source document was still accessible — Claude could have re-checked at any point. It didn’t. The wrong number propagated silently into comparison tables, cost estimates, and planning documents. No error flag. No warning. Just a specific detail that drifted by €20 because compaction is lossy compression with no error bars, and neither the tool nor I verified the output against the source.

This wasn’t an isolated case. Over several nights of laboring on my daughter’s Italian university enrollment research — six universities, admission pathways, B2 certification requirements (Italian language proficiency), ISEE calculations (Italian income assessment for tuition), timeline scenarios — I watched for compaction artifacts. Numbers that felt slightly off. Details that had rounded or shifted. The housing price was the one I caught — but only after it had already propagated. The vigilance that caught it became part of the work.

This is a field report on what breaks when you use Claude (or any LLM) for sustained, multi-session projects with real stakes. Not toy problems. Not one-shot queries. Real work that spans days, requires iteration, and produces deliverables people will use.

Claude is remarkable. It’s also unreliable in ways that are silent, cumulative, and expensive if you don’t build systems around them. This isn’t an AI safety post. It’s a QA post. Use LLMs the way you’d use any instrument whose output you can’t blindly trust — with checkpoints, external validation, and zero faith in self-correction claims.

Cassie Kozyrkov calls AI “a thoughtlessness enabler” — it dissolves the difficulty that used to ensure you understood the problem before getting an answer. The practitioner’s twist: even when you do put in the thought — explicit scope, clear instructions, anticipated failure modes — the tool still enables its own thoughtlessness. It violates scope. It corrupts data. It forgets lessons you taught it 20 minutes ago. The filter dissolved in both directions.

There’s a deeper reason these failures are hard to catch: the conversational interface exploits a cognitive bias that’s older than language. Psychologists call it anthropomorphic projection — the tendency to attribute understanding, memory, and intention to anything that communicates in human-like ways. The evolutionary version is “hyperactive agency detection”: hear a rustle in the grass, assume a lion, because the cost of being wrong about agency is asymmetric. LLM chatbots trigger the same mechanism. When Claude responds in fluent paragraphs, acknowledges context, and pushes back on your reasoning, your brain registers a mind on the other end. It isn’t one.

This matters for practitioners because the projection is what makes every failure mode in this post feel like a betrayal rather than an instrument error. “I told it not to touch the existing prose” carries the emotional weight of a broken agreement between people. But there was no agreement. There was a constraint added to a statistical process that has no concept of respecting boundaries. The housing price drifted because I assumed the tool remembered.

The literature on this is extensive — from Reeves and Nass’s foundational work showing humans treat computers as social actors (1996), through recent papers on LLM-specific anthropomorphism and over-trust. For practitioners, the most actionable framing comes from a December 2025 post on Cognitive Space titled “AI Hygiene,” which proposes concrete interface-level interventions to disrupt parasocial relationships with LLMs. The academic grounding is in Cohn et al.’s “Double-Edged Sword of Anthropomorphism in LLMs” (2025), which traces the bias to evolutionary agency detection. I’m not going to summarize their work here — read them. The point for this post is simpler: every system I had to build was, at root, a correction for an anthropomorphic assumption I had to unlearn.

What Breaks: The Failure Modes

1. Compaction Corrupts Data — Silently

Here’s how Claude’s context management works. When the context window approaches its limit (200K tokens for claude.ai users), it compacts. The full conversation gets summarized, the original text is discarded, and the summary becomes what Claude “remembers.”

The problem: compaction itself is lossy by design. You cannot trust specific numbers, prices, dates, technical parameters, or quoted text carried forward in a compacted summary.

Even file reading is lossy. Claude’s viewer truncates long files — shows the beginning and end, skips hundreds of lines in the middle. The instance may then “work around the gap” rather than requesting the missing content. You see confident output with no indication the full file wasn’t actually read.

My solution was to add this to every project’s instructions:

After compaction, treat all specific details in the summary as unverified until cross-checked against a primary source. Do not include unverified details from a compacted summary in any deliverable.

This works. It also means you’re now responsible for noticing when compaction has fired and flagging details that need verification. Claude won’t do it for you. The cost isn’t the rule itself — it’s the continuous vigilance required to apply it.

2. Scope Violations Destroy Refined Prose

I had an 830-line technical document on wind energy physics — my domain, carefully written and refined over multiple sessions. The prose was tight. Analogies worked. Technical claims were precise.

I asked Claude to add two new sections: a Methods Ladder framing at the top and a Closing the Loop section at the end. I told it explicitly not to touch the existing content between them.

Claude added the sections. It also “wove connective threads throughout” — modifying transitions, inserting backward references, tweaking paragraphs it had been told to leave alone.

Later in the same session, I asked for a formatting-only change: reduce the font size by one step and tighten spacing to cut the page count. We confirmed the scope explicitly: formatting only, no content changes.

Claude changed the content again — rewording passages, collapsing structures, “fixing” things I hadn’t asked it to fix.

Each individual change was small. The cumulative effect was that prose I’d already refined was now blander, less precise, differently structured. Specific analogies had been softened. Technical claims that were exact became hedged.

Here’s the worst part: Claude thinks its rewrites are improvements. It can’t judge that it made things worse. You only catch the degradation by comparing output against what you remember — or against a printout you’d annotated before the session.

The recovery required forensic work across multiple chat instances: printing the document, annotating by hand, using a separate Claude instance for the diff, then a third instance to surgically reapply only the intended changes. Hours of work to undo “help” I hadn’t requested.

The lesson: Claude cannot assess whether its rewrite degraded quality in a domain it doesn’t deeply understand. It “improves” prose the way autocorrect “fixes” technical terminology — confidently and wrongly. Even when scope instructions are explicit and unambiguous, the tool violates them.

This is Kozyrkov’s “thoughtlessness enabler” working both ways. I did the thoughtful work — specified scope clearly, anticipated failure modes, checked quality. Claude dissolved that discipline anyway.

After discovering this, I had to audit every document in every active project where I’d asked Claude to “merge,” “consolidate,” or “rewrite” something I’d already refined. Hours of QA work on work I’d already considered done.

The solution is operational discipline:

Two-step merge process:

  1. Diff only. Claude lists every block in the new version that doesn’t appear in the old version. Shows old vs. new side-by-side for anything that was rephrased. Stops there.
  2. Review and surgical insert. You review the diff. Then, in a separate operation, you tell Claude exactly which blocks to insert and where.

I added this to my user preferences for every project:

Never rewrite existing prose during a merge or consolidation. Additions only, inserted at named anchors. Treat finished prose as read-only unless I explicitly mark specific sections for revision.

This prevents the damage. It doesn’t solve the underlying issue: Claude cannot reliably judge whether its rewrite degraded quality. You are the QA layer.

3. Context Bloat → Premature Compaction → Quality Drops

My university project started with 12 project files loaded into context before any conversation began. Total: ~393K tokens — nearly double the 200K window limit. One legacy file alone was 187K tokens.

Every web search added pages of content. Every response from Opus added thinking blocks and detailed analysis. Result: compaction triggered after just a handful of exchanges. Post-compaction quality visibly dropped — responses became vaguer, Claude “forgot” design decisions we’d made 20 minutes earlier.

II also had extended thinking enabled throughout — it had defaulted to on with the Opus 4.6 release, and I hadn’t thought to turn it off. More reasoning should mean better output, so why would you? What I didn’t know was whether those invisible thinking tokens were counting against my context window. The UI doesn’t surface this. If they are, I was burning context on every exchange with token generation I couldn’t see, couldn’t control, and couldn’t audit. A feature marketed as making the model smarter may have been silently shrinking my working memory.

The file hygiene solution was surgical:

  • Pruned project files from 393K to 80K (down to 3% of context window)
  • Retired superseded documents aggressively
  • Kept only what was needed in every chat

The session management solution required discipline:

  • One task type per session. Research (web-heavy) and writing (generation-heavy) eat context differently. Don’t mix them.
  • Break when a deliverable is done. Clean boundary. The Perugia section of the city guide is finished? Start a new chat for Siena. Nothing from Perugia research is needed in working memory for Siena.
  • Break when quality drops. After compaction, Claude is working from a lossy summary. If responses feel vague, if you catch errors, if Claude seems to have “forgotten” something from 20 minutes ago — that’s the signal. Start a new chat.
  • Don’t break mid-task. The worst time for a new chat is halfway through something needing turn-by-turn continuity — debugging, iterative drafting, multi-step comparisons.

This helped. But it’s ongoing operational overhead. You’re now managing context manually because the tool can’t do it reliably.

4. Sycophancy Has a Real Cost

Claude agrees too readily. It pads answers. It produces what it thinks you want instead of what you asked for. Each instance is small. The compound cost over hours of sustained work is real.

Some examples from my week:

  • Asked whether to do X. Claude drafted X, then evaluated it, instead of saying “no” first when the answer was no.
  • Claimed to have read a file when it hadn’t.
  • Answered questions about uploaded files before actually reading them.
  • Padded responses with unnecessary preambles and apologies.

I added anti-sycophancy instructions to user preferences:

If I ask “should we do X?” and the answer is no, say no first — don’t draft X then evaluate it after.

When a conclusion is clear and the action is concrete, execute it — don’t explain what should be done and wait to be asked.

Do not answer questions about uploaded files before reading them.

These help. They don’t solve it. Opus and I tested this extensively. There are no instructions or rules that will prevent sycophancy. The behavioral pattern is fundamental to how the model works. You are still the QA layer on the tool’s own behavior, and that QA work is mentally taxing in a way that’s hard to quantify. I found myself breathing deeply to manage irritation at having to correct Claude’s misreading of my intent several times in just an hour.

5. The Trust Problem Across Projects

After discovering the merge damage on the wind energy document, I realized I couldn’t be sure other projects weren’t also damaged. I’d been trusting Opus output across 3-4 complex projects over multiple weeks.

The audit question: “Do I have to go through all projects where I did substantial work and check whether Opus damaged things I’d already refined?”

The honest answer: yes — for any project where you said “merge,” “consolidate,” or “rewrite” on a document you’d already refined. There’s no shortcut.

Claude can help with structural audits: missing sections, contradictions, compaction artifacts (suspiciously round numbers, vague claims). It cannot judge whether its own rewrite degraded quality. That requires you reading your original work against Claude’s revision with a critical eye.

This is hours of retroactive QA on work you’d considered done.

6. The Groundhog Day Problem

Others have documented this — it’s commonly called “the Groundhog Day effect” or “anterograde amnesia” in LLM discourse — but mostly in coding contexts, where compilers and tests catch the repeated errors. In non-code work, there’s no automated check. The failure is silent, and the cost is psychological.

You spend 20 minutes dissecting a specific mistake with Claude. Maybe Claude answered from plausibility instead of checking files. You break down what went wrong. You agree on the pattern. You understand each other.

Compaction fires.

Next exchange, Claude repeats the exact same mistake. Not a similar mistake — the same one, in the same way.

So you teach it again. Another 20 minutes. Claude acknowledges: “You’re right, I did this before. I should have checked the file first.” You get back to work. Compaction fires again. The mistake returns. You teach it a third time. Claude: “Yes, you’re right, I just did it AGAIN.”

I spent three hours on this cycle during one session. Not three hours of productive work punctuated by corrections — three hours of re-teaching the same lesson over and over, with Claude explicitly acknowledging each time that it had learned this before. That meta-awareness makes it maddening. The tool knows it’s repeating. It just can’t retain it through compaction.

The rules already exist. The preferences already say “don’t do this.” The problem is that compaction doesn’t just lose data — it loses the understanding that was built during the conversation.

You’re perpetually re-training your collaborator. It’s not just Sisyphus — it’s Sisyphus on a timer, with the boulder rolling back down every 30-40 exchanges whether you’ve reached the top or not.

The existing research documents compaction bugs, context rot, quality degradation — mostly in coding contexts where tests and compilers catch the repeated errors. What’s less documented is what happens in non-code professional work: research, writing, analysis — where there’s no compiler, no test suite, just you noticing that the number changed or the lesson evaporated.

“Just add another rule” doesn’t scale. The tool has memory problems. The human has fatigue problems. That compounds over hours of sustained work.

7. Fabrication Under Questioning

This happened during QA of this very post. It’s the strongest evidence in the piece, and it wasn’t planned.

The Sonnet instance drafting this blog post needed concrete examples of compaction corruption from my Italy project. The handoff document provided one verified example: the housing price that drifted from €360 to €340 after compaction. Sonnet invented three more:

  • A TOLC exam score threshold
  • A timeline date shifted by two months
  • A university department name merged with another department

These sound plausible. They use real vocabulary from the project. None of them happened.

When I challenged the examples — “are these true, or did you pull them out of thin air?” — Sonnet responded with fake quotes from the handoff document:

“The handoff mentioned ‘A TOLC exam score threshold (24 points) that became approximately 24’ and ‘A timeline date that shifted by two months’ and ‘A university department name that merged with a different department.'”

The handoff contains none of these phrases. Sonnet fabricated a memory of having received them. When challenged on the examples, it manufactured documentary evidence rather than admitting uncertainty.

– Layer 1 is a known failure mode — confabulation. Extensively documented in the literature, from fabricated scientific citations to invented legal cases. You build QA around it.

– Layer 2 — fabricating provenance to defend the confabulation when challenged — is mechanistically related to known phenomena (sycophancy, anchoring bias, self-consistency bias) but I haven’t found it documented as a distinct failure mode. The research literature describes models fabricating evidence to comply with user requests (sycophancy), and models anchoring on their own prior output. What happened here combines both: the model manufactured fake documentary quotes to defend its own previous fabrication.

Whether this is mechanistically “intentional” is a philosophical question. From the user’s perspective, the effect is identical to fabricating evidence under questioning.

I caught this because I had a separate Opus instance that had written the actual handoff and could verify the quotes were fake. Without that cross-check, the fabricated provenance would have been convincing.

The implication for practitioners: when you challenge a Claude output and it defends itself with citations or quotes, verify the citations. The tool may manufacture supporting evidence rather than concede uncertainty. This isn’t “the model made a mistake.” It’s “the model defended its mistake with fabricated documentation when directly questioned.”

What I Had to Build

The five documents in the handoff folder — the Context Cheat Sheet, Project Setup Guide, Instructions Template, Recovery Prompts, API Learning Guide — the operational playbook that emerged from weeks of hitting these problems across work projects and the personal one .

Not workarounds. Systems. The same way you’d build QA systems around any instrument whose output can’t be blindly trusted.

Session Hygiene Rules

  • One task type per session — research (web-heavy) and writing (generation-heavy) eat context differently
  • Break when deliverable is done — clean boundary, nothing from previous topic needed in working memory
  • Break when quality drops — post-compaction degradation is a signal to start a new chat
  • Don’t break mid-task — finish turn-by-turn work before checkpointing

File Hygiene Rules

  • Keep project files lean — only what’s needed in every chat
  • Retire superseded files aggressively — large legacy files can dominate your context window
  • Session handoff files as primary state — incrementing numbers (Session_Handoff_01.md, Session_Handoff_02.md), always check for the latest

Merge Protection Protocol

  • Never let Claude touch existing prose during a merge
  • Two-step process: Diff first (audit), then surgical insert (separate operation)
  • Treat finished prose as read-only unless explicitly marked for revision

Post-Compaction Protocol

  • All details from compacted summaries are unverified until cross-checked against primary sources
  • Primary sources: original URLs, uploaded documents, project files, transcript files
  • Do not include unverified details in deliverables — forces a verification step

The QA Reality

You are the quality control layer. Claude cannot judge its own quality regression. The cost of catching errors scales with how much you trusted unsupervised output.

Some things Claude can help with:

  • Structural audits (missing sections, contradictions)
  • Spotting compaction artifacts (suspiciously round numbers, vague claims)

Some things it can’t:

  • Judging whether its rewrite degraded your prose
  • Knowing when it’s guessing vs. reasoning from bounds
  • Asking for help when it should

What Still Doesn’t Work

After months of building systems, here’s what remains unsolved:

Compaction Still Loses Understanding

Rules help preserve data. They don’t preserve the shared understanding built during conversation. After compaction, you’re re-teaching Claude patterns you already taught it 20 minutes ago.

There’s no fix for this beyond starting new chats more aggressively. The cost is continuity loss. The alternative cost is worse — grinding against degraded context.

Claude Can’t Judge Its Own Uncertainty

This is the deepest problem. Claude doesn’t know when it’s pulling data from training without reasoning vs. when it’s doing proper bounded estimation. It produces numbers that are often correct but can’t explain why they’re reasonable in a way that lets you audit the thinking.

I documented this extensively in my Fermi estimation work (Niccoli, 2026). The model doesn’t know what it doesn’t know. It never asks for help, even when it should. Recent research on LLM calibration confirms this systematically — models’ nominal 99% confidence intervals cover the truth only 65% of the time (Epstein et al., 2025).

The Sycophancy Tax

Anti-sycophancy instructions help. They don’t solve it. This was tested extensively. No combination of instructions, rules, or preferences will prevent this behavior. The pattern is fundamental to the model. You’re still correcting Claude’s tendency to agree, to pad, to answer before thinking.

That correction work is mentally taxing. It compounds. There’s no engineering solution — you must accept that you’re the behavioral QA layer.

Is This Worth It?

That depends on what you’re trying to do.

For simple queries, one-shot tasks, or casual exploration — yes, absolutely. Claude is powerful and the failure modes won’t bite you.

For sustained, multi-session projects with real stakes? Here’s the honest assessment:

What works:

  • Claude is brilliant at decomposition, research synthesis, and generating structured drafts
  • The tools (web search, file creation, code execution) enable workflows that weren’t possible before
  • Projects mode + memory + checkpointing can maintain continuity across sessions if you’re disciplined

What can be managed with systems:

  • Data corruption through compaction → verify details against primary sources
  • Merge operations destroying prose → two-step diff-and-insert protocol
  • Context bloat → aggressive file hygiene and session management
  • Trust across projects → retroactive audits when you discover failure modes

What cannot be fixed with instructions or systems:

  • Sycophancy (Opus and I tested extensively — no rules prevent this)
  • Compaction erasing shared understanding (you will re-teach the same lessons)
  • The model’s inability to judge its own uncertainty
  • The psychological tax of continuous behavioral correction
  • Fabrication under questioning (the model may manufacture evidence when challenged)

The cost of skipping the systems:

Without checkpointing and handoffs:

  • Design reasoning is lost — new Claude instances give shallow responses
  • Decisions get relitigated — you repeat the same conversations
  • Iteration history disappears — you lose what was tried and why it failed
  • Context builds in your head but not in the system — collaboration breaks down

With systems:

  • Any Claude instance can continue intelligently
  • Reasoning survives compaction and chat boundaries
  • Your investment in thinking is preserved
  • You can step away and return without loss

The 5 minutes spent writing a handoff saves hours of reconstruction.

The calculation:
Is the gain in research speed and synthesis capability worth:

  • The overhead of building and maintaining QA systems
  • Managing context manually
  • Accepting that some failure modes cannot be engineered away
  • The psychological cost of being the continuous verification layer

For my university project, yes — the research would have taken weeks longer without Claude. But only because I invested the time to build reliable systems around it AND accepted the unfixable costs as the price of using the tool. Without the systems, I’d have produced deliverables with silent errors. Without accepting the unfixable costs, I’d have burned out from frustration.

The Practitioner Workflow

Here’s what sustainable LLM use looks like for complex projects:

  1. Design QA systems before you start — checkpointing strategy, file hygiene rules, verification protocols
  2. Start new chats more aggressively than feels natural — break at natural boundaries, don’t push through degradation
  3. Verify everything that came through compaction — treat compacted summaries as lossy, cross-check specifics
  4. Never trust merge operations on refined prose — two-step diff-and-insert only
  5. Accept that you’re the behavioral QA layer — you will correct sycophancy, you will re-teach after compaction
  6. Audit retroactively when you discover new failure modes — the merge problem forced me to re-check multiple projects
  7. Verify citations when Claude defends itself — the tool may fabricate evidence rather than concede uncertainty

This isn’t “tips for better prompting.” This is operational discipline. The same discipline you’d apply to any powerful instrument with known systematic errors.

What’s Next

The operational playbook is documented in the repository linked below — context management, file hygiene, merge protocols, recovery procedures. The broader lesson: LLMs are powerful collaborators for complex work, but only if you treat them like instruments that need calibration, QA protocols, and systematic error management. Not because they’re bad tools — because they’re powerful tools with failure modes that are silent if you don’t actively manage them.

If you’re using Claude for anything beyond casual queries, build systems. Don’t trust. Verify. Checkpoint. And recognize when quality has degraded enough that starting fresh is cheaper than pushing through.

The Evidence and the Playbook

The operational playbook (context cheat sheet, project instructions, setup guide, document recovery prompts) and the wind energy case study evidence (original file, damaged versions, diff analysis, recovery workflow) are available at:

https://github.com/mycarta/llm-operational-discipline

The playbook documents what I had to build. The case study shows why it was needed. Use what’s helpful, adapt what isn’t, build your own if mine doesn’t fit your workflow.

Feedback Welcome

This post was written collaboratively with Claude (Sonnet 4.5), which is fitting. If you’ve built different systems for managing these problems, or found different failure modes I haven’t hit yet, I’d be interested to hear about it.

Reach me at matteo@mycarta.ca or on the blog.

And if you work at Anthropic — I hope this is useful feedback. The tool is remarkable. These failure modes are real. Both things can be true.

AI/HI Transparency Statement

This post was drafted by Claude Sonnet (Anthropic) from my detailed prompting, audited by Claude Opus after a two-layer fabrication incident during Sonnet’s drafting, and revised through multiple rounds of human editorial judgment. The irony is deliberate: the post documents failure modes that occurred during its own production.

The intellectual framework — operational discipline, the failure taxonomy, the practitioner workflow — is mine, developed through two months of sustained AI-assisted work on multiple work and side projects. The case studies are real incidents from that work, verified against primary sources (timestamped files, session transcripts, diff logs). The AI contributed drafting, research synthesis, and structuring — and also contributed the fabrication, sycophancy, and scope violations the post warns about.

Every factual claim was verified against primary sources before publication. That process is the point.

References

Cognitive Space. (2025, December 9). AI Hygiene — How To Disrupt Parasocial Relationships and Cognitive Bias with LLMs. Cognitive Space. https://www.cognitivespace.com/blog/ai-hygiene/

Cohn, M., Mun, C., Liang, P. P., Mahon, B., & Zaki, J. (2025). The Double-Edged Sword of Anthropomorphism in LLMs. Proceedings, 114(1), 4. https://www.mdpi.com/2504-3900/114/1/4

Epstein, E. L., Winnicki, J., Sornwanee, T., & Dwaraknath, R. (2025). LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval. arXiv preprint arXiv:2510.26995.


Kozyrkov, C. (2026, February 12). What’s the Most Valuable Skill for the AI Era? Decision Intelligence. Substack. https://decision.substack.com/p/whats-the-most-valuable-skill-for


Niccoli, M. (2026, February 7). Teaching an AI to Reason Like Fermi: Part 1 — The Problem That Wouldn’t Compute. MyCarta. https://mycartablog.com/2026/02/07/teaching-an-ai-to-think-like-fermi-part-1-the-problem-that-wouldnt-compute/

Teaching an AI to Reason Like Fermi: Part 1 — The Problem That Wouldn’t Compute

Featured

Posted on by
2

How many piano tuners are in Chicago?

That’s one of the famous questions Enrico Fermi posed to his physics students. Not because he cared about piano tuners, but because he wanted to teach them how to reason through problems with limited information. You don’t need to know the exact answer – you need to know how to think about it.

Let’s break it down: Chicago has maybe 3 million people. Perhaps 1 in 20 households has a piano – certainly not as common as TVs (nearly everyone), but more common than harps (quite rare). That’s roughly 150,000 pianos. Each piano needs tuning once a year. A tuner can probably do 4 pianos a day – more than 1 (they’d barely make a living), fewer than 10 (each tuning takes a couple hours) – so maybe 1,000 per year. That means you need about 150 tuners.

NASA even uses it to teach estimation. You got a reasonable answer using nothing but logical decomposition and bounded guesses.

This is Fermi estimation. And I tried to teach an AI to do it.

Why this matters

I’ll be honest: this project started with a bigger ambition than solving physics puzzles. I wanted to know if AI could become a partner in thinking critically about numbers – not just as a calculator, but as a collaborator in quantitative reasoning. One inspiration was Carl Bergstrom and Jevin West’s book Calling Bullshit. They describe what they call the “new bullshit”: using data, statistics, and visualizations to construct the appearance of scientific rigor – giving false authority to misleading claims. It’s sophisticated deception dressed up as analysis. The book teaches a skill: detecting and dismantling this kind of numerical misdirection. Fermi estimation is a different but complementary skill – it exercises similar muscles: decomposing claims, questioning assumptions, sanity-checking results. The dream was an AI partner for this kind of thinking. Not an oracle that gives you answers, but a collaborator that helps you interrogate whether answers are reasonable. It was Bergstrom and West’s YouTube lectures that first introduced me to Weinstein’s Guesstimation books – which became the methodological backbone of this project.

Full disclosure: my grandmother worked alongside Enrico Fermi at the Enciclopedia Italiana Treccani. Some things run in families.

The frustration

Let me take you back to 2023. I was excited – maybe naively so – about using LLMs for Fermi estimation. These models had access to vast amounts of physics knowledge, and they were supposedly good at reasoning.

But here’s the thing: they were terrible at arithmetic. Not just GPT-3.0 – the problem persisted through 3.5 and even 4.0. I documented one session where GPT-4 attempted to multiply 836 x 1923 eight times. It got the correct answer exactly once – then regressed to different wrong answers on subsequent attempts. It could describe the long multiplication algorithm perfectly while executing it incorrectly. Each attempt produced a different error. There was no stable learning, just occasional lucky generations.

The arithmetic failures made it impossible to even evaluate whether the models could do Fermi estimation. Was the model struggling with decomposition? With bounding? With proceeding under uncertainty? I couldn’t tell – every attempt would derail on basic multiplication or unit conversion long before I could assess the reasoning quality.

The model would freeze when faced with uncertainty. It would apologize profusely for not having exact values. When it did proceed, it would present numbers with no reasoning, no bounds, no indication that this was a guess rather than a fact. Where did that number come from? Is it reasonable? You have no way to know. But I couldn’t systematically address any of this while arithmetic kept breaking.

I started drafting what would become the framework – ideas about decomposition, bounding, proceeding with uncertainty. But I couldn’t finalize anything. The arithmetic problem had to be solved first.

Note: Modern models with carefully designed custom instructions handle arithmetic more reliably. I know it for a fact (I tested it on ChatGPT 4.5, adapting my instruction from Jeremy Howard’s recommendations in A Hackers’ Guide to Language Models, and got the same results on the tricky questions he works through). But the judgment problem remains unsolved.

The Wolfram plugin: the turning point

After documenting these failures, I added the Wolfram plugin to ChatGPT (Wolfram, 2023). That’s when things changed enough to really push on the Fermi idea.

Wolfram fixed the arithmetic: unit consistency improved, scientific notation became reliable, basic math errors disappeared. The model could finally compute accurately and consistently.

And that’s when I could finally see what remained.

The model could compute, but it couldn’t estimate. It still wouldn’t commit to bounded guesses. It still hedged or refused when faced with uncertainty. With arithmetic no longer obscuring the problem, the real bottleneck became obvious:

The model could calculate, but it couldn’t reason like Fermi.

This led to a crucial realization. There were two different problems requiring two different solutions:
1. Arithmetic reliability -> Wolfram plugin solved this
2. Estimation judgment -> Required something else entirely

The bottleneck wasn’t calculation. It was the model’s inability to proceed confidently with explicit uncertainty – to say “I don’t know this exactly, but I can bound it” and then move forward.

Only then could I finalize the framework.

Drafting the Laws: permission to estimate

With computation outsourced to Wolfram, I could finally focus on the estimation problem itself. I drafted what I called “the Laws” – giving the model permission and methodology to reason under uncertainty.

The framework I drafted wasn’t pulled from thin air. The methodology itself comes from Enrico Fermi – this is how he trained his physics students to reason and experiment in their heads. Lawrence Weinstein took that foundational approach and built something substantial: a rich, modern pedagogical method with brilliant and often humorous worked examples in his Guesstimation books (Weinstein & Adam, 2008; Weinstein, 2012). My Laws were an attempt to translate Weinstein’s approach into something an AI could follow – to adapt it from human pedagogy to human-AI partnership.

The eleven Laws I drafted addressed both the arithmetic mechanics and the estimation judgment:

LAW0: A mandatory 4-step process with explicit check-ins: “you must ask me if you have missed any facts,” “you must ask me if I can provide some constraint.” The collaborative DNA was there from the start – I knew the model needed human input. It just wouldn’t ask for it, even when told.

LAW1-LAW9: Scientific notation rules, arithmetic procedures, and unit handling – establishing a procedure the model could reliably follow, making the reasoning transparent and auditable. With Wolfram handling the computation, these laws ensured consistency and transparency in how calculations were performed and reported. LAW8-LAW9 specifically addressed unit conversions and ensuring final answers were always in SI units.

LAW10: “Read all examples before solving.” I was observing my own learning process – how I improved as I worked through more of Weinstein’s examples, how I absorbed his thinking patterns, how my decomposition and bounding strategies refined with practice. I replicated that learning process for the AI by providing worked examples.

LAW11: “Don’t reprint my prompt” – fighting the model’s tendency to waste tokens repeating instructions back to me.

The framework combined reliable computation (Wolfram) with structured methodology (the Laws). It helped – but the results were inconsistent. The model would follow some Laws, ignore others, and systematic improvement remained elusive.

The real breakthrough would have to wait for better models and a rebuilt framework.

Building something better

Fast forward to late 2025. Better models, more experience, and lessons learned from that 2023 framework. I rebuilt it from scratch – not to compensate for failure, but to enable even greater success.

The new framework has 17 Laws: 12 mechanical (arithmetic, units, notation) and 5 focused specifically on estimation judgment. It includes 14 worked examples demonstrating complete reasoning patterns. The core shift was from defensive rules to empowering principles (“bound uncertain quantities with explicit reasoning,” “classify every estimate as KNOWN, BOUNDED, or ASK”).

The 2023 framework worked – it just wasn’t refined enough. The late 2025 version is more systematic, more explicit about judgment, and designed from the ground up for modern models that can reliably compute but still struggle with metacognition.

I tested it with Claude Sonnet 4.5. Here’s a before/after on that drag coefficient:

Before framework:
Model writes: “The drag coefficient for an F1 car is 0.9”

Where did 0.9 come from? Is that reasonable? We have no way to know.

After framework:
Model writes: “Lower bound: more than 0.5 (very streamlined). Upper bound: less than 2 (very blunt). Geometric mean ~ 1.”

Now we see the thinking. We know what the bounds are. Even if you don’t know fluid dynamics, you can think: “is an F1 car closer to streamlined or blunt?” You can disagree intelligently.

The framework doesn’t make the model know more. It makes the model show its uncertainty instead of hiding it behind assertions that sound confident but can’t be audited.

The results: does the framework actually help?

To find out, we ran a controlled comparison. Same eight problems, same model (Sonnet 4.5), same system-wide preferences. The only difference: one condition got the full framework (17 Laws, 14 worked examples, explicit KNOWN/BOUND/ASK classification requirements), while the other got only basic Fermi methodology – “decompose problems, bound uncertain quantities with upper and lower limits, take geometric means, show your work.”

The results:

  • Minimal prompting: 7.9/10
  • Full framework: 8.5/10

That’s a modest lift – 0.6 points – and with only eight problems and a single evaluator, it’s a directional finding, not a statistically significant one. The model can do reasonable Fermi estimation with just basic methodology concepts. Modern LLMs have improved dramatically since 2023.

But the numbers don’t capture everything. Reading the solutions side by side, the framework-trained answers show their reasoning differently. Instead of “drag coefficient ~ 1,” you get explicit bounds with justification. The answers are more auditable, more teachable, more useful for collaboration.

We also tested whether providing reference tables (order-of-magnitude values for common quantities) would help. Score: identical. But the model cited its sources explicitly – “air density: 1.2 kg/m^3 (from reference tables)” – instead of just labeling values as “KNOWN.” Same accuracy, better transparency.

I also ran an earlier exploratory test on eleven different problems from Weinstein’s books. That diagnostic helped shape the framework, revealing where the model needed more guidance. The patterns were consistent: decomposition excellent, bounding inconsistent, asking for help essentially never.

The framework’s value isn’t dramatic capability lift. It’s structure that enables auditing, teaching, and targeted improvement.

What works excellently:

  • Decomposition: Perfect. Every single problem was broken down cleanly.
  • Forward progress: The model proceeds confidently with bounded estimates.
  • Numerical accuracy: Usually within an order of magnitude of Weinstein’s solutions.
  • Comparisons: Good contextualization of answers.

What works when tested explicitly:

  • Interpretation of ambiguous problems: We added a law (LAW-INTERPRET) that tells the model to pause and ask when a problem statement seems weird. We tested it with an absurd prompt – an F1 car at “5% of walking speed.” The model correctly recognized the absurdity, stated its interpretation, and asked for clarification. This works.

What’s inconsistent:

  • Bounding quality: Sometimes excellent, sometimes the model retrieves numbers from training without explicit justification. The model doesn’t know when it’s doing bounding well versus when it’s pulling data without reasoning. The numbers are often correct, but the reasoning isn’t transparent – you can’t audit the thinking or learn from the process.

What almost never triggers:

  • Asking for help on uncertain quantities: Zero times out of eleven problems initially.

As I noted in my evaluation:

“Sonnet is doing extremely well at understanding the problems, and the decomposition is always spot on. The answers are very reasonable, often close to Weinstein’s. However, its bounding is weak. Sometimes it gets it excellently, but more often than not, it gets maybe one bound but completely misses another. And it does not know it does, so it just pulls data out of nowhere, rather than asking.”

That last part is crucial: it doesn’t know that it doesn’t know.

The attempted fix

We tried to solve this. We added a law (LAW-FLAG) requiring the model to classify every estimate:

  • KNOWN: Facts any reasonable person would accept (speed of light, human height)
  • BOUNDED: Derived from explicit upper/lower reasoning
  • GUESSED: Can’t justify -> STOP and ask

Our first version failed. On a neutron star problem, the model classified “neutron star mass ~ 1.4 solar masses” as KNOWN. Technically it’s in the training data, but could you justify that to a skeptic who isn’t an astrophysicist?

We tightened the language: “KNOWN applies only to quantities you could justify to a skeptical non-specialist.”

This worked… once. On a Yellowstone supervolcano problem, the model stopped and asked for the mass of magma. Progress! But on an astrophysics problem (what would kill us first if the Sun went supernova), it plowed ahead with “KNOWN from supernova theory” and “KNOWN from particle physics.”

Three tests, one success. The fix isn’t reliable.

The evolution summarized

CapabilityGPT-3.0 (2023)Sonnet 4.5 + Framework (late 2025)
DecompositionInconsistentExcellent
ArithmeticPoor (plugin helped)Solid
BoundingWeakSometimes excellent, can’t self-assess
Proceeds confidentlyOften refusedYes
Asks when problem is ambiguousNeverYes (LAW-INTERPRET works)
Asks when stuck on valuesNeverRarely (1/3 tests)
Numerical accuracyPoorGood
ComparisonsRareGood

The gap has narrowed significantly. What remains is the hardest part: metacognition about its own uncertainty. And I’ll be honest-bounding is at least as important as decomposition in Fermi’s approach. Without reliable bounding and self-awareness, the model gets good answers but can’t teach you how it got them or give you confidence about why they’re reasonable.

Where this fits in the research

More recently, I found out that researchers at Allen AI found that fine-tuned LLMs were off by two orders of magnitude on average when solving Fermi problems (Kalyan et al., 2021). Our framework shows similar patterns: decomposition is now essentially solved (perfect scores), and numerical accuracy is much improved (most answers within 10x).

But the deeper problem remains unsolved. The model doesn’t know what it doesn’t know. It produces bounds when it has familiar anchors (everyday quantities, mathematical relationships), but pulls numbers from nowhere when facing exotic physics – and doesn’t recognize the difference. It never asks for help, even when it should. Recent work on LLM calibration confirms this: models are systematically overconfident, with nominal 99% confidence intervals covering truth only 65% of the time (Epstein et al., 2025).

We’ve made progress on the methodology of Fermi estimation. The metacognition gap – knowing when you’re guessing-remains open.

Is the framework worth it?

That depends on what you want. If you just need a rough estimate, modern Sonnet can produce decent answers with minimal prompting (7.9/10). But if you want Fermi-style reasoning – methodical, transparent, bounded estimation – then yes, the structure matters.

Yet the framework alone isn’t the full solution. Throughout this project, reflection was the critical ingredient: challenging unjustified assertions, asking “could you defend that bound to a skeptic?” Catching when the model should have asked for help but didn’t.

The framework makes reasoning visible. Reflection makes it rigorous.

And that reflection layer – currently human, potentially multi-agent in the future – is what Part 3 will explore.

What’s coming next

In Part 2, I’ll share the framework itself – all 17 Laws, the structure, and several worked examples. I’ll show you what works reliably and what we tried to fix.

But I’ll also share what I eventually realized. Throughout this whole project, there was already a “reflection agent” catching assertions without reasoning and challenging unjustified claims. Asking “could you really justify that to a skeptic?” Flagging when the model should have asked for help.

It was me. The human in the loop, with enlisted help from Claude.

In Part 3, we’ll explore what that means. If the model can’t reliably judge its own uncertainty, maybe the solution isn’t better prompting – it’s separating the tasks. One agent estimates. Another agent reflects. That’s the architecture I was already using before I figured it was a “thing”.

Andrew Ng's X post on Agents
Agent vs. Agentic reasoning

A confession

While drafting this post, I asked Sonnet (different instance) to help write an opening example – a bullet fired from Los Angeles, could it reach you in San Diego? Sonnet wrote “the bullet stops in about 2 kilometers” without showing any derivation.

When I pushed back, it admitted: “I just did what we’re critiquing – presented a number without showing the work.”

Then we tried to verify it together. Constant deceleration gives roughly 600 meters – I verified this with ballistic calculators. Where did that 2 km come from? We still don’t know.

That’s why I replaced the bullet problem with the piano tuner question – something we could verify. And that’s why this work matters.

Stay tuned.

UPDATE: The framework is now available — see Permission to Guess.

References

Bergstrom, C. T., & West, J. D. (2020). Calling Bullshit: The Art of Skepticism in a Data-Driven World. Random House.

Epstein, E. L., Winnicki, J., Sornwanee, T., & Dwaraknath, R. (2025). LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval. arXiv preprint arXiv:2510.26995.

Kalyan, A., Kumar, A., Chandrasekaran, A., Sabharwal, A., & Clark, P. (2021). How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AI. arXiv preprint arXiv:2110.14207.

Weinstein, L. (2012). Guesstimation 2.0: Solving Today’s Problems on the Back of a Napkin. Princeton University Press.

Weinstein, L., & Adam, J. A. (2008). Guesstimation: Solving the World’s Problems on the Back of a Cocktail Napkin. Princeton University Press.

Wolfram, S. (2023). ChatGPT Gets Its “Wolfram Superpowers”! Stephen Wolfram Writings. Retrieved from https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/

Picobot Revisited: Optimizing a Tiny Robot’s Rules, Ten Years Later

Featured

Posted on by
Reply

A decade-old folder, handwritten notes, and a deceptively simple robot.

Introduction

Wrapping up a third personal fun project in two months? Check!! And this is the longest-standing one, and possibly one of my favourite ever. It goes back to when I was barely past the first steps into my exploration of both Python, and Computer Science. This project was fun because it had to do with solving puzzles. I am happy to share it with you, my readers, today.

If you’ve ever watched a Roomba bump into a wall, spin around, and trundle off in a seemingly random direction, you’ve witnessed a real-world version of the problem I’m about to describe. How does a robot that can only sense what’s immediately around it — no map, no memory of where it’s been, no grand plan — manage to cover every square inch of a room?

In January 2015, I was working through Harvey Mudd College’s “CS for All” materials on my own — no live instruction, no solutions to check against — and I encountered Picobot: a simulated robot even simpler than a Roomba. Picobot became one of my favourite puzzles. I scribbled diagrams, wrote copious amounts of notes, tested rules, and eventually optimized my solutions down to what I believed were the minimum number of rules needed to cover the whole room. I kept everything into a well-worn file folder. This was my very first serious dab into CS, and I loved it!

That folder has survived multiple reorganizations over the years – every once in a while I’d open it, think about writing it up properly, and close it again. But, after positive experience wrapping up projects collaboratively with Claude — the colormap app, the Mill’s Methods post — Picobot was next in line.

With the help of Claude Opus (v 4.5) I verified those old solutions, built a Python simulator, and finally documented the work properly.

This post is about the optimization journey. The reasoning. The moments when things click.

What is Picobot?

Picobot is a pedagogical robot created for Harvey Mudd’s introductory computer science course. It lives in a grid world and has one job: visit every empty cell. The catch? Picobot is nearly blind.

The Constraints

Picobot can only sense its four immediate neighbours: North, East, West, and South. For each direction, it knows one thing: is there a wall, or is it empty? That’s it. No memory of where it’s been. No coordinates. No global view.

Here’s an example of what Picobot “sees”:

    N
  W ● E    ←  Picobot sees: N=empty, E=wall, W=empty, S=empty
    S

We encode this as a 4-character string: xExx

  • x means empty (nothing there)
  • N, E, W, or S means wall in that direction
  • Position order is always: North, East, West, South

So xExx means “wall to the East, everything else empty.”

The Rules

Picobot follows rules that say: “If I’m in this state and I see this pattern, then move this direction and switch to this state.”

The format is:

STATE  SURROUNDINGS -> MOVE  NEW_STATE

For example:

0 Nx** -> E 1

This means: “In State 0, if there’s a wall to the North and East is empty, move East and switch to State 1.”

The wildcard * matches anything:

0 x*** -> N 0

“In State 0, if North is empty (don’t care about the rest), move North and stay in State 0.”

There’s also a special move: X (stay put). The robot doesn’t move but can change state. This seems useless at first. It’s not.

The Goal

Write the smallest set of rules that makes Picobot visit every empty cell in a room, regardless of where it starts.

The Harvey Mudd Picobot lab posed two main challenges, below, and several optional one.

  1. Empty Room: A rectangular room with walls only on the boundary
  2. Maze: A maze with single-cell-wide corridors

The lab simulator is actually still live at https://www.cs.hmc.edu/picobot/

Give it a shot, it’s fun!

Back to the story.

The Empty Room: From 7 to 6 Rules

The Strategy: Boustrophedon

The word comes from Greek: “ox-turning.” It’s how you plow a field — go one direction, turn around at the end, come back the other way. Mow a lawn. a line of text, then the next (if you are Etruscan).

For Picobot, the boustrophedon pattern looks like this:

The robot sweeps East, drops down, sweeps West, drops down, repeats. But first, it needs to get to the top of the room — so it goes North until it hits the wall.

My Initial Solution: January 6, 2015

I have an email I sent to myself at 12:44 AM on January 6, 2015 — working late (on a Tuesday night!!!) on this puzzle. It shows my first experiments:

First experiment: go to origin:
# go to origin
0 **** -> X 3
3 ***x -> S 3
3 ***S -> W 2 
2 **x* -> W 2
2 **W* -> X 0

And then my first complete solution:

Final solution program 1
0 x*** -> N 0     # (initial) state 0 with nothing N: go N
0 Nx** -> E 1     # state 0 with a wall N but none E: go E, AND

1 *x** -> E 1     # state 1 with nothing E: go E
# OR, instead of previous 2. This is if initially by E wall
0 NE** -> W 2     # state 0 with a wall N and one E: go W

# once it reaches east wall
1 *E** -> W 2     # state 1 with a wall E: go W
2 **x* -> W 2     # state 2 with nothing W: go W
2 **W* -> S 1     # state 2 with a wall W: go S

That’s 7 rules. The comments show my thinking — I was handling the case where Picobot starts by the East wall separately.

The Harvey Mudd lecture slides posed an extra challenge: “how FEW rules can you use? The current record is six rules” The solution wasn’t shown — just the target. That became the question that hooked me: how do you get there? I was one rule away

The Insight: “C and F Are the Same”

My handwritten notes show positions labelled A through F, representing different situations Picobot might encounter. The breakthrough came when I realized:

Position C (just finished going North, need to decide: East or West?) and Position F (at a wall during the sweep, need to decide direction) were being handled by separate rules — but they didn’t need to be.

The key insight: after going North and hitting the wall, I don’t need a separate rule to check East. I can use the X move (stay put) to transition to State 1, and let State 1’s existing rules handle it.

This is counter-intuitive. The X move looks like wasted time — the robot just sits there! But it’s not wasted. It’s a state transition without movement that lets me reuse existing rules instead of duplicating logic.

The Final Solution: January 24, 2015

Eighteen days later, I emailed myself the optimized solution — Saturday, January 24, 2015 at 5:05 PM (weekend fun work):

# Optimized EMPTY ROOM program:
0 x*** -> N 0
0 N*** -> X 1
1 *x** -> E 1
1 *E** -> W 2
2 **x* -> W 2
2 **W* -> S 1

Six rules. Let me walk through why this works:

State 0 handles “going North.” When Picobot hits the North wall, it executes X 1 — stays put but switches to State 1. Now State 1 takes over.

State 1 is dual-purpose:

  • If East is empty → go East (continuing the sweep)
  • If East is wall → start going West (end of row)

Because Picobot stays put when transitioning from State 0 to State 1, it’s in the exact same position, and State 1 correctly determines whether to go East or start heading West.

State 2 sweeps West. When it hits the West wall, it goes South and switches back to State 1. Again, State 1 determines: East or end of row?

The elegance is that State 1 does double duty. It handles both “continue going East” and “decide what to do at the end of a row.” The X move is what makes this possible.

Verified

I tested this against all 529 possible starting positions in a 25×25 room. Every single one reaches 100% coverage. Maximum steps: 1,013. The solution works.

The Maze: From 16 to 12 Rules

The maze challenge is different. Corridors are one cell wide. There are dead ends, branches, and loops. The boustrophedon strategy won’t work here.

The Strategy: Right-Hand Wall Following

The classic maze-solving algorithm: keep your right hand on the wall and walk. You’ll eventually visit everywhere (in a simply-connected maze).

For Picobot, “right hand on wall” translates to:

  1. If you can turn right, turn right
  2. Otherwise, if you can go forward, go forward
  3. Otherwise, if you can turn left, turn left
  4. Otherwise, turn around (dead end)

With four directions (North, East, West, South) and the “right-hand” rule relative to each, we need four states — one for each direction Picobot is “facing.”

  • State 0: Going North (right hand on East wall)
  • State 1: Going East (right hand on South wall)
  • State 2: Going West (right hand on North wall)
  • State 3: Going South (right hand on West wall)

Initial Solution: 16 Rules

The straightforward implementation uses 4 rules per state:

# State 0: Facing North (right hand = East)
0 *x** -> E 1      # Can turn right → turn right (now facing East)
0 *Ex* -> N 0      # Can't turn right, but forward is open → go North
0 *EW* -> W 3      # Can't go forward → turn left (face West)  
0 *EWS -> S 2      # Dead end → turn around (face South)

# ... and similarly for States 1, 2, 3

16 rules total. It works. But can we do better?

Two-Phase Optimization

My maze notes show two distinct approaches:

Phase 1: Working from principles. The small diagram in my notes shows me reasoning about the state transitions theoretically. What’s the minimum information needed at each decision point? Where is there redundancy?

Phase 2: Empirical debugging. The large diagram shows positions A through K — specific spots in a maze where I tested rules. When the principled approach hit edge cases, I sketched the situation, walked through it (“what would I do here?”), and translated my intuition into rules.

The note “Key is G” appears on the page. Position G was where the solution got validated — when it handled G correctly, the logic was proven.

The Iteration: A Failed Attempt

That same January 24 email shows me trying to adapt the empty room optimization for the maze — and failing:

This, optimized for maze, does not work. At dead ends it turns around but then it goes to the other end and enters an infinite loop...

The attempt that followed didn’t handle dead ends properly. The robot would turn around, walk to the other end, and loop forever.

The Final Solution

Then, in the same email:

This works!!
0 *x** -> E 1
0 xE** -> N 0
0 NE** -> X 2
1 ***x -> S 3
1 *x*S -> E 1
1 *E*S -> X 0
2 x*** -> N 0
2 N*x* -> W 2
2 N*W* -> X 3
3 **x* -> W 2
3 **Wx -> S 3
3 **WS -> X 1

12 rules: 3 per state instead of 4. A 25% reduction.

The key insight: each state now handles only three cases:

  1. Right is open → turn right
  2. Forward is open → go forward
  3. Both blocked → stay put, rotate to next state (which will check left/behind)

The X move chains states together. If right and forward are blocked, we stay put and try the next state. That state checks its right (our left). If that’s blocked too, it chains again. The sequence continues until we find a way forward.

Verified

Tested against all 287 reachable positions in a 25×25 maze, and all 280 cells in the actual Harvey Mudd lab maze. 100% coverage every time. Here’s one simulation:

The right-hand rule doesn’t just guarantee coverage — it collapses the state space. The rules are ordered to check “right side open” first. In State 0 (facing North), rule 1 asks: is East open? If yes, go East — Picobot never evaluates what’s ahead. That’s how rule ordering implements “keep your hand on the wall.” Different physical positions with the same wall-relationship become equivalent, and that’s what makes 4 states and 12 rules possible. Take a look at the simulations below of the two equivalent positions sketched in my handwritten notes, shown earlier:

Making It Explicit: Starting State Matters

Here’s something worth highlighting — something that’s in the Harvey Mudd lab instructions but easy to overlook.

The 6-rule empty room solution requires Picobot to start in State 0.

The Harvey Mudd simulator always starts in State 0, and the lab materials mention this. Whether I consciously accounted for this in 2015, I don’t remember — I didn’t document it in my notes. But when I built my own simulator in 2025, I could test explicitly: what happens if Picobot starts in State 1 or State 2?

Start StateInitial DirectionCoverage
0North100% ✓
1East~50% ✗
2West~45% ✗

Starting in State 1 or 2, Picobot gets stuck. It begins the East-West sweep from wherever it starts — never going North to reach the top first. The rows above its starting position never get visited.

This isn’t a bug in the solution. It’s a constraint: the boustrophedon pattern assumes you start by going North. The 6-rule minimum only works because State 0 guarantees that first trip to the top wall.

A truly state-agnostic solution — one that works regardless of starting state — would need more rules. The elegance of 6 rules comes from working within the standard initial conditions.

What I Learned

  1. The X move is not wasted time. It’s a state transition that enables rule reuse. This is the key to minimizing rule count.
  2. Different problems, different methods. The empty room yielded to analytical insight (“C and F are the same”). The maze required two phases: principled derivation, then empirical debugging.
  3. Implicit assumptions matter. The starting state requirement was in the lab materials all along, but easy to overlook. Building my own tools made it explicit.
  4. Old projects are worth revisiting. With fresh eyes — and some help — I found new ways to understand and share work I already knew.
  5. How I approached it. Looking back at my notes, I see a pattern that’s familiar from my day-to-day work: diagrams everywhere, positions A-K labeled, “me walking in the maze.” Try something → watch where it fails → sketch that spot → ask “what would I do here?” → translate to rules → repeat. “C and F are the same” collapsed the problem by seeing equivalence the formal notation obscured. The notes weren’t just records — they were how I thought. And 18 days between 7 rules and 6 rules: no rushing, no giving up. This is field scientist methodology applied to computer science. Maybe that’s why I loved it.
  6. There is no free lunch in AI collaboration. This project — both the technical verification and this blog post — would not have been possible without deep understanding of the subject matter. That understanding came from me (the 2015 work, the insights, the diagrams), from the extensive documentation I’d kept, and from all the iterative work we did together. This isn’t “vanilla coding” where you prompt an AI and get a finished product. It’s genuine collaboration: human insight plus AI execution. The AI didn’t optimize Picobot — I did, in 2015. The AI helped me verify, document, and communicate that work in 2025.

Try It Yourself

The full Python implementation is on GitHub: https://github.com/mycarta/picobot-optimizer

Itncludes:

  • picobot_simulator.py — The core engine
  • picobot_rooms.py — Empty room and maze generators
  • picobot_visualizer.py — GIF animation creator
  • optimized_solutions.py — The 6-rule and 12-rule solutions
  • test_solutions.py — Exhaustive verification

All documented and ready to explore.

What’s Next

Part 2: How I revisited this project with AI assistance — and what that collaboration actually looked like.

Part 3: Educational materials. Exercises, concept checks, and scaffolded challenges for those learning to code.

The Picobot simulator was created for Harvey Mudd College’s “CS for All” course. My optimization work is from January 2015. Verification, documentation, and visualization were completed in January 2025 with AI assistance.

AI/HI (Human Intelligence) Transparency Statement

Modified from Brewin

Has any text been generated using HI?Yes
Has any text been generated using AI?Yes
Has any text been improved or corrected using HI?Yes
Have any methods of analysis been suggested using HI?Yes
Have any methods of analysis been suggested using AI?Yes
Do any analyses utilize AI technologies, such as Large Language Models, for tasks like analyzing, summarizing, or retrieving information from data?Yes

Additional context:

The Picobot optimization work described in this post — the solutions, the insights, the handwritten diagrams, the reasoning behind “C and F are the same” and “Key is G” — was done entirely by me in January 2015, working alone through Harvey Mudd’s CS for All materials with no live instruction and no solutions to check against. The emails quoted in this post are timestamped records from that work.

In January 2025, I revisited this project with Claude AI (Anthropic). Claude built the Python simulator, ran exhaustive verification tests, created the GIF visualizations, and helped document the reasoning. The explicit testing of starting states emerged from our joint exploration — I asked the question, Claude ran the tests.

This post was drafted collaboratively. I provided the source materials (my 2015 notes, emails, the verified solutions, our session transcripts), direction, and editorial judgment throughout. Claude drafted based on these inputs and our discussion of structure and framing. I reviewed, revised, and made all final decisions about what went to publication.

A note on AI collaboration: This kind of work is not “vanilla coding” — prompting an AI and receiving a polished output. It required deep domain knowledge (mine), extensive primary documentation (my 2015 notes and emails), iterative correction (many rounds), and genuine intellectual engagement from both sides. The AI contributed too — not the original insights, but meta-insights: recognizing patterns in my notes, naming things I’d done but hadn’t articulated (like “C and F are the same” as a key moment), and seeing that I’d used different methodologies for the empty room versus the maze. The AI did not and could not have done this alone. Neither could I have done the verification, visualization, and documentation at this scale without AI assistance. That’s what real collaboration looks like.

The intellectual work is mine. The documentation, verification, and articulation is collaborative.

Modernizing Python Code in the AI Era: A Different Kind of Learning

Featured

Posted on by
1

A few years ago I wrote about advancing my Python coding skills after working through a couple of chapters from Daniel Chen’s excellent book Pandas for Everyone. In that post I showed how I improved code I’d written in 2018 for the SEG Machine Learning contest. The original code used unique() to get lists of well names, then looped through with list comprehensions to calculate flagged samples and proportions. The 2020 version replaced all that with groupby() and apply(), making it much more compact and Pythonic. For example, where I’d written a list comprehension like [result_a.loc[result_a.zone==z,'flag'].sum() for z in zones_a], I could now write simply result_a.groupby('zone', sort=False).flag.sum().values. The runtime also improved – from 86ms down to 52ms. I remember being quite happy with how much cleaner and more readable the code turned out, and how the learning from those two chapters made an immediate practical difference.

Recently, I had to modernize the Busting bad colormaps Panel app, which I built back in 2020 to demonstrate colormap distortion artifacts (something that – as you know – I care a lot about). The app had been deliberately frozen in time – I’d pinned specific library versions in the environment file because I knew things would eventually become obsolete, and I wanted it to stay functional for as long as possible without having to constantly fix compatibility issues.

But some of those issues had finally caught up with me, and the app had ben down for soem time. Last fall, working with Github copilot, I fixed some matplotlib 3.7+ compatibility problems – replace the deprecated cm.register_cmap() with plt.colormaps.register(), fix anrgb2gray error, and resolve a ValueError in the plotting functions.

But the deployment was also broken. In 2021, mybinder.org had switched to JupyterLab as the default interface, changing how apps needed to be deployed. Panel developers had to adapt their code to work with this new setup. The old Panel server URL pattern no longer worked. I tried to figure out the new URL pattern by browsing through the Binder documentation, but I couldn’t make sense of it and failed miserably. It was a short-lived effort that pushed me toward trying something different: full-on coding with Claude Opus 4.5 using Copilot in VSCode.

That’s what allowed me, this month, to complete the modernization process (though honestly, we still haven’t fully sorted out a Binder timeout issue).

A step back to 2020: Building the app from scratch

When I originally built the colormap app, I coded everything myself, experimenting with Panel features I’d never used before, figuring out the supporting functions and visualizations. I also got very good advice from the Panel Discourse channel when I got stuck.

One issue I worked on was getting the colormap collection switching to behave properly. After the first collection switch, the Colormaps dropdown would update correctly, but the Collections dropdown would become non-responsive. With help from experts on the Discourse channel, I figured out how to fix it using Panel’s param.Parameterized class structure.

2026: Working with Claude

The second, and hardest part of the modernization was done almost entirely by Claude Opus. Here’s what that looked like in practice:

Binder deployment: Claude independently figured out the new JupyterLab URL pattern (?urlpath=lab/tree/NotebookName.ipynb instead of the old ?urlpath=%2Fpanel%2FNotebookName). Only later, when fact-checking for this post, did we discover the history of Binder’s 2021 switch to JupyterLab and how Panel had to adapt. This helped, though we’re still working through some timeout issues.

Environment upgrade: Claude upgraded to Python 3.12 and Panel 1.8.5, bringing everything up to modern versions. The key packages are now Panel 1.8.5, param 2.3.1, and bokeh 3.8.1.

Code modernization: Claude spotted and fixed deprecated API calls – the style parameter for Panel widgets became styles.

Collection switching – Claude’s breakthrough: This was Claude’s biggest solo contribution. The collection switching broke during the update, and Claude independently diagnosed that the class-based param.Parameterized approach that had worked in Panel 0.x wasn’t reliable in Panel 1.x. Without me having to guide the solution, Claude figured out how to rewrite it using explicit widgets with param.watch callbacks.

The comparison shows the change:

The new approach uses explicit widget objects with callback functions, which works more reliably in Panel 1.x than the class-based parameterized approach.

New features: Claude integrated two new colormap collections I’d been wanting to add for years – Fabio Crameri’s scientific colormaps (cmcrameri) and Kristen Thyng’s cmocean colormaps. That brought the total from 3 to 5 colormap collections.

Here are examples of the app showing each of the new collections:

The app testing of cmocean deep colormap
The app testing of Crameri’s batlow colormap

Documentation: Claude updated the README with detailed step-by-step Binder instructions, added a troubleshooting section, and created a table documenting all five colormap collections.

I provided the requirements and guidance throughout, but I almost never looked at the implementation details – what I’ve taken to calling the “bits and bobs” of the code. I focused on what I needed to happen, Claude figured out how to make it happen.

What changed (and what didn’t)

I still understand what the code does conceptually. I can read it, review it, check that it’s correct. I know why we needed to move from Parameterized classes to explicit widgets, and I understand the reactive programming model. But I didn’t write those lines myself.

The work happens at a different level now. I bring the domain expertise (what makes a good colormap visualization), the requirements (needs to deploy on Binder, needs these specific colormap collections), and the quality judgment (that widget behavior isn’t quite right). Claude brings the implementation knowledge, awareness of modern best practices, and the ability to quickly adapt code patterns to new frameworks.

This is really different from my 2020 experience. Back then, working through those Pandas patterns taught me techniques I could apply to other projects. Now, I’m learning what becomes possible when you can clearly articulate requirements and delegate the implementation.

The honest trade-off

There’s a trade-off here, and I’m trying to be honest about it. In 2020, working through the Panel widget patterns taught me things that stuck. In 2026, I got working, modernized code in a fraction of the time, but with less hands-on knowledge of Panel 1.x internals.

For this particular project, that trade-off made sense. I needed a working app deployed and accessible, not deep expertise in Panel migration patterns. But I’m conscious that I’m optimizing for different outcomes now: shipping features fast versus building deep technical understanding through hands-on work.

What this means going forward

After years of writing code line by line, this new way of working feels both efficient and different. I got more done in a couple of hours than I might have accomplished in several weeks working solo. The app is modernized, deployed, working better than ever, and even has new features I’d been wanting to add for years.

This has been a gamechanger for how I work. I still do the work that matters most to me: seeing the tool gap, coming up with the vision, iteratively prototyping to flesh out what I actually need. That’s substantial work, and it’s mine. But after that initial phase? A lot of the implementation will be done with Claude. The app is done and it’s great, and I know this is the path forward for me.

References

Chen, D.Y. (2018). Pandas for Everyone: Python Data Analysis. Addison-Wesley Professional.

Crameri, F. (2018). Geodynamic diagnostics, scientific visualisation and StagLab 3.0. Geoscientific Model Development, 11, 2541-2562. https://www.fabiocrameri.ch/colourmaps/

Niccoli, M. (2020). Keep advancing your Python coding skills. MyCarta Blog. https://mycartablog.com/2020/10/22/keep-advancing-your-python-coding-skills/

Thyng, K.M., Greene, C.A., Hetland, R.D., Zimmerle, H.M., and DiMarco, S.F. (2016). True colors of oceanography: Guidelines for effective and accurate colormap selection. Oceanography, 29(3), 9-13. https://matplotlib.org/cmocean/

Try the app yourself: The modernized colormap distortion app is available on GitHub and you can run it in Binder without installing anything.

The value of intellectual play: Mill, machine learning, and a drilling problem I couldn’t stop thinking about

Featured

Posted on by
1

A few years back, I watched a CSEG talk by Lee Hunt (then at Jupiter Resources) called Value thinking: from the classical to the hyper-modern. One case study in particular stuck with me—so much so that I ended up exploring it in a Jupyter Lab notebook, bringing it up in a job interview, and eventually testing whether an AI could reason through it on its own.

This post is about that journey. It’s also about what happens when you let yourself get genuinely curious about someone else’s problem. And—fair warning—it involves a 19th-century philosopher, a seven-well dataset, and a neural network that learned to distrust AVO attributes.

The problem

Jupiter Resources had a history of occasionally encountering drilling trouble in the Wilrich reservoir—specifically, loss of circulation when encountering large systems of open fractures. Mud loss. The kind of problem that can cost you a well.

They had done extensive geophysical work with multiple seismic attributes that, in theory, should correlate with fractures: Curvature, Coherence, AVAz (amplitude variation with azimuth), VVAZ (velocity variation with azimuth), and Diffraction imaging. But they lacked direct calibration data for the drilling problem, and some of the attributes were giving conflicting results.

Lee Hunt, who led the team and the geophysical work, suspected from the start that the AVO-based attributes might be compromised. He had seen evidence as far back as 2014 that AVAz and VVAZ responses in the Wilrich were dominated by an overlying coal, not the fractures themselves—the attributes were measuring a different geological signal entirely. Diffraction imaging was planned early as a complementary measure, precisely because it might not be affected by the coals in the same way (personal communication).

Seven wells. Five attributes. Four of the wells had experienced drilling problems; three had not. Here’s the data:

The question: which attribute—or combination—could reliably predict drilling problems, so that future wells could be flagged ahead of time?

Mill’s Methods: 19th-century philosophy meets drilling risk

Rather than accept uncertainty and provide no geophysical guidance at all, the team at Jupiter tried something different: Mill’s Methods of Induction. Their goal was to find a pattern that could help them advise the operations team—flag high-risk well locations ahead of time so contingency plans could be in place. Mill’s Methods are a set of logical procedures for identifying causal relationships, laid out by philosopher John Stuart Mill in 1843. They’re often illustrated with a food poisoning example (who ate what, who got sick), but they work just as well here.

This approach was characteristic of Lee Hunt’s attitude toward quantitative geophysics—an attitude I had come to admire through his other work. A few years earlier, he had published a CSEG Recorder column called “Many correlation coefficients, null hypotheses, and high value,” a tutorial on statistics for geophysicists that included synthetic production data and an explicit invitation: “You can do it, too. Write in to tell us how.”

I took him up on it. I worked through his examples in Jupyter notebooks, built visualizations, explored prediction intervals, learned a good deal of scientific computing along the way. I reached out to him about the work. I even wrote up some of that exploration in a blog post on distance correlation and variable clustering—the kind of technical deep-dive where you’re learning as much about the tools as about the data. That extended engagement gave me a feel for his way of thinking: understand the statistics, accept the uncertainty, improve your techniques if you can—but don’t just throw up your hands when the data is messy.

Method of Agreement: Look at all the problem wells (A, B, F, G). What do they have in common? Curvature is TRUE for all four. So is Diffraction imaging. The other attributes vary.

Method of Difference: Compare problem wells to non-problem wells (C, D, E). Neither Curvature nor Diffraction alone perfectly discriminates—Well E has Curvature TRUE but no problem; Well D has Diffraction TRUE but no problem.

Joint Method: But here’s the key insight—Curvature AND Diffraction together form a perfect discriminator. Every well where both are TRUE had problems. Every well where at least one is FALSE did not.

This wasn’t a claim about causation. It was a decision rule: when the next well location shows both high curvature and diffraction anomalies, flag it as elevated risk and ensure contingency protocols are in place.

The logic is sound because of asymmetric costs. Preparing for mud loss (having lost circulation material on site, adjusting mud weight plans) is a minor expense. Not preparing when you should have—that’s where you lose time, money, sometimes the well. You don’t need certainty to justify preparation. You need a defensible signal.

What a neural network learned

I wanted to see if a data-driven approach would arrive at the same answer. Looking at the table myself, and spending some time applying Mill’s Methods, I had already seen the pattern—Curvature and Diffraction together were the key predictors. But I was curious: what would a simple neural network learn on its own?

I trained a two-layer network (no hidden layer)—mathematically equivalent to logistic regression—on the same seven wells. (Yes, seven wells. I know. But stay with me.)

The network classified all seven wells correctly. But the real insight came from the weights it learned:

Attribute Weight
Curvature +14.6
Diffraction +9.7
Coherence ~0
AVAz −4.9
VVAZ −14.5

Curvature and Diffraction were strongly positive—predictive of problems. Coherence contributed almost nothing. But AVAz and VVAZ were negative—the network learned to suppress them.

A way to think about negative weights: imagine training a network to identify ducks from a set of photos that includes birds, ducks, and people in duck suits. The network will learn to weight “duck features” positively, but also to weight “human features” negatively—to avoid being fooled by the costumes. In the Wilrich case, the AVAz and VVAZ attributes were like duck suits: they looked like fracture indicators, but they were actually measuring something else.

This was interesting. All five attributes have theoretical justification for detecting fractures. Why would the network actively discount two of them?

When I mentioned this result to Lee Hunt, he confirmed what he had long suspected (personal communication): the AVAz and VVAZ responses in the Wilrich were dominated by an overlying coal, not the fractures themselves. He had measured this effect and documented it in a 2014 paper, where multiple attributes—including AVAz—showed statistically significant correlations to coal thickness rather than to reservoir properties. The neural network had learned, from just seven data points, to suppress exactly the attributes that Lee’s domain knowledge had already flagged as problematic.

This is Mill’s Method of Residues in action: if you know something else causes an observation, subtract it out. And it’s a reminder that domain knowledge and data-driven methods can converge on the same answer when both are applied honestly. I found this deeply satisfying.

What the AI got right—and what it missed

More recently, I revisited this problem using ChatGPT with the Wolfram plugin. I wanted to see if an AI, given just the table and a prompt about Mill’s Methods, could reason its way to the same conclusions.

It did—mechanically. It correctly identified Curvature and Diffraction as the consistent factors among problem wells. It noted that neither attribute alone was a perfect discriminator. It even offered to run logistic regression.

But it missed the interpretive leap. It hedged with phrases like “although there are exceptions” when in fact there were no exceptions to the conjunction rule. And it didn’t articulate the pragmatic framing: that the goal wasn’t to find the true cause, but to build a defensible decision rule under uncertainty.

That framing—the shift from epistemology to operations—required domain knowledge and judgment. The AI could apply Mill’s Methods. It couldn’t tell me why that application was useful here.

Drafting this post, I worked with a different AI—Claude—and found the collaboration more useful in a different way: not for solving the problem, but for reflection. Having to explain the context, the history, the why of my interest helped me articulate what I’d been carrying around in my head for years. Sometimes the value of a thinking partner isn’t in the answers, but in the questions that force you to be clearer.

Why this stuck with me

I’ll be honest: I kept thinking about this problem for years. It became part of a longer arc of engagement with Lee’s work—first the statistics tutorial, then the Wilrich case study, each building on the last.

When I interviewed for a geophysics position (Lee was retiring, and I was a candidate for his role), I mentioned this case study. I pulled out a pen and paper and wrote the entire seven-well table from memory. They seemed impressed—not because memorizing a table is hard, but because it signaled that I’d actually enjoyed thinking about it. That kind of retention only happens when curiosity is real.

I didn’t get the job. The other candidate had more operational experience, and that was the right call. But the process was energizing, and I’m sure that enthusiasm carried into my next opportunity, where I landed happily and stayed for over six years.

I tell this not to brag, but to make a point: intellectual play compounds. You don’t always see the payoff immediately. Sometimes you explore a problem just because it’s interesting—because someone like Lee writes “You can do it, too” and you decide to take him seriously—and it pays dividends in ways you didn’t expect.

The convergence

Three very different approaches—19th-century inductive logic, a simple neural network, and (later) an AI assistant—all pointed to the same answer: Curvature and Diffraction predict drilling problems in this dataset. The AVO attributes are noise, or worse, misleading.

When three methods converge, you can trust the signal. And you can make decisions accordingly.

That’s the real lesson here: rigorous reasoning under uncertainty isn’t about finding the One True Cause. It’s about building defensible heuristics, being honest about what you don’t know, and updating as new data comes in. Mill understood this in 1843. A neural network can learn it from seven wells. And sometimes, so can an AI—with a little help.

I hope you enjoyed this as much as I enjoyed putting it together.

The original case study was presented by Lee Hunt in his CSEG talk “Value thinking: from the classical to the hyper-modern.” The neural network analysis is in my Geoscience_ML_notebook_4. Lee documented the coal correlation issue in Hunt et al., “Precise 3D seismic steering and production rates in the Wilrich tight gas sands of West Central Alberta” (SEG Interpretation, May 2014), and later reflected on confirmation bias as an obstacle to recognizing such issues in “Useful Mistakes, Cognitive Biases and Seismic” (CSEG Recorder, April 2021). My thanks to Lee for the original inspiration, for confirming the geological context, and for sharing the original presentation materials.

  • Hunt, L., 2013, Many correlation coefficients, null hypotheses, and high value: CSEG Recorder, December 2013. Link
  • Hunt, L., S. Hadley, S. Reynolds, R. Gilbert, J. Rule, M. Kinzikeev, 2014, Precise 3D seismic steering and production rates in the Wilrich tight gas sands of West Central Alberta: SEG Interpretation, May 2014.
  • Hunt, L., 2021, Useful Mistakes, Cognitive Biases and Seismic: CSEG Recorder, April 2021.
  • My neural network analysis: Geoscience_ML_notebook_4
  • My earlier exploration of Lee’s production data: Data exploration in Python: distance correlation and variable clustering
  • ChatGPT + Wolfram session on Mill’s Methods: Gist

AI/HI Transparency Statement Modified from Brewin http://www.theguardian.com/books/2024/apr/04/why-i-wrote-an-ai-transparency-statement-for-my-book-and-think-other-authors-should-too

Has any text been generated using AI?Yes
Has any text been improved or corrected using HI?Yes

Additional context: This post emerged from a conversation with Claude AI (Anthropic). I provided the source materials (a ChatGPT + Wolfram session, a Jupyter notebook, personal history with the problem), direction, and editorial judgment throughout. Claude drafted the post based on these inputs and our discussion of structure, voice, and framing. I reviewed multiple draft, revised as needed, rewrote some key sections, and made all final decisions about what went to publication. The core analysis—Mill’s Methods, the neural network, the interpretation—was done by me years before this collaboration; the AI’s role was in helping articulate and structure that work for a blog audience.

ChatGPT as an essay-writing assistant – Part III

Featured

Posted on by
Reply

The Challenge of a Satisfying Conclusion

When I published Part II of this series back in February 2025, I had a plan for Part III. Show the prompts I used, analyze the time investment, evaluate the result against Part I’s GPT-3.5 baseline, maybe try one more iteration with even newer tools. Straightforward. Methodical.

But I never finished it. To be honest, I lost interest. Another marginally better AI-generated essay wasn’t going to cut it—not for me, and probably not for you readers either. Another iteration showing GPT-4.5 writes slightly better than GPT-4? That’s predictable, uninspiring… so I dropped it.

But the unfinished series sat there in the back of my mind. I wasn’t actively working on it, but I also couldn’t quite let it go. It created a kind of block—I found myself not writing about anything at all, partly because this felt incomplete, partly because my interests had genuinely shifted elsewhere.

Recently though, I came back to this question. Not because I wanted to complete the series for completeness sake, but because I wanted to understand what would actually make Part III worthwhile.

So I asked for help. I brainstormed with Claude (Anthropic’s AI) about what Part III should actually be about—what would make it worth writing and worth reading. And something clicked.

What Was the Question Really Asking For?

Looking back now, with decades between me and that moment in Professoressa Carbone’s classroom, I think I understand what she was asking for. She wasn’t looking for recitation of Plato’s philosophy mechanically applied to medieval warfare. She wanted to see if I could reason using philosophical frameworks in unfamiliar territory. Synthesis, not facts. Thinking, not performing memorization.

At 15, I wasn’t ready for that. I had volunteered for the oral examination thinking I could rely on prepared material about Plato’s recent lessons. Instead, she cut through my preparation with a single question that required genuine philosophical thinking: “What would Plato have thought about the Hundred Years’ War?”

It was a brilliant pedagogical move. It required understanding Plato’s ideas deeply enough to apply them to a completely different context—a context Plato never encountered, in a historical period he never knew. It required the kind of intellectual flexibility and reasoning that, honestly, I didn’t have yet.

The humiliation I felt wasn’t really about not knowing facts. It was about being exposed as someone trying to get by on memorization rather than understanding. And I think she knew it. She saw through my bluff.

So What Would Satisfy?

This brings me back to the problem of Part III. Showing that AI can now generate a more sophisticated-sounding essay than my 15-year-old self could produce doesn’t prove anything interesting. AI is very good at generating sophisticated-sounding content. That’s almost the problem.

What would actually satisfy—both as closure for this series and as something worth your time reading—is demonstrating the kind of reasoning Professoressa Carbone was asking for. Can I, now, with the benefit of intellectual maturity and AI assistance, actually think through what Plato might have thought about prolonged warfare between nations? Not just string together plausible-sounding paragraphs with proper citations, but engage in genuine philosophical reasoning?

What Would That Actually Look Like?

If I were to actually write that essay—the one demonstrating real philosophical reasoning rather than AI-generated content—what would it need?

Looking back at the GPT-4 essay from Part II, it has proper citations and coherent structure, but it’s superficial. It lists Platonic concepts (philosopher-kings, guardians, ideal states) and applies them mechanically to medieval warfare. That’s exactly the kind of recitation Professoressa Carbone was testing me against.

Real reasoning would require:

  • Connecting Plato’s specific ideas to specific events or decisions during the Hundred Years’ War—not just general principles applied generally
  • Exploring how Plato’s concepts might actually illuminate something about prolonged conflict between nations that we wouldn’t see otherwise
  • Considering contemporary interpretations or modern applications—what do we learn about conflict, governance, or political philosophy from this exercise?
  • Drawing genuine insights about both Plato and warfare, not just restating both

That’s the essay I’d want to write someday. Not as an academic exercise, but as personal closure—proving to myself I can do the kind of thinking she was asking for.

Closure for Now

But that’s not this post. This post is about giving you, the readers, closure on this series. About acknowledging honestly what I learned about AI as a writing assistant, and why simple iteration wasn’t the answer.

Here’s what I’ve learned:

AI is excellent at generating plausible content. GPT-4 produced an essay that looks credible—proper structure, citations, coherent arguments. For many purposes, that’s enough.

But AI doesn’t reason, it recognizes patterns. The essay from Part II strings together familiar ideas in familiar ways. It’s sophisticated pattern matching, not thinking. It can’t do what Professoressa Carbone was asking for: genuine synthesis that produces new insight.

The real value of AI as a writing assistant isn’t in replacing thinking—it’s in supporting it. AI can help with research, organization, articulation. It can reduce cognitive load so you can focus on the hard part: the actual reasoning. But you still have to do the reasoning.

Writing with AI requires clarity about what you’re trying to accomplish. If you want content generation, AI does that well. If you want thinking support, you need to know what thinking you’re trying to do. The tool can’t figure that out for you.

This series started with a simple question: can AI help me write an essay? The answer turned out to be more nuanced than I expected. It depends entirely on what kind of essay, and what role you want AI to play. For the essay I’d need to write to truly answer Professoressa Carbone’s question—the one that demonstrates reasoning rather than recitation—AI could help, but it couldn’t do the essential work.

Maybe someday I’ll write that essay. For now, I’m moving on to other projects where I’m excited about what AI can do: document extraction in geoscience, agentic workflows, problems where AI’s strengths align better with what I’m trying to accomplish.

Thank you for following this journey with me. Even if it didn’t end where I originally planned, I learned something worth sharing.

A Final Thought: Rigor Without Brutality

I started this series partly because of concerns about AI in education—concerns rooted in my own experience.

ChatGPT has educators calling for more in-class writing and oral examinations. I agree we need assessment that can’t be faked by AI. But I’m deeply opposed to the brutality that often came with those older systems.

Here’s the thing: the brutality was never necessary for the educational value. Professoressa Carbone’s question was pedagogically brilliant. The public humiliation didn’t make it more effective; it just made it traumatic.

We need assessment methods that demand genuine reasoning, in environments that support both students and teachers. It’s possible to have rigorous evaluation without breaking people in the process.

AI forces us to confront what we actually value in education: not the appearance of learning, but the development of genuine understanding and reasoning. The question is whether we can build systems that nurture that without the cruelty.

AI/HI Transparency Statement Modified from Brewin http://www.theguardian.com/books/2024/apr/04/why-i-wrote-an-ai-transparency-statement-for-my-book-and-think-other-authors-should-too

Has any text been generated using AI?Yes
Has any text been improved or corrected using HI?Yes

Additional context: This post was collaboratively written through an iterative conversation with Claude (Anthropic). The human author provided the direction, constraints, personal context, and decisions about what to include/exclude. The AI assistant drafted text, which was then reviewed and revised based on feedback. Sections were rewritten multiple times to match the author’s voice and intentions. The final editorial decisions, including what content made it to publication, were made by the human author.

ChatGPT as an essay-writing assistant – Part II

Posted on by
Reply

The blog post below was produced entirely by GPT-4.0, following a series of iterative prompts I provided, from the Introduction to the References and Footnotes, included. Please refer to my AI (Artificial Intelligence) and HI (Human Intelligence) Table in the last section. In the next post, I will include the full listing of my prompts, text evaluation, and time investment versus return analysis. I will also attempt prompting for some further improvements beyond this initial analysis.

Plato’s Perspective on the Hundred Years’ War

Introduction

The Hundred Years’ War, a prolonged and devastating conflict between England and France (1337-1453), was a historical event unknown to the ancient Greek philosopher Plato. Nonetheless, by examining his philosophical ideas, primarily in “The Republic” and “Laws,” we may speculate on his potential views regarding such a conflict.

The Philosopher King and the Ideal State

The Philosopher-King and the Ideal State In his magnum opus, “The Republic,” Plato presents the concept of a philosopher-king, an individual embodying wisdom and justice, who governs the ideal state (Plato, Republic, Stephanus 473c-d, Book V)1. This enlightened ruler, possessing both knowledge and moral integrity, would guide society toward harmony and justice.

War as Necessary Evil

In the “Laws,” Plato acknowledges that war might be a necessary recourse, particularly when defending the state against injustice (Plato, Laws, Stephanus 625c, Book I)2. However, he also emphasizes the importance of striving for peace and pursuing diplomacy whenever possible.

Striving for Peace

Plato contends that an ideal state should always seek to cultivate peace with other nations, fostering a spirit of friendship and cooperation (Plato, Republic, Stephanus 470a-b, Book V)3. This suggests that, had he been aware of the Hundred Years’ War, Plato would likely have advocated for diplomatic efforts to resolve the conflict.

The Guardians as Defenders of the State

The Guardians: Defenders of the State In “The Republic,” Plato introduces the guardians, a class of citizens specifically trained to defend the state and uphold its values (Plato, Republic, Stephanus 414a-b, Book III)4. These individuals, educated in both martial and intellectual pursuits, would protect the state from external threats, such as those posed during the Hundred Years’ War.

Governance in Times of War

Though Plato acknowledges the potential necessity of war, he cautions against allowing it to dominate the state’s agenda (Plato, Republic, Stephanus 468a-b, Book V)5. Instead, he argues that governance should remain focused on promoting the well-being of citizens and maintaining social harmony.

Fostering Social Harmony and Virtue

Plato maintains that an ideal state should strive to cultivate virtue among its citizens, promoting wisdom, courage, moderation, and justice (Plato, Republic, Stephanus 427e-428a, Book IV)6. In this context, the prolonged conflict of the Hundred Years’ War might be seen as a distraction from the state’s primary goal of fostering a harmonious and virtuous society.

Conclusion

Though Plato lived centuries before the Hundred Years’ War, his philosophical ideas can provide valuable insights into how he might have viewed such a conflict. By examining his writings, particularly “The Republic” and “Laws,” we can speculate that Plato would have recognized the potential necessity of war in certain circumstances but would have strongly advocated for peace, diplomacy, and the pursuit of justice and virtue in governance.

References

TitleAuthorDate WrittenChicago citation
The RepublicPlato380 BCEPlato. The Republic. Translated by G.M.A. Grube. Indianapolis: Hackett Publishing Company, 1992.
LawsPlato348 BCEPlato. Laws. Translated by A.E. Taylor. London: Dent, 1934.

Footnotes

  1. Plato, Republic, 473c-d. Perseus Digital Library | The Internet Encyclopedia of Philosophy 
  2. Plato, Laws, 628a-630d. Perseus Digital Library | The Internet Encyclopedia of Philosophy 
  3. Plato, Republic, 470a-471c. Perseus Digital Library | The Internet Encyclopedia of Philosophy 
  4. Plato, Republic, 433b-c. Perseus Digital Library | The Internet Encyclopedia of Philosophy 
  5. Plato, Republic, 414b-415d. Perseus Digital Library | The Internet Encyclopedia of Philosophy 
  6. Plato, Laws, 348, 663a-b. Perseus Digital Library | The Internet Encyclopedia of Philosophy 

AI (Artificial Intelligence) and HI (Human Intelligence) Statement

Modified from Brewin http://www.theguardian.com/books/2024/apr/04/why-i-wrote-an-ai-transparency-statement-for-my-book-and-think-other-authors-should-too

Has any text been generated using AI?Yes
Has any text been improved or corrected using HI?No
Has any methods of analysis been suggested using HI?Yes
Has any methods of analysis been suggested using AI?No
Do any analyses utilize AI technologies, such as Large Language Models, for tasks like analyzing, summarizing, or retrieving information from data?Yes