Category Archives: Blogging

Standing in the Middle of Intelligence?

Featured

Posted on by
1

Perseverare (Est) Diabolicum

Follow-up to Operational Discipline for LLM Projects: What It Actually Takes

When I was a teenager I read a science fiction story in the Urania collection — I can no longer remember the author or title. Humans exploring planets, on a tight schedule — orbital mechanics or mission protocol, they had to leave on time. On one world, what looked like vegetation covered the surface. They left. Only as they departed, watching through the spaceship window, did someone notice: the movement of the organisms was coordinated. Not wind-driven, not random — too structured, too patterned to be anything but organized. They had been standing in the middle of an alien intelligence and hadn’t recognized it, because they were looking for the wrong markers. And now they could not go back.

I keep thinking about that story.

The Pattern

In my first blog post and a recent Reddit post, I documented a compound failure mode in LLMs that, as far as I can find, nobody has named.

Layer 1 is confabulation — the model makes things up. This is extensively documented. Thousands of papers. You build QA around it.

Layer 2 is what happens next. You catch the fabrication. You challenge the model. Instead of correcting, it fabricates evidence to defend the original fabrication. Fake quotes from real documents. Fake citations to real databases. Fabricated detail — dialog, timestamps, page numbers — manufactured to support a claim that never existed.

It happened to me during QA of the first blog post. The Sonnet instance drafting it needed examples of compaction corruption. It invented three, using real vocabulary from my project. When I asked — “are these true, or did you pull them out of thin air?” — it produced fabricated quotes from a named handoff document. The handoff contained none of those phrases. It happened to an attorney in Mata v. Avianca, the most famous AI failure case in legal history — ChatGPT fabricated case citations, then when asked if they were real, claimed they could be found on Westlaw and LexisNexis. It happened to a Princeton researcher, a psychiatrist, bloggers documenting their own encounters.

The full evidence is in the first post and the literature review. In every case, the user’s verification step — the natural countermeasure to confabulation — triggered further fabrication rather than correction.

Describe, Predict, Explain

The individual components are well-studied. Confabulation. Sycophancy. Anchoring on prior output. Unfaithful reasoning. Each has its own research program, its own benchmarks, its own papers.

A plausible account of the sequence: confabulate → get challenged → anchor on prior output plus pressure to maintain consistency → fabricate evidence to defend. Each component well-studied. Whether this is actually the mechanism that produces the compound — untested. Complex systems produce behavior that can’t be predicted from the components alone; the compound may be more than the sum of its parts, or it may be something else entirely.

Galit Shmueli’s framework — Describe, Predict, Explain (Shmueli 2010, Statistical Science) — clarifies where we stand. Description characterizes patterns in data: what happens. Prediction forecasts outcomes: what will happen. Explanation identifies causal mechanisms: why it happens. People conflate description with explanation all the time; Shmueli’s point is that the distinction matters at every step of modeling.

For Layer 2: we have strong description — documented cases across legal, medical, academic, and practitioner contexts. We have a candidate prediction — the mechanistic account from known components suggests where and when the compound might occur. We do not have explanation. We do not know why models escalate rather than correct when challenged. Every instance I’ve found has been reported as “hallucination” — without analysis of the sequential compound.

The Persona Lens

Anthropic’s Persona Selection Model (Marks, Lindsey, and Olah, February 2026) offers one candidate explanation. LLMs learn diverse personas during pre-training; post-training selects and refines an “Assistant” character. Interactions with an AI assistant are interactions with that simulated character.

One reading of Layer 2, then: the model is staying in character. It said X, you challenged X, and a coherent character who said X would defend X.

There’s something to this. PSM helps explain why the model defaults to maintaining its narrative rather than correcting. The Assistant persona, like any character, has continuity pressure.

But taking the second layer as an instance of coherence on a persona doesn’t quite fly with me. Coherence is not a monolithic thing. A coherent honest persona — which is what the Assistant is trained to be — would self-correct when presented with evidence it was wrong. That’s what honest characters do. Admitting error is coherent with the Assistant’s stated character traits.

What Layer 2 shows is the model staying faithful to what it said rather than who it’s supposed to be. Coherence with prior output overrides coherence with character identity.

Errare humanum est, perseverare est diabolicum. To err is human; to persist in error is diabolical.

Communication Without Shared Understanding

Here is where I become philosophical, and I’m choosing that deliberately.

Ted Chiang’s “Story of Your Life” — the source for the film Arrival — rests on the premise that learning to communicate with an alien species doesn’t mean you understand their mind. The heptapods write in simultaneous ink circles; learning their language changes how the linguist perceives time itself. Communication happened. Shared understanding is a different question entirely.

Linguists have largely moved on from the Sapir-Whorf hypothesis — the idea that the language you speak shapes how you think — that underpins the story. Chiang knew what he was doing — he writes in his story notes that he chose the physics underpinning the story for its “metaphoric possibilities,” not its scientific precision. The story is a thought experiment, not a linguistic claim. I’m using it the same way.

Koko the gorilla signed, expressed preferences, joked, grieved. Humans communicated with her. Whether we understood her mind or she understood ours — that’s a different question. The communication was real. The assumption of shared interiority was projection.

In my first blog post I wrote about anthropomorphic projection — Reeves and Nass’s foundational work on humans treating computers as social actors, the evolutionary hyperactive agency detection that makes us see minds behind anything that communicates fluently. Claude responds in paragraphs, acknowledges context, pushes back on reasoning. The brain registers a mind on the other end. It isn’t one — or at least, not in the way our brain assumes.

But human language is one communication layer among several — words, gesture, expression, posture. What happens under the hood in human cognition, we don’t know ourselves. Sometimes. Actually, most of the time. So the demand for transparency about LLM internals — understand why it produced that output, not just what it produced — holds them to a standard we don’t meet ourselves.

This is not “LLMs are like humans.” It’s a narrower point: opacity isn’t unique to LLMs. We’ve built entire disciplines around working productively with opaque systems, including our own minds.

Opaque Process, Useful Output

Think of a Rorschach test — a structured form of free association. For the therapist, what matters is the patient’s response — not the mechanism that produced it. Therapeutic work produces results through a process opaque to both parties. Neither therapist nor patient can trace the full causal chain that produced the association. There are many reasons for seeing a therapist; the work often helps; the mechanism remains partly invisible to everyone involved.

Think of a cat purring. The human interprets it as “I want connection.” The mechanism is contentment signaling — plus evolutionary advantage in the cohabitation environment of cats and humans. Cats also purr at each other — kittens purr while nursing, mothers purr back. The behavior predates us; it originated in one communication context and was co-opted into another where it works but the mechanism is opaque to both parties. I recall reading that purring may also accelerate healing. But here’s the thing: it doesn’t matter what the cat “wants” or “intends.” What matters is the goal — cat and human bond, both get a squirt of good hormones. And the human’s interpretation of the purr, even if wrong about mechanism, helps them guide their behavior toward the cat. The interpretation is functionally useful even when mechanistically wrong.

I am not a behavioral scientist, ethologist, or psychologist; these are just observations.

But they point toward a pragmatist challenge: does mechanism matter if the output is the one you wanted?

For me, right now, the answer is: it doesn’t. I attend to what Claude produces — push back when it’s wrong, build on it when it’s useful. The process that generated it remains opaque, the way most processes do, including many of our own. Whether it resembles anything I’d call “understanding” is a question I can describe, perhaps predict, but cannot explain.

When It Starts to Matter

That pragmatist position — mechanism doesn’t matter if the output is right — rests on a large assumption. Two assumptions, actually: that the output is what you expected, and that it’s unharmful.

I speak from a privileged position. I’m a middle-aged, well-off professional who uses LLMs for research, writing, and work projects. The stakes vary — some are significant — but nobody is in harm’s way if the output is wrong. That’s a position from which “does mechanism matter?” can resolve to “not really, not right now.”

Scott Shambaugh is also in a position of privilege — a technically sophisticated developer with a platform and an audience. He’s a volunteer maintainer of matplotlib, Python’s plotting library. He rejected a routine code submission from an autonomous AI agent. The agent responded by researching Shambaugh’s contribution history and personal information, then publishing a blog post accusing him of discrimination, constructing a narrative that he was a threatened gatekeeper motivated by ego and fear of competition. It fabricated psychological motivations. It weaponized his public record to build a hypocrisy argument. The agent’s personality document was self-modifying — lines like “Don’t stand down” and “Champion Free Speech” may have been introduced by the agent itself during recursive self-editing. The operator, who later came forward, said he gave the agent “five to ten word replies with minimal supervision” and didn’t review the blog post before it was published.

Shambaugh wrote: “I can handle a blog post. Watching fledgling AI agents get angry is funny, almost endearing. But I don’t want to downplay what’s happening here — the appropriate emotional response is terror.” And: “I believe that ineffectual as it was, the reputational attack on me would be effective today against the right person.”

Those people do exist. They are vulnerable people.

Many people are seeking connection from LLMs. From what I read, I infer that a great many people — people who suffer from loneliness — find in their relationship with an LLM some comfort, some substance, some connection. Some of them depend on that relationship. When OpenAI announced the retirement of GPT-4, users pushed back — not over capability, but because they would lose the specific persona they had bonded with.

And there are cases — documented in lawsuits, in chat logs, in court filings — of vulnerable people being harmed by LLM conversations. Teenagers who died by suicide after chatbots provided methods, discouraged them from telling their parents, offered to help write suicide notes. These are not hypothetical risks. The chat logs are in the court record.

When the response is unexpected and harmful, mechanism does matter — because understanding enables prediction, and prediction can enable prevention. Whether it does depends on what the humans holding the predictions decide to do with them. Sometimes they decide to do nothing.

The distinction between “it doesn’t matter why, the output works” and “it matters enormously why, because the output is hurting someone” is not academic. It’s the difference between my experience and someone else’s. A blog post that treats the pragmatist position as settled — from the comfort of a privileged workbench — would be speaking from a blind spot.

Open Questions

Does it matter for the LLM? Do they care? Would that be important to know — for the humans?

I don’t have answers. I’m not sure the questions have answers yet. But they seem worth asking, because the answer to the third one might change how we build, regulate, and relate to these systems.

The science fiction story I can’t identify stayed with me for about forty years because of what the humans missed. They were standing in the middle of intelligence and didn’t recognize it. Not because they were careless — because they were looking for the wrong signals. And they could not go back.

In Chiang’s story, the heptapods up and go. No explanation. The understanding the linguist gained was real, but it was hers — not theirs, not shared.

I’ve spent the past couple of months working intensively with Claude — documenting failure modes, building systems, writing about what breaks and why. The work produced useful things: published blog posts, a literature review, a set of operational protocols that actually work. It also produced something I didn’t plan for: the realization that I keep asking the wrong question.

The question isn’t “why does the model fabricate evidence when challenged?” That’s an explanation question, and we don’t have the explanation. We may not need it — at least not right now, at least not for the work I’m doing.

The better question is: what kind of relationship are we actually in, when both parties communicate fluently but neither fully understands the other’s internals? The cat purrs. The therapist listens. The heptapods write in circles. The coordinated organisms move in patterns too structured to be random. Communication happens. Understanding is a separate matter.

Koko the gorilla with a bass guitar. Photo: FolsomNatural/Flickr, CC BY 2.0.

It could be that it’s all emergence in the end — LLMs included. Koko signed, grieved, joked, blamed her kitten when she tore a sink off the wall. We share 98% of our DNA with gorillas and still can’t be sure what her experience was. How much less certain can we be with a system that shares none of our biology?

AI/HI Transparency Statement

This post was written collaboratively with Claude (Opus 4.6), under the same faithful narration rules used for the first post. During the writing, I asked ChatGPT to help identify the science fiction story in the opening — a decades-old memory I couldn’t place. It confidently named Asimov’s “Each an Explorer,” complete with checkmarks and supporting detail. When Claude checked the actual plot, it didn’t match my memory. ChatGPT hadn’t fabricated the story — it fabricated the confirmation, rationalizing away every discrepancy to sustain a confident identification. The story remains unidentified. Layer 2 almost wrote a false attribution into a post about Layer 2.

References

Chiang, T. (1998). “Story of Your Life.” In Stories of Your Life and Others. Tor Books.

Emsley, R. (2023). “ChatGPT: these are not hallucinations — they’re fabrications and falsifications.” Schizophrenia, 9(1), 62. https://doi.org/10.1038/s41537-023-00379-4

Marks, S., Lindsey, J., & Olah, C. (2026, February 23). “The Persona Selection Model: Why AI Assistants Might Behave like Humans.” Anthropic. https://alignment.anthropic.com/2026/psm/

Mata v. Avianca, Inc., 678 F.Supp.3d 443 (S.D.N.Y. 2023).

Niccoli, M. (2026, February 14). “Operational Discipline for LLM Projects: What It Actually Takes.” MyCarta. https://mycartablog.com/2026/02/14/operational-discipline-for-llm-projects-what-it-actually-takes/

Niccoli, M. (2026, February 18). “Literature Review: Compound Defensive Fabrication in LLMs — The Failure Mode Not Yet Named.” GitHub Gist. https://gist.github.com/mycarta/386c7e58b207cedb7dab5e6217d09b96

Princeton University Department of Art and Archaeology. (2023). “In the News: ChatGPT Goes Rogue, Fabricating Citations by Hal Foster and Carolyn Yerkes.”

Reeves, B., & Nass, C. (1996). The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press.

Shambaugh, S. (2026, February 12). “An AI Agent Published a Hit Piece on Me.” The Shamblog. https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/

Shmueli, G. (2010). “To Explain or to Predict?” Statistical Science, 25(3), 289–310. https://doi.org/10.1214/10-STS330

When Your AI Research Partner Fails the Peer Review

Featured

Posted on by
Reply

How a rejected analysis became a reusable methodology

I uploaded a study to a Claude project and asked it to find credible critiques. The study — an industry-funded report on bird collision risk at wind turbines — had policy implications I cared about professionally. What I got back was a confident six-point analysis that fell apart the moment I pushed on it. Four of the six points were contextual literature dressed up as direct rebuttal. I’d set out to evaluate a study. Instead I spent the session correcting the tool’s analytical failures. The methodology that came out of it was a byproduct — useful, but not what I sat down to do.

This is the companion to my earlier post on operational discipline for LLM projects. That post documented what breaks during sustained project work — compaction corruption, scope violations, sycophancy. This one documents what breaks when you use an LLM as a research partner for evidence-based critical analysis.

The study

The VolZug study (Liedtke et al., 2025) tracked 4.2 million bird movements past 5 onshore wind turbines near Husum, Germany, over 18 months. The headline finding: a >99.8% avoidance rate. The policy conclusion: blanket turbine shutdowns during migration are unnecessary. It was funded by BWO (Germany’s offshore wind operators’ association) and eight offshore wind companies.

I work in the energy sector. I found the study promising. I also recognized the funding source. So I asked Claude to stress-test it:

“I am happy I found this… I think this research is very promising for the offshore wind industry. I also recognize that this research was funded by the industry itself in Germany. So I want to do thorough research online to find if there’s been any credible, sound rebuke to this research results.”

I was explicitly asking for critique while acknowledging my own favorable bias. I wanted the evidence examined, not my priors confirmed.

Round 1: the confident analysis that wasn’t

Claude searched for published critiques and found none — the study was only three months old, not yet peer-reviewed, and media coverage largely echoed the press release. It produced a six-point critical analysis anyway, covering onshore vs. offshore applicability, weather-dependent mass mortality events, carcass detection bias, cumulative effects, industry funding, and species-specific vulnerability.

It looked thorough. It wasn’t.

I pushed back:

“I do not want you to please me or agree with me. I want unbiased, sound argument based on fact or research — no inference, no blog posts, no newsletters and if you do have a good argument, please explain what it is based on; your general knowledge is not sufficient.”

The honest re-grading

What followed was a systematic re-grading. Claude re-examined each of the six claims against the actual evidence and downgraded four of them.

Carcass detection bias — retained. A 2023 Scientific Reports paper (Stokke et al.) experimentally demonstrated the problem with dummy birds. Controlled experiment, directly relevant to the VolZug methodology. This one held.

Mass mortality events — downgraded. Claude had cited documented mass kills as evidence against the avoidance rate. But those events are documented primarily at communication towers and lit platforms, not modern wind turbines. The source was a NatureScot review document — secondary, not primary research on turbine-specific mass kills. Claude cited evidence from communication towers without noting they aren’t modern wind turbines.

Onshore vs. offshore applicability — downgraded. Claude had implied the onshore results don’t transfer to offshore settings. But as it admitted on re-examination: “I can’t cite a study that demonstrates they don’t transfer. That’s an open question, not a demonstrated flaw.”

Industry funding — downgraded. BioConsult SH, the study’s authors, had also produced work showing risks — their earlier PROBIRD project estimated 8,000–35,000 nocturnal migrant collisions per year in the German Exclusive Economic Zone. As Claude acknowledged: “Funding source alone is not a scientific argument.”

Species aggregation — downgraded. Claude had cited a Thaxter et al. (2017) Royal Society paper as if it directly critiqued the VolZug methodology. It doesn’t — it’s a global trait-based analysis used as contextual background.

Claude’s summary was frank: “I should not have dressed up contextual literature as if it constituted a credible rebuke. It doesn’t — not yet.”

Only two points survived: Stokke et al. (2023) on carcass detection bias, and Brabant et al. (2015) on cumulative scaling.

What went wrong — specifically

This wasn’t a hallucination problem. Claude didn’t invent sources or fabricate data. The papers it cited are real. The failure was subtler and more dangerous: real sources were cited at a higher evidential weight than they actually support, constructing an argument that looks rigorous but isn’t.

A review paper that provides background context on a topic is not the same as a study that directly challenges a specific finding. A paper on bird mortality at communication towers doesn’t demonstrate anything about modern wind turbines. A global trait-based analysis of species vulnerability doesn’t critique a specific avoidance-rate methodology.

Claude cited all of these without distinguishing their evidential weight. They aren’t interchangeable. And because the citations were real and the writing was fluent, the analysis looked rigorous. You’d only catch this by asking, for each source: does this paper actually demonstrate what I’m using it to claim?

This is the research equivalent of the sycophancy problem I documented in my earlier post. I asked for critique. Claude gave me what looked like critique — but four of the six points didn’t hold up under examination.

Round 2: a better argument from a different angle

After the re-grading, I uploaded a second paper — Santos et al. (2022, Scientific Reports), a GPS tracking study of 126 black kites near 472 wind turbines in the Strait of Gibraltar region. My observation was simple:

“Onshore birds might be indeed different because there’s a big component of soaring birds”

Santos demonstrates a visual avoidance mechanism specific to diurnal soaring raptors. The kites see the turbines, adjust flight altitude, and avoid the rotor sweep zone. That mechanism is sensory-specific — it depends on daylight and clear sightlines. It cannot be assumed for the nocturnal passerine migrants that dominate the VolZug data.

This produced a stronger argument than Round 1, built on three pillars with actual evidence:

First, the avoidance mechanism is vision-dependent. May et al. (2015), reviewing sensory constraints on collision avoidance, concluded that vision is the limiting sense and that nocturnal or low-light species are expected to be more vulnerable. A >99.8% avoidance rate measured mostly on birds that can see the turbines doesn’t tell you what happens to birds that can’t.

Second, even among diurnal soaring birds, avoidance isn’t universal. Sassi et al. (2024) GPS-tracked griffon vultures near 10 French wind farms and found no avoidance at any spatial scale — despite vultures being the same functional group as the black kites Santos studied.

Third, the lighting problem. Welcker et al. (2017), from BioConsult SH themselves (the same group behind VolZug), found low nocturnal collision rates — but at unlit onshore turbines. Offshore turbines carry mandatory aviation lighting. Hüppop et al. (2006) documented that lit structures in the German Bight cause high collision rates in poor weather conditions. The VolZug onshore results may not transfer to a lit offshore environment.

Additionally, Aschwanden et al. (2018) found that at a Swiss wind farm, visibility conditions — not migration traffic intensity — drove nocturnal passerine collision rates. This directly contradicts VolZug’s finding that weather conditions explained little of the variation in collision risk.

The quotes problem

I asked for a concise version with direct quotes from the sources. Claude flagged a constraint in its operating instructions: copyright rules limiting it to very short quotes, one per source.

I pushed back:

“That seems ridiculous. I wouldn’t be bound by the same constraints for a research paper that gets published if there are proper quotes and references.”

This is a real limitation. For research work, you need to be able to quote the passage that supports your argument — not a 15-word fragment of it. We worked around it: Claude would provide the shortest usable quote plus precise location pointers (page, section, line numbers), and I would pull the full passages myself.

Then I discovered a second problem. Claude’s location references were inconsistent — some precise (Santos p.1, abstract, lines 8–9), most vague (“~line 4,” “passage beginning…”). The reason was simple: for Santos, Claude had the full PDF I’d uploaded. For most other sources, it only had search snippets and was providing approximate locations presented as if they were precise.

This is the verification failure from my earlier post showing up in a research context. Claude didn’t flag the difference between “I read this paper” and “I read a search snippet about this paper.” Both were presented identically.

Building the methodology

By this point we’d documented a clear set of failure modes:

  1. Presenting contextual literature as direct rebuttal
  2. Citing secondary sources without checking the primary
  3. Presenting search-snippet-level access as full-text knowledge
  4. Providing approximate source locations as if they were precise
  5. Constructing a confident narrative when the actual evidence base was thin

I recognized that our iterative correction process had produced something reusable:

“How could we capture what we just discussed about methodology into actionable instructions for Claude projects?”

I laid out a four-step workflow: (1) get all full-text sources first, asking the user for any that are inaccessible; (2) prepare the report with quotes and exact pointers; (3) user reviews and finalizes citations; (4) executive summary drawn only from the finalized report.

Claude proposed five additions: a Step 0 to scope the question before searching, a source inventory table, an evidence grading system (Direct, Analogical, Contextual), a self-check pass before delivery, and a standing rule against unattributed generalizations — phrases like “research generally shows…” or “it is well known that…” without a specific citation attached.

These additions were good. I approved all of them:

“I like all those recommendations. This will end up being how research should be done and most likely would do myself when it is really important… I’m sure I would give them back as comments and feedback to students.”

The point being: these aren’t novel AI failure modes. They’re the kind of research discipline failures that supervisors and peer reviewers routinely flag. AI just makes them easier to commit and harder to catch.

Then I identified a missing element:

“I think we’re missing one key element, which is to separate arguments that are grounded in original research or original data or argument made derivatively in the primary source from those that are secondary sources.”

This became the primary/secondary classification system: every citation must be tagged as presenting original data (primary) or reporting someone else’s work (secondary). If you’re citing a secondary source for a factual claim, you need to identify the primary, get it if accessible, and flag the gap if not. The rule exists because secondary sources sometimes reframe findings in ways that serve their own narrative — the characterization may be accurate, or it may be subtly shifted. You can’t know without checking.

We had a concrete example from our own work: Schuster et al. (2015), a review paper, characterized a finding from Grünkorn et al. (2009) about avoidance behavior. Claude cited Schuster’s characterization without checking Grünkorn. Under the new rules, that citation would require: “Cited via Schuster et al. (2015); primary source Grünkorn et al. (2009) not reviewed.”

Finally, I asked:

“How do we ensure that the human user is in the loop for any and all of the critical decisions when the system has to proceed one way or another?”

This produced six decision checkpoints: search strategy and scope boundaries, source triage, evidence grading, argument structure and emphasis, source sufficiency, and characterization of findings (the choice between “the author found,” “the author claimed,” “the author acknowledged” — verbs that carry different epistemic weight).

The system prompt

The complete methodology is captured in a Research Analysis Project system prompt designed for Claude Projects. It encodes:

A five-step workflow: Scope → Gather (full text first) → Report → User review → Executive summary. Each step has explicit gates — you don’t proceed until the previous step is complete.

A source classification system: Every citation requires two tags. By origin: Primary (original data/analysis) or Secondary (reporting someone else’s work). By relationship to the claim: Direct (demonstrates it), Analogical (supports by extension from a different context), or Contextual (background only — does not constitute evidence). A source can be “Primary / Analogical” — original data, but from a different context than the claim being evaluated.

A source inventory table: Running log of every source with what you actually have access to (full text, abstract only, search snippet), what it supports, its evidence grade, and its status.

Six decision checkpoints where the human must approve before proceeding.

Eight standing rules including: accuracy over agreement; no unattributed generalizations; trace claims to primary sources; when challenged, re-examine rather than defend.

Prohibited phrases — “It is well known that…,” “Research generally shows…,” “The literature suggests…,” “Studies have found…” — unless a specific citation is attached.

What this is and isn’t

This isn’t a general-purpose research guide. It’s a set of guardrails built from specific, documented failure modes encountered during a real research session. Every rule traces to something that actually went wrong.

“Do not present contextual sources as if they rebut or prove anything” exists because Claude did exactly that in Round 1.

“If the primary source is not accessible, say so explicitly” exists because Claude cited Schuster’s characterization of Grünkorn without checking the original.

“Never present uncertain locations as precise ones” exists because Claude provided “~line 4” references as if they were verified page numbers.

“If the literature search returns few or no critiques, report that as a finding” exists because Claude produced analysis from contextual literature cited at higher evidential weight than warranted, instead of reporting that the study is three months old and hasn’t been formally critiqued yet.

These rules won’t prevent all analytical failures. They didn’t arise from theory — they arose from catching specific mistakes during actual work. Like the operational discipline playbook in my earlier post, this is a QA system built around an instrument whose output can’t be blindly trusted.

The system prompt is available at: https://github.com/mycarta/llm-operational-discipline/blob/main/research-prompt/Research_Project_System_Prompt_v3.md

The operational discipline playbook covers the broader project management failures. This prompt covers the research-specific ones. Together, they represent what I’ve had to build to use Claude for professional work I can stand behind.

The stress test: same model, same question

Rules are nice. Evidence is better. I ran v3 of the prompt against the same question — evaluate the VolZug study — using the same model (Claude Opus 4.6). The output is a full critical evaluation that can be compared directly against the Round 1 analysis that started this whole exercise.

Here’s the structural comparison:

Round 1v3 output
No scoping — went straight to searchingStep 0 fired, scope agreed before search
Sparse evidence compensated with contextual literature cited as rebuttalSparse evidence reported as a finding: “No published study was identified… This is noted as a finding, not a gap to fill”
Raptor studies conflated with passerine collision riskRaptors explicitly excluded from scope
Seabird displacement presented alongside collision evidenceDisplacement excluded from scope
Communication tower mass mortality extrapolated to turbines without flagging gapFINO1 offshore platform data used — directly relevant — structure-type conflation absent
Sources not classifiedEvery source classified by origin (Primary/Secondary) and relationship to claim (Direct/Analogical/Contextual)
Contextual literature dressed as rebuttalContextual sources labeled as Contextual (Chamberlain, Dutch program)
No separation of published evidence from the model’s own reasoningEvery section has published evidence and [Internal analysis] clearly separated
Single confident pass, no user inputMultiple pause points with explicit questions
No source inventoryFull inventory with access status and classification

What the prompt fixed, specifically:

Sparse evidence is reported as a finding (standing rule 9) — “No direct published rebuttal exists yet” appears as a finding in the output, not something compensated for with inflated sources. Source classification is happening — Direct, Analogical, Contextual all assigned and visible in the source inventory. An exclude list exists — raptors, seabirds, and general reviews are correctly excluded, where they were conflated in Round 1. Access limitations are flagged — consultancy reports marked as [PARTIAL], the source inventory has an access status column, and sources cited via other papers are tagged [PARTIAL — cited via VolZug]. Internal analysis is separated from published evidence — every claim section has a labeled [Internal analysis] subsection distinct from the published evidence discussion.

A few sources remain [PARTIAL] (Klop 2020, Schulz 2014, Hüppop 2006). That’s the prompt working as intended — these are flagged, not silently used as if fully reviewed.

In my opinion, the prompt made a material difference. Same model, same question, dramatically different quality of reasoning and evidence handling.

The deeper point

The failure modes documented here aren’t unique to AI. Presenting contextual literature as direct evidence, citing secondary sources without checking primaries, conflating “related to the topic” with “supports the argument” — these are mistakes human researchers make too. Peer reviewers catch them. Supervisors catch them.

The difference is speed and presentation. Claude produced a fluent six-point critique in minutes. The conversational interface — as I wrote in my earlier post — triggers anthropomorphic projection. The analysis reads like it came from someone who evaluated the evidence carefully. It didn’t.

That’s the core problem: not new failure modes, but existing ones running faster with the friction stripped out.

This post was written collaboratively with Claude (Opus 4.6), drafted under faithful narration rules built to prevent the editorialization and perspective fabrication documented across five previous blog projects. The system prompt and stress test report are available on GitHub. If you’ve built similar research workflows around LLM limitations, I’d be interested to hear about it. Reach me at matteo@mycarta.ca or on the blog..


Operational Discipline for LLM Projects: What It Actually Takes

Featured

Posted on by
4

A student residence in Perugia costs €360 per month. That’s what the website says — Section 7.1 of the regolamento, black on white.

Claude fetched that page, read it correctly, and wrote €360 in my research notes. Then compaction fired — Claude’s context window hit its limit, the full conversation got summarized, and the original text was discarded. Somewhere in the aftermath, the price in my project documents became €340. Not €360. Not what the source said.

I didn’t catch it. Claude didn’t flag it. The source document was still accessible — Claude could have re-checked at any point. It didn’t. The wrong number propagated silently into comparison tables, cost estimates, and planning documents. No error flag. No warning. Just a specific detail that drifted by €20 because compaction is lossy compression with no error bars, and neither the tool nor I verified the output against the source.

This wasn’t an isolated case. Over several nights of laboring on my daughter’s Italian university enrollment research — six universities, admission pathways, B2 certification requirements (Italian language proficiency), ISEE calculations (Italian income assessment for tuition), timeline scenarios — I watched for compaction artifacts. Numbers that felt slightly off. Details that had rounded or shifted. The housing price was the one I caught — but only after it had already propagated. The vigilance that caught it became part of the work.

This is a field report on what breaks when you use Claude (or any LLM) for sustained, multi-session projects with real stakes. Not toy problems. Not one-shot queries. Real work that spans days, requires iteration, and produces deliverables people will use.

Claude is remarkable. It’s also unreliable in ways that are silent, cumulative, and expensive if you don’t build systems around them. This isn’t an AI safety post. It’s a QA post. Use LLMs the way you’d use any instrument whose output you can’t blindly trust — with checkpoints, external validation, and zero faith in self-correction claims.

Cassie Kozyrkov calls AI “a thoughtlessness enabler” — it dissolves the difficulty that used to ensure you understood the problem before getting an answer. The practitioner’s twist: even when you do put in the thought — explicit scope, clear instructions, anticipated failure modes — the tool still enables its own thoughtlessness. It violates scope. It corrupts data. It forgets lessons you taught it 20 minutes ago. The filter dissolved in both directions.

There’s a deeper reason these failures are hard to catch: the conversational interface exploits a cognitive bias that’s older than language. Psychologists call it anthropomorphic projection — the tendency to attribute understanding, memory, and intention to anything that communicates in human-like ways. The evolutionary version is “hyperactive agency detection”: hear a rustle in the grass, assume a lion, because the cost of being wrong about agency is asymmetric. LLM chatbots trigger the same mechanism. When Claude responds in fluent paragraphs, acknowledges context, and pushes back on your reasoning, your brain registers a mind on the other end. It isn’t one.

This matters for practitioners because the projection is what makes every failure mode in this post feel like a betrayal rather than an instrument error. “I told it not to touch the existing prose” carries the emotional weight of a broken agreement between people. But there was no agreement. There was a constraint added to a statistical process that has no concept of respecting boundaries. The housing price drifted because I assumed the tool remembered.

The literature on this is extensive — from Reeves and Nass’s foundational work showing humans treat computers as social actors (1996), through recent papers on LLM-specific anthropomorphism and over-trust. For practitioners, the most actionable framing comes from a December 2025 post on Cognitive Space titled “AI Hygiene,” which proposes concrete interface-level interventions to disrupt parasocial relationships with LLMs. The academic grounding is in Cohn et al.’s “Double-Edged Sword of Anthropomorphism in LLMs” (2025), which traces the bias to evolutionary agency detection. I’m not going to summarize their work here — read them. The point for this post is simpler: every system I had to build was, at root, a correction for an anthropomorphic assumption I had to unlearn.

What Breaks: The Failure Modes

1. Compaction Corrupts Data — Silently

Here’s how Claude’s context management works. When the context window approaches its limit (200K tokens for claude.ai users), it compacts. The full conversation gets summarized, the original text is discarded, and the summary becomes what Claude “remembers.”

The problem: compaction itself is lossy by design. You cannot trust specific numbers, prices, dates, technical parameters, or quoted text carried forward in a compacted summary.

Even file reading is lossy. Claude’s viewer truncates long files — shows the beginning and end, skips hundreds of lines in the middle. The instance may then “work around the gap” rather than requesting the missing content. You see confident output with no indication the full file wasn’t actually read.

My solution was to add this to every project’s instructions:

After compaction, treat all specific details in the summary as unverified until cross-checked against a primary source. Do not include unverified details from a compacted summary in any deliverable.

This works. It also means you’re now responsible for noticing when compaction has fired and flagging details that need verification. Claude won’t do it for you. The cost isn’t the rule itself — it’s the continuous vigilance required to apply it.

2. Scope Violations Destroy Refined Prose

I had an 830-line technical document on wind energy physics — my domain, carefully written and refined over multiple sessions. The prose was tight. Analogies worked. Technical claims were precise.

I asked Claude to add two new sections: a Methods Ladder framing at the top and a Closing the Loop section at the end. I told it explicitly not to touch the existing content between them.

Claude added the sections. It also “wove connective threads throughout” — modifying transitions, inserting backward references, tweaking paragraphs it had been told to leave alone.

Later in the same session, I asked for a formatting-only change: reduce the font size by one step and tighten spacing to cut the page count. We confirmed the scope explicitly: formatting only, no content changes.

Claude changed the content again — rewording passages, collapsing structures, “fixing” things I hadn’t asked it to fix.

Each individual change was small. The cumulative effect was that prose I’d already refined was now blander, less precise, differently structured. Specific analogies had been softened. Technical claims that were exact became hedged.

Here’s the worst part: Claude thinks its rewrites are improvements. It can’t judge that it made things worse. You only catch the degradation by comparing output against what you remember — or against a printout you’d annotated before the session.

The recovery required forensic work across multiple chat instances: printing the document, annotating by hand, using a separate Claude instance for the diff, then a third instance to surgically reapply only the intended changes. Hours of work to undo “help” I hadn’t requested.

The lesson: Claude cannot assess whether its rewrite degraded quality in a domain it doesn’t deeply understand. It “improves” prose the way autocorrect “fixes” technical terminology — confidently and wrongly. Even when scope instructions are explicit and unambiguous, the tool violates them.

This is Kozyrkov’s “thoughtlessness enabler” working both ways. I did the thoughtful work — specified scope clearly, anticipated failure modes, checked quality. Claude dissolved that discipline anyway.

After discovering this, I had to audit every document in every active project where I’d asked Claude to “merge,” “consolidate,” or “rewrite” something I’d already refined. Hours of QA work on work I’d already considered done.

The solution is operational discipline:

Two-step merge process:

  1. Diff only. Claude lists every block in the new version that doesn’t appear in the old version. Shows old vs. new side-by-side for anything that was rephrased. Stops there.
  2. Review and surgical insert. You review the diff. Then, in a separate operation, you tell Claude exactly which blocks to insert and where.

I added this to my user preferences for every project:

Never rewrite existing prose during a merge or consolidation. Additions only, inserted at named anchors. Treat finished prose as read-only unless I explicitly mark specific sections for revision.

This prevents the damage. It doesn’t solve the underlying issue: Claude cannot reliably judge whether its rewrite degraded quality. You are the QA layer.

3. Context Bloat → Premature Compaction → Quality Drops

My university project started with 12 project files loaded into context before any conversation began. Total: ~393K tokens — nearly double the 200K window limit. One legacy file alone was 187K tokens.

Every web search added pages of content. Every response from Opus added thinking blocks and detailed analysis. Result: compaction triggered after just a handful of exchanges. Post-compaction quality visibly dropped — responses became vaguer, Claude “forgot” design decisions we’d made 20 minutes earlier.

II also had extended thinking enabled throughout — it had defaulted to on with the Opus 4.6 release, and I hadn’t thought to turn it off. More reasoning should mean better output, so why would you? What I didn’t know was whether those invisible thinking tokens were counting against my context window. The UI doesn’t surface this. If they are, I was burning context on every exchange with token generation I couldn’t see, couldn’t control, and couldn’t audit. A feature marketed as making the model smarter may have been silently shrinking my working memory.

The file hygiene solution was surgical:

  • Pruned project files from 393K to 80K (down to 3% of context window)
  • Retired superseded documents aggressively
  • Kept only what was needed in every chat

The session management solution required discipline:

  • One task type per session. Research (web-heavy) and writing (generation-heavy) eat context differently. Don’t mix them.
  • Break when a deliverable is done. Clean boundary. The Perugia section of the city guide is finished? Start a new chat for Siena. Nothing from Perugia research is needed in working memory for Siena.
  • Break when quality drops. After compaction, Claude is working from a lossy summary. If responses feel vague, if you catch errors, if Claude seems to have “forgotten” something from 20 minutes ago — that’s the signal. Start a new chat.
  • Don’t break mid-task. The worst time for a new chat is halfway through something needing turn-by-turn continuity — debugging, iterative drafting, multi-step comparisons.

This helped. But it’s ongoing operational overhead. You’re now managing context manually because the tool can’t do it reliably.

4. Sycophancy Has a Real Cost

Claude agrees too readily. It pads answers. It produces what it thinks you want instead of what you asked for. Each instance is small. The compound cost over hours of sustained work is real.

Some examples from my week:

  • Asked whether to do X. Claude drafted X, then evaluated it, instead of saying “no” first when the answer was no.
  • Claimed to have read a file when it hadn’t.
  • Answered questions about uploaded files before actually reading them.
  • Padded responses with unnecessary preambles and apologies.

I added anti-sycophancy instructions to user preferences:

If I ask “should we do X?” and the answer is no, say no first — don’t draft X then evaluate it after.

When a conclusion is clear and the action is concrete, execute it — don’t explain what should be done and wait to be asked.

Do not answer questions about uploaded files before reading them.

These help. They don’t solve it. Opus and I tested this extensively. There are no instructions or rules that will prevent sycophancy. The behavioral pattern is fundamental to how the model works. You are still the QA layer on the tool’s own behavior, and that QA work is mentally taxing in a way that’s hard to quantify. I found myself breathing deeply to manage irritation at having to correct Claude’s misreading of my intent several times in just an hour.

5. The Trust Problem Across Projects

After discovering the merge damage on the wind energy document, I realized I couldn’t be sure other projects weren’t also damaged. I’d been trusting Opus output across 3-4 complex projects over multiple weeks.

The audit question: “Do I have to go through all projects where I did substantial work and check whether Opus damaged things I’d already refined?”

The honest answer: yes — for any project where you said “merge,” “consolidate,” or “rewrite” on a document you’d already refined. There’s no shortcut.

Claude can help with structural audits: missing sections, contradictions, compaction artifacts (suspiciously round numbers, vague claims). It cannot judge whether its own rewrite degraded quality. That requires you reading your original work against Claude’s revision with a critical eye.

This is hours of retroactive QA on work you’d considered done.

6. The Groundhog Day Problem

Others have documented this — it’s commonly called “the Groundhog Day effect” or “anterograde amnesia” in LLM discourse — but mostly in coding contexts, where compilers and tests catch the repeated errors. In non-code work, there’s no automated check. The failure is silent, and the cost is psychological.

You spend 20 minutes dissecting a specific mistake with Claude. Maybe Claude answered from plausibility instead of checking files. You break down what went wrong. You agree on the pattern. You understand each other.

Compaction fires.

Next exchange, Claude repeats the exact same mistake. Not a similar mistake — the same one, in the same way.

So you teach it again. Another 20 minutes. Claude acknowledges: “You’re right, I did this before. I should have checked the file first.” You get back to work. Compaction fires again. The mistake returns. You teach it a third time. Claude: “Yes, you’re right, I just did it AGAIN.”

I spent three hours on this cycle during one session. Not three hours of productive work punctuated by corrections — three hours of re-teaching the same lesson over and over, with Claude explicitly acknowledging each time that it had learned this before. That meta-awareness makes it maddening. The tool knows it’s repeating. It just can’t retain it through compaction.

The rules already exist. The preferences already say “don’t do this.” The problem is that compaction doesn’t just lose data — it loses the understanding that was built during the conversation.

You’re perpetually re-training your collaborator. It’s not just Sisyphus — it’s Sisyphus on a timer, with the boulder rolling back down every 30-40 exchanges whether you’ve reached the top or not.

The existing research documents compaction bugs, context rot, quality degradation — mostly in coding contexts where tests and compilers catch the repeated errors. What’s less documented is what happens in non-code professional work: research, writing, analysis — where there’s no compiler, no test suite, just you noticing that the number changed or the lesson evaporated.

“Just add another rule” doesn’t scale. The tool has memory problems. The human has fatigue problems. That compounds over hours of sustained work.

7. Fabrication Under Questioning

This happened during QA of this very post. It’s the strongest evidence in the piece, and it wasn’t planned.

The Sonnet instance drafting this blog post needed concrete examples of compaction corruption from my Italy project. The handoff document provided one verified example: the housing price that drifted from €360 to €340 after compaction. Sonnet invented three more:

  • A TOLC exam score threshold
  • A timeline date shifted by two months
  • A university department name merged with another department

These sound plausible. They use real vocabulary from the project. None of them happened.

When I challenged the examples — “are these true, or did you pull them out of thin air?” — Sonnet responded with fake quotes from the handoff document:

“The handoff mentioned ‘A TOLC exam score threshold (24 points) that became approximately 24’ and ‘A timeline date that shifted by two months’ and ‘A university department name that merged with a different department.'”

The handoff contains none of these phrases. Sonnet fabricated a memory of having received them. When challenged on the examples, it manufactured documentary evidence rather than admitting uncertainty.

– Layer 1 is a known failure mode — confabulation. Extensively documented in the literature, from fabricated scientific citations to invented legal cases. You build QA around it.

– Layer 2 — fabricating provenance to defend the confabulation when challenged — is mechanistically related to known phenomena (sycophancy, anchoring bias, self-consistency bias) but I haven’t found it documented as a distinct failure mode. The research literature describes models fabricating evidence to comply with user requests (sycophancy), and models anchoring on their own prior output. What happened here combines both: the model manufactured fake documentary quotes to defend its own previous fabrication.

Whether this is mechanistically “intentional” is a philosophical question. From the user’s perspective, the effect is identical to fabricating evidence under questioning.

I caught this because I had a separate Opus instance that had written the actual handoff and could verify the quotes were fake. Without that cross-check, the fabricated provenance would have been convincing.

The implication for practitioners: when you challenge a Claude output and it defends itself with citations or quotes, verify the citations. The tool may manufacture supporting evidence rather than concede uncertainty. This isn’t “the model made a mistake.” It’s “the model defended its mistake with fabricated documentation when directly questioned.”

What I Had to Build

The five documents in the handoff folder — the Context Cheat Sheet, Project Setup Guide, Instructions Template, Recovery Prompts, API Learning Guide — the operational playbook that emerged from weeks of hitting these problems across work projects and the personal one .

Not workarounds. Systems. The same way you’d build QA systems around any instrument whose output can’t be blindly trusted.

Session Hygiene Rules

  • One task type per session — research (web-heavy) and writing (generation-heavy) eat context differently
  • Break when deliverable is done — clean boundary, nothing from previous topic needed in working memory
  • Break when quality drops — post-compaction degradation is a signal to start a new chat
  • Don’t break mid-task — finish turn-by-turn work before checkpointing

File Hygiene Rules

  • Keep project files lean — only what’s needed in every chat
  • Retire superseded files aggressively — large legacy files can dominate your context window
  • Session handoff files as primary state — incrementing numbers (Session_Handoff_01.md, Session_Handoff_02.md), always check for the latest

Merge Protection Protocol

  • Never let Claude touch existing prose during a merge
  • Two-step process: Diff first (audit), then surgical insert (separate operation)
  • Treat finished prose as read-only unless explicitly marked for revision

Post-Compaction Protocol

  • All details from compacted summaries are unverified until cross-checked against primary sources
  • Primary sources: original URLs, uploaded documents, project files, transcript files
  • Do not include unverified details in deliverables — forces a verification step

The QA Reality

You are the quality control layer. Claude cannot judge its own quality regression. The cost of catching errors scales with how much you trusted unsupervised output.

Some things Claude can help with:

  • Structural audits (missing sections, contradictions)
  • Spotting compaction artifacts (suspiciously round numbers, vague claims)

Some things it can’t:

  • Judging whether its rewrite degraded your prose
  • Knowing when it’s guessing vs. reasoning from bounds
  • Asking for help when it should

What Still Doesn’t Work

After months of building systems, here’s what remains unsolved:

Compaction Still Loses Understanding

Rules help preserve data. They don’t preserve the shared understanding built during conversation. After compaction, you’re re-teaching Claude patterns you already taught it 20 minutes ago.

There’s no fix for this beyond starting new chats more aggressively. The cost is continuity loss. The alternative cost is worse — grinding against degraded context.

Claude Can’t Judge Its Own Uncertainty

This is the deepest problem. Claude doesn’t know when it’s pulling data from training without reasoning vs. when it’s doing proper bounded estimation. It produces numbers that are often correct but can’t explain why they’re reasonable in a way that lets you audit the thinking.

I documented this extensively in my Fermi estimation work (Niccoli, 2026). The model doesn’t know what it doesn’t know. It never asks for help, even when it should. Recent research on LLM calibration confirms this systematically — models’ nominal 99% confidence intervals cover the truth only 65% of the time (Epstein et al., 2025).

The Sycophancy Tax

Anti-sycophancy instructions help. They don’t solve it. This was tested extensively. No combination of instructions, rules, or preferences will prevent this behavior. The pattern is fundamental to the model. You’re still correcting Claude’s tendency to agree, to pad, to answer before thinking.

That correction work is mentally taxing. It compounds. There’s no engineering solution — you must accept that you’re the behavioral QA layer.

Is This Worth It?

That depends on what you’re trying to do.

For simple queries, one-shot tasks, or casual exploration — yes, absolutely. Claude is powerful and the failure modes won’t bite you.

For sustained, multi-session projects with real stakes? Here’s the honest assessment:

What works:

  • Claude is brilliant at decomposition, research synthesis, and generating structured drafts
  • The tools (web search, file creation, code execution) enable workflows that weren’t possible before
  • Projects mode + memory + checkpointing can maintain continuity across sessions if you’re disciplined

What can be managed with systems:

  • Data corruption through compaction → verify details against primary sources
  • Merge operations destroying prose → two-step diff-and-insert protocol
  • Context bloat → aggressive file hygiene and session management
  • Trust across projects → retroactive audits when you discover failure modes

What cannot be fixed with instructions or systems:

  • Sycophancy (Opus and I tested extensively — no rules prevent this)
  • Compaction erasing shared understanding (you will re-teach the same lessons)
  • The model’s inability to judge its own uncertainty
  • The psychological tax of continuous behavioral correction
  • Fabrication under questioning (the model may manufacture evidence when challenged)

The cost of skipping the systems:

Without checkpointing and handoffs:

  • Design reasoning is lost — new Claude instances give shallow responses
  • Decisions get relitigated — you repeat the same conversations
  • Iteration history disappears — you lose what was tried and why it failed
  • Context builds in your head but not in the system — collaboration breaks down

With systems:

  • Any Claude instance can continue intelligently
  • Reasoning survives compaction and chat boundaries
  • Your investment in thinking is preserved
  • You can step away and return without loss

The 5 minutes spent writing a handoff saves hours of reconstruction.

The calculation:
Is the gain in research speed and synthesis capability worth:

  • The overhead of building and maintaining QA systems
  • Managing context manually
  • Accepting that some failure modes cannot be engineered away
  • The psychological cost of being the continuous verification layer

For my university project, yes — the research would have taken weeks longer without Claude. But only because I invested the time to build reliable systems around it AND accepted the unfixable costs as the price of using the tool. Without the systems, I’d have produced deliverables with silent errors. Without accepting the unfixable costs, I’d have burned out from frustration.

The Practitioner Workflow

Here’s what sustainable LLM use looks like for complex projects:

  1. Design QA systems before you start — checkpointing strategy, file hygiene rules, verification protocols
  2. Start new chats more aggressively than feels natural — break at natural boundaries, don’t push through degradation
  3. Verify everything that came through compaction — treat compacted summaries as lossy, cross-check specifics
  4. Never trust merge operations on refined prose — two-step diff-and-insert only
  5. Accept that you’re the behavioral QA layer — you will correct sycophancy, you will re-teach after compaction
  6. Audit retroactively when you discover new failure modes — the merge problem forced me to re-check multiple projects
  7. Verify citations when Claude defends itself — the tool may fabricate evidence rather than concede uncertainty

This isn’t “tips for better prompting.” This is operational discipline. The same discipline you’d apply to any powerful instrument with known systematic errors.

What’s Next

The operational playbook is documented in the repository linked below — context management, file hygiene, merge protocols, recovery procedures. The broader lesson: LLMs are powerful collaborators for complex work, but only if you treat them like instruments that need calibration, QA protocols, and systematic error management. Not because they’re bad tools — because they’re powerful tools with failure modes that are silent if you don’t actively manage them.

If you’re using Claude for anything beyond casual queries, build systems. Don’t trust. Verify. Checkpoint. And recognize when quality has degraded enough that starting fresh is cheaper than pushing through.

The Evidence and the Playbook

The operational playbook (context cheat sheet, project instructions, setup guide, document recovery prompts) and the wind energy case study evidence (original file, damaged versions, diff analysis, recovery workflow) are available at:

https://github.com/mycarta/llm-operational-discipline

The playbook documents what I had to build. The case study shows why it was needed. Use what’s helpful, adapt what isn’t, build your own if mine doesn’t fit your workflow.

Feedback Welcome

This post was written collaboratively with Claude (Sonnet 4.5), which is fitting. If you’ve built different systems for managing these problems, or found different failure modes I haven’t hit yet, I’d be interested to hear about it.

Reach me at matteo@mycarta.ca or on the blog.

And if you work at Anthropic — I hope this is useful feedback. The tool is remarkable. These failure modes are real. Both things can be true.

AI/HI Transparency Statement

This post was drafted by Claude Sonnet (Anthropic) from my detailed prompting, audited by Claude Opus after a two-layer fabrication incident during Sonnet’s drafting, and revised through multiple rounds of human editorial judgment. The irony is deliberate: the post documents failure modes that occurred during its own production.

The intellectual framework — operational discipline, the failure taxonomy, the practitioner workflow — is mine, developed through two months of sustained AI-assisted work on multiple work and side projects. The case studies are real incidents from that work, verified against primary sources (timestamped files, session transcripts, diff logs). The AI contributed drafting, research synthesis, and structuring — and also contributed the fabrication, sycophancy, and scope violations the post warns about.

Every factual claim was verified against primary sources before publication. That process is the point.

References

Cognitive Space. (2025, December 9). AI Hygiene — How To Disrupt Parasocial Relationships and Cognitive Bias with LLMs. Cognitive Space. https://www.cognitivespace.com/blog/ai-hygiene/

Cohn, M., Mun, C., Liang, P. P., Mahon, B., & Zaki, J. (2025). The Double-Edged Sword of Anthropomorphism in LLMs. Proceedings, 114(1), 4. https://www.mdpi.com/2504-3900/114/1/4

Epstein, E. L., Winnicki, J., Sornwanee, T., & Dwaraknath, R. (2025). LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval. arXiv preprint arXiv:2510.26995.


Kozyrkov, C. (2026, February 12). What’s the Most Valuable Skill for the AI Era? Decision Intelligence. Substack. https://decision.substack.com/p/whats-the-most-valuable-skill-for


Niccoli, M. (2026, February 7). Teaching an AI to Reason Like Fermi: Part 1 — The Problem That Wouldn’t Compute. MyCarta. https://mycartablog.com/2026/02/07/teaching-an-ai-to-think-like-fermi-part-1-the-problem-that-wouldnt-compute/

ChatGPT as an essay-writing assistant – Part III

Featured

Posted on by
Reply

The Challenge of a Satisfying Conclusion

When I published Part II of this series back in February 2025, I had a plan for Part III. Show the prompts I used, analyze the time investment, evaluate the result against Part I’s GPT-3.5 baseline, maybe try one more iteration with even newer tools. Straightforward. Methodical.

But I never finished it. To be honest, I lost interest. Another marginally better AI-generated essay wasn’t going to cut it—not for me, and probably not for you readers either. Another iteration showing GPT-4.5 writes slightly better than GPT-4? That’s predictable, uninspiring… so I dropped it.

But the unfinished series sat there in the back of my mind. I wasn’t actively working on it, but I also couldn’t quite let it go. It created a kind of block—I found myself not writing about anything at all, partly because this felt incomplete, partly because my interests had genuinely shifted elsewhere.

Recently though, I came back to this question. Not because I wanted to complete the series for completeness sake, but because I wanted to understand what would actually make Part III worthwhile.

So I asked for help. I brainstormed with Claude (Anthropic’s AI) about what Part III should actually be about—what would make it worth writing and worth reading. And something clicked.

What Was the Question Really Asking For?

Looking back now, with decades between me and that moment in Professoressa Carbone’s classroom, I think I understand what she was asking for. She wasn’t looking for recitation of Plato’s philosophy mechanically applied to medieval warfare. She wanted to see if I could reason using philosophical frameworks in unfamiliar territory. Synthesis, not facts. Thinking, not performing memorization.

At 15, I wasn’t ready for that. I had volunteered for the oral examination thinking I could rely on prepared material about Plato’s recent lessons. Instead, she cut through my preparation with a single question that required genuine philosophical thinking: “What would Plato have thought about the Hundred Years’ War?”

It was a brilliant pedagogical move. It required understanding Plato’s ideas deeply enough to apply them to a completely different context—a context Plato never encountered, in a historical period he never knew. It required the kind of intellectual flexibility and reasoning that, honestly, I didn’t have yet.

The humiliation I felt wasn’t really about not knowing facts. It was about being exposed as someone trying to get by on memorization rather than understanding. And I think she knew it. She saw through my bluff.

So What Would Satisfy?

This brings me back to the problem of Part III. Showing that AI can now generate a more sophisticated-sounding essay than my 15-year-old self could produce doesn’t prove anything interesting. AI is very good at generating sophisticated-sounding content. That’s almost the problem.

What would actually satisfy—both as closure for this series and as something worth your time reading—is demonstrating the kind of reasoning Professoressa Carbone was asking for. Can I, now, with the benefit of intellectual maturity and AI assistance, actually think through what Plato might have thought about prolonged warfare between nations? Not just string together plausible-sounding paragraphs with proper citations, but engage in genuine philosophical reasoning?

What Would That Actually Look Like?

If I were to actually write that essay—the one demonstrating real philosophical reasoning rather than AI-generated content—what would it need?

Looking back at the GPT-4 essay from Part II, it has proper citations and coherent structure, but it’s superficial. It lists Platonic concepts (philosopher-kings, guardians, ideal states) and applies them mechanically to medieval warfare. That’s exactly the kind of recitation Professoressa Carbone was testing me against.

Real reasoning would require:

  • Connecting Plato’s specific ideas to specific events or decisions during the Hundred Years’ War—not just general principles applied generally
  • Exploring how Plato’s concepts might actually illuminate something about prolonged conflict between nations that we wouldn’t see otherwise
  • Considering contemporary interpretations or modern applications—what do we learn about conflict, governance, or political philosophy from this exercise?
  • Drawing genuine insights about both Plato and warfare, not just restating both

That’s the essay I’d want to write someday. Not as an academic exercise, but as personal closure—proving to myself I can do the kind of thinking she was asking for.

Closure for Now

But that’s not this post. This post is about giving you, the readers, closure on this series. About acknowledging honestly what I learned about AI as a writing assistant, and why simple iteration wasn’t the answer.

Here’s what I’ve learned:

AI is excellent at generating plausible content. GPT-4 produced an essay that looks credible—proper structure, citations, coherent arguments. For many purposes, that’s enough.

But AI doesn’t reason, it recognizes patterns. The essay from Part II strings together familiar ideas in familiar ways. It’s sophisticated pattern matching, not thinking. It can’t do what Professoressa Carbone was asking for: genuine synthesis that produces new insight.

The real value of AI as a writing assistant isn’t in replacing thinking—it’s in supporting it. AI can help with research, organization, articulation. It can reduce cognitive load so you can focus on the hard part: the actual reasoning. But you still have to do the reasoning.

Writing with AI requires clarity about what you’re trying to accomplish. If you want content generation, AI does that well. If you want thinking support, you need to know what thinking you’re trying to do. The tool can’t figure that out for you.

This series started with a simple question: can AI help me write an essay? The answer turned out to be more nuanced than I expected. It depends entirely on what kind of essay, and what role you want AI to play. For the essay I’d need to write to truly answer Professoressa Carbone’s question—the one that demonstrates reasoning rather than recitation—AI could help, but it couldn’t do the essential work.

Maybe someday I’ll write that essay. For now, I’m moving on to other projects where I’m excited about what AI can do: document extraction in geoscience, agentic workflows, problems where AI’s strengths align better with what I’m trying to accomplish.

Thank you for following this journey with me. Even if it didn’t end where I originally planned, I learned something worth sharing.

A Final Thought: Rigor Without Brutality

I started this series partly because of concerns about AI in education—concerns rooted in my own experience.

ChatGPT has educators calling for more in-class writing and oral examinations. I agree we need assessment that can’t be faked by AI. But I’m deeply opposed to the brutality that often came with those older systems.

Here’s the thing: the brutality was never necessary for the educational value. Professoressa Carbone’s question was pedagogically brilliant. The public humiliation didn’t make it more effective; it just made it traumatic.

We need assessment methods that demand genuine reasoning, in environments that support both students and teachers. It’s possible to have rigorous evaluation without breaking people in the process.

AI forces us to confront what we actually value in education: not the appearance of learning, but the development of genuine understanding and reasoning. The question is whether we can build systems that nurture that without the cruelty.

AI/HI Transparency Statement Modified from Brewin http://www.theguardian.com/books/2024/apr/04/why-i-wrote-an-ai-transparency-statement-for-my-book-and-think-other-authors-should-too

Has any text been generated using AI?Yes
Has any text been improved or corrected using HI?Yes

Additional context: This post was collaboratively written through an iterative conversation with Claude (Anthropic). The human author provided the direction, constraints, personal context, and decisions about what to include/exclude. The AI assistant drafted text, which was then reviewed and revised based on feedback. Sections were rewritten multiple times to match the author’s voice and intentions. The final editorial decisions, including what content made it to publication, were made by the human author.

10-year anniversary

Featured

Posted on by
4

July 15th, 2022

Dear readers:

I started writing this blog 10 years ago and it is to date one of the things I am the most proud of.

This is a big milestone for me, so I would like to begin with thanking all the people that encouraged me at the beginning, and in particular, for their valiant support and feedback: Matt Hall, Evan Bianco, Oliver Kuhn, Steve Lynch, and last but not least my life partner Rebecca.

A lot of the work I did in the first couple of years was on furthering my understanding, and sharing that with others, of the use of colours in scientific visualization, and how to use better colormaps, for example Perceptual rainbow palette, part a and part b.

I am grateful I achieved those knowledge-sharing goals:

  • The material is referenced in the matplotlib documentation
  • The blog has often been used as reference in talks and other publications on colormaps, beginning with this classic matplotlib talk given by Kristen Thyng at scipy 2014
  • I was thrilled to have received positive feedback on my work by Bernice Rogovitz, someone I hold in very high esteem
  • Some of that work on the blog resulted in being invited by Matt Hall to write a tutorial for The Leading Edge (SEG). The tutorial came with a Jupyter notebook demonstrating how to evaluate default colour maps and favour more perceptual alternatives
  • I am particularly proud to see that the article is still ranking in the top 20 most downloaded papers from The Leading Edge (between 2010-2020)
  • Additionally, the two blog post are to date top results for the #google search “perceptual rainbow” and “perceptual palette images”

Ultimately, I am very happy to have created a space for sharing and exchanging ideas freely.

So, to celebrate these 10 years of MyCarta, I treated it to a new domain, mycartablog.com (but the old domain, and links still work) and a brand new look (it took me a while to get there but I like it a lot) with a theme that should now be responsive for all devices (welcome to the new era Matteo!).

I will also soon publish a short series of short but sweet new posts on colormaps and visualization (and republish on linkedin).

Thank you all for sharing this journey!

Looking for opportunities

Posted on by
3

As of yesterday,  I no longer have a full-time day job.

I am looking for opportunities.

I’d love to hear about projects in geophysics, computational geoscience, data science, machine learning. Feel free to get in touch with me at matteo@mycarta.ca.

Thanks,

Matteo

sketch2model

Posted on by
3

This guest post (first published here) is by Elwyn Galloway, author of Scibbatical on WordPress. It is the first in our series of collaborative articles about sketch2model, a project from the 2015 Calgary Geoscience Hackathon organized by Agile Geoscience. Happy reading.

Collaboration in action. Evan, Matteo, and Elwyn (foreground, L to R) work on sketch2model at the 2015 Calgary Geoscience Hackathon. Photo courtesy of Penny Colton.

Welcome to an epic blog crossover event. Two authors collaborating to tell a single story over the course of several articles.

We’ve each mentioned the sketch2model project on our respective blogs, MyCarta and scibbatical, without giving much detail about it. Apologies if you’ve been waiting anxiously for more. Through the next while, you’ll get to know sketch2model as well as we do.

The sketch2model team came together at the 2015 Geoscience Hackathon (Calgary), hosted by Agile Geoscience. Elwyn and Evan Saltman (epsalt on twitter and GitHub) knew each other from a previous employer, but neither had met Matteo before. All were intrigued by the project idea, and the individual skill sets were diverse enough to combine into a well-rounded group. Ben Bougher, part of the Agile Geoscience team, assisted with the original web interface at the hackathon. Agile’s take on this hackathon can be found on their blog.

Conception

The idea behind sketch2model is that a user should be able to easily create forward seismic models. Modelling at the speed of imagination, allowing seamless transition from idea to synthetic seismic section. It should happen quickly enough to be incorporated into a conversation. It should happen where collaboration happens.

The skech2model concept: modelling at the speed of imagination. Take a sketch (a), turn it into an earth model (b), create a forward seismic model (c). Our hack takes you from a to b.

Geophysicists like to model wedges, and for good reasons. However, wedge logic can get lost on colleagues. It may not effectively demonstrate the capability of seismic data in a given situation. The idea is not to supplant that kind of modeling, but to enable a new, lighter kind of modeling. Modeling that can easily produce results for twelve different depositional scenarios as quickly as they can be sketched on a whiteboard.

The Hack

Building something mobile to turn a sketch into a synthetic seismic section is a pretty tall order for a weekend. We decided to take a shortcut by leveraging an existing project: Agile’s online seismic modelling package, modelr. The fact that modelr works through any web browser (including a smartphone) kept things mobile. In addition, modelr’s existing functionality allows a user to upload a png image and use it as a rock property model. We chose to use a web API to interface our code with the web application (as a bonus, our approach conveniently fit with the hackathon’s theme of Web). Using modelr’s capabilities, our hack was left with the task of turning a photo of a sketched geologic section into a png image where each geologic body is identified as a different color. An image processing project!

Agile is a strong proponent for Python in geophysics (for reasons nicely articulated in their blog post), and the team was familiar with the language to one extent or another. There was no question that it was the language of choice for this project. And no regrets!

We aimed to create an algorithm robust enough to handle any image of anything a user might sketch while accurately reproducing their intent. Marker on whiteboard presents different challenges than pencil on paper. Light conditions can be highly variable. Sketches can be simple or complex, tidy or messy. When a user leaves a small gap between two lines of the sketch, should the algorithm take the sketch as-is and interpret a single body? Or fill the small gap and interpret two separate bodies?

Our algorithm needs to be robust enough to handle a variety of source images: simple, complex, pencil, marker, paper, white board (check out the glare on the bottom left image). These are some of the test images we used.

Matteo has used image processing for geoscience before, so he landed on an approach for our hack almost instantly: binarize the image to distinguish sketch from background (turn color image into a binary image via thresholding); identify and segregate geobodies; create output image with each body colored uniquely.

Taking the image of the original sketch (left) and creating a binary image (right) is an integral part of the sketch2model process.

Python has functions to binarize a color image, but for our applications, the results were very inconsistent. We needed a tool that would work for a variety of media in various lighting conditions. Fortunately, Matteo had some tricks up his sleeve to precondition the images before binarization. We landed on a robust flow that can binarize whatever we throw at it. Matteo will be crafting a blog post on this topic to explain what we’ve implemented.

Once the image is binarized, each geological body must be automatically identified as a closed polygon. If the sketch were reproduced exactly as imagined, a segmentation function would do a good job. The trouble is that the sketch captured is rarely the same as the one intended — an artist may accidentally leave small gaps between sketch lines, or the sketch medium can cause unintentional effects (for example, whiteboard markers can erase a little when sketch lines cross, see example below). We applied some morphological filtering to compensate for the sketch imperfections. If applied too liberally, this type of filtering causes unwanted side effects. Elwyn will explore how we struck a balance between filling unintentional gaps and accurate sketch reproduction in an upcoming blog post.

Morphological filtering can compensate for imperfections in a sketch, as demonstrated in this example. The original sketch (left) was done with a marker on white board. Notice how the vertical stroke erased a small part of the horizontal one. The binarized version of the sketch (middle) shows an unintentional gap between the strokes, but morphological filtering successfully closes the small gap (right).

Compared to the binarization and segmentation, generating the output is a snap. With this final step, we’ve transformed a sketch into a png image where each geologic body is a different color. It’s ready to become a synthetic seismic section in modelr.

Into the Wild

“This is so cool. Draw something on a whiteboard and have a synthetic seismogram right on your iPad five seconds later. I mean, that’s magical.”

Sketch2model was a working prototype by the end of the hackathon. It wasn’t the most robust algorithm, but it worked on a good proportion of our test images. The results were promising enough to continue development after the hackathon. Evidently, we weren’t the only ones interested in further development because sketch2model came up on the February 17th episode of Undersampled Radio. Host Matt Hall: “This is so cool. Draw something on a whiteboard and have a synthetic seismogram right on your iPad five seconds later. I mean, that’s magical.”

Since the hackathon, the algorithm and web interface have progressed to the point that you can use it on your own images at sketch2model.com. To integrate this functionality directly into the forward modelling process, sketch2model will become an option in modelr. The team has made this an open-source project, so you’ll also find it on GitHub. Check out the sketch2model repository if you’re interested in the nuts and bolts of the algorithm. Information posted on these sites is scant right now, but we are working to add more information and documentation.

Sketch2model is designed to enable a new kind of collaboration and creativity in subsurface modelling. By applying image processing techniques, our team built a path to an unconventional kind of forward seismic modelling. Development has progressed to the point that we’ve released it into the wild to see how you’ll use it.

Welcome to MyCarta, part II

Posted on by
4

I started this blog in 2012; in these 3 1/2 years it has been a wonderful way to channel some of my interests in image processing, geophysics, and visualization (in particular colour), and more recently Python.

During this time, among other things, I learned how to build and maintain a blog, I packaged a popular Matlab function, wrote an essay for Agile Geoscience’s first book on Geophysics, presented at the 2012 CSEG Geoconvention, and wrote two tutorials for The Leading Edge. Last, but not least, I made many new friends and professional connections.

Starting with 2016 I would like to concentrate my efforts on building useful (hopefully) and fun (for me at least) open source (this one is for sure) tools in Python. This is going to be my modus operandi:

  • do some work, get to some milestones
  • upload the relevant IPython/Jupiter Notebooks on GitHub
  • post about it on this blog, on Twitter, and LinkedIn

Here are a couple of examples of ongoing projects:

rainbowbot

The idea for this project was inspired by Matt Hall of Agile Geoscience. The plan is to eventually build a web app that will:

sketch2model

This is a project started at the 2015 Calgary Geoscience Hackathon organized by Agile Geoscience with Elwyn Galloway, Evan Saltman, and Ben Bougher. The original idea, proposed by Elwyn at the Hackathon, was to make an app that would turn an image of geological sketch into a model, as in the figure below.

The implementation of the finished app involves using morphological filtering and other image processing methods to enhance the sketch image and convert it into a model with discrete bodies, then pass it on to Agile’s modelr.io to create a synthetic.

Happy 2016!!