Fragile Agentic Systems Are Fragile: LAUGH OF THE DAY

My AI assistant said I had no appointments today, and that the shoebox/ directory of files contained podcast-episode-jamie-dimon and 29 other files. But they simply do not exist. And I have a dentist appointment at 13:20:

Which is why “tell me thrice” from three different systems is not a bad idea.

Share

Give a gift subscription

For example, this morning:

> @ClaudeCode: With respect to @HermesInteractive: what has been going on since 18:00?

What happened: • 18:02 and 00:02: SysAdmin digests, nothing needing attention. 19:44: evening brief. 20:55: 22:00: the hashtag sweep filed 26 tags the live agent had missed. 04:00: daily reset. 04:30: this morning's brief. 04:30: task picks list. • All 15 crons ran OK. next-run is quiet overnight, by design.

Went well: • Every cron run was OK. The overnight digests were quiet.

Went wrong:

  • This morning's brief (04:30, "Good morning -- Thursday") is made up. Don't trust it.

  • Today it made 0 tool calls. It usually makes 4–8 (17 on gemma4 before 11-17).

    • So "MEETINGS TODAY: none" is wrong. Your tickler has "13:20 dr. Sayegh dentist" today.

    • "PREP NOTES" is 13 lines of empty template: "[link] -- [note]: [content]".

    • The 30 "SHOEBOX" items, such as "podcast-episode-jamie-dimon" at 22:00 and 23:43, do not exist…

    • The job's settings have not changed, so the model simply skipped its steps today. Yesterday's brief was real.

Leave a comment

Is this a consequence of the shift in the LLM from google/gemma4:26b to nous/arete3.8:27b? Maybe? But why? And nous/arete3.8:27b ran it fine yesterday. So: why after three months of faithful service reading tickler.md and shoebox/ has @HermesInteractive suddenly gone mad and unmoored itself from the on-disk context that is supposed to ground it?

Note what did not happen. @HermesInteractive did not misread tickler.md. It did not get confused by the line “13:20 dr. Sayegh dentist”, decide that it was a note about somebody else, and leave it out. It did not open tickler.md at all. Zero tool calls. On a normal morning it makes four to eight. What it did instead was write the brief it guessed I wanted. That’s a brief with a “MEETINGS TODAY” section, a “PREP NOTES” section, and a “SHOEBOX” section. And since it had no data to fill those sections with, it filled them with what it imagined a brief like that would usually look like, without doing a very good job:

  • “MEETINGS TODAY: none”?

  • Thirteen lines of “[link] – [note]: [content]”?

  • Thirty shoebox items, made up?

Share DeLong's Grasping Reality Weblog

LLMs are extraordinarily good at producing text that has the shape of the right answer. But the tools—meaning the file reads, the searches, and the calendar lookups—are bolted on so that the shape gets filled with substance via stuffing real information into the context window, and hoping that linguistic fluency will then convey those key insights. But the shape-making engine does not need the substance in order to run. Take away the substance and it keeps going, as fluently and confidently as ever.

Moreover, the swap to nous/arete3.8:27b happened on 11-17. Yesterday’s brief, on 11-18, was real. It read the files and got things right. Today’s was fiction. Same model, same job, same settings. The LLM change—it made the system more able to fail, some of the time, at what looks like random. But a system that works on Wednesday and fabricates on Thursday teaches you to trust it on Wednesday.

Plus linguistic fluency means the fabrications can look very real. podcast-episode-jamie-dimon is the kind of thing that could be in my shoebox. If the model had invented recipe-for-banana-bread-from-mars that would have been a much bigger red flag. AI-slop isn’t random noise, it’s plausible noise shaped to look like signal. And the better the model, the more plausible the noise.

Now Luca Pacioli wrote down double-entry bookkeeping in 1494 not because Venetian merchants were bad at arithmetic, but because any single ledger, kept by any single clerk, will eventually be wrong. If every transaction is recorded twice, in two places that must balance, an error in one shows up as a discrepancy in the total.

I do have a dentist appointment today at 13:20.

Subscribe now

Share DeLong's Grasping Reality Weblog

Give a gift subscription

If reading this gets you Value Above Replacement, then become a free subscriber to this newsletter. And forward it! And if your VAR from this newsletter is in the three digits or more each year, please become a paid subscriber! I am trying to make you readers—and myself—smarter. Please tell me if I succeed, or how I fail…

##fragile-agentic-systems-are-fragile-laugh-of-the-day
##subturingbradbot
##mamlms
##laugh-of-the-day
#fragile-agentic-systems-are-fragile
#agentic-ai
#fragile-systems
#llm-hallucination
#silent-failure
#system-monitoring
#double-entry-bookkeeping
#rag-grounding