HOISTED FROM COMMENTS: RAFAEL KAUFMANN: Carving Nature at the Joints: Faithful Representation, the Platonic Dream, & the Unreasonable Near-Success of GPT LLM MAMLMs
Perhaps why reality’s low-dimensional secrets let machines mimic us so well—and where the limits remain to the surprising power—and peril—of compressing the world into latent spaces. Or, when brute-force statistics become near-“intelligence,” and what’s still missing from the algorithmic mind…
Why do large language models seem so uncannily adept at predicting human discourse? And given that they are so successful at language games, why are they so inept at reality-description. Except, of course, where they amaze us by not being inept at reality-description. But unless you are a subject-matter expert, you can be easily fooled by their hallucinations?
Rafael Kaufmann’s incisive commentary, spotlighted here, contends that the surprising efficacy of LLMs is rooted in the principle of “faithful representation”—the idea that the world, at some deep level, is compressible into a handful of latent-variable dimensions, just as Plato once imagined. The real reality is not the buzzing chaotic confusion to which we try to give names, but the underlying logic and order of the Forms. This property, he argues, is why both scientific models and machine learning algorithms can “work” at all: reality’s underlying simplicity allows for effective compression, prediction, and imitation.
But as we automate the carving of nature at its joints, the question of true understanding looms ever larger. In the process of being stochastic parrots that emulate internet s***posters—LLMs must construct internal representations that map the hidden structures of language. The hope was that such internal representations would also map onto the underlying structures of reality—for since language is a tool to describe the world and how it works, the two should be, if not one-and-the-same, very close. Their statistical training will first find the same compressed mappings that undergird human language-games and then make the leap to the same compressed mappings that undergird human understanding. That was the hope. That is still the bet of those who see AGI arriving from GPT LLM MAMLMs any day now.
Yet, so far, they largely fail. But might they succeed? Someday? Soon?
Kaufmann’s commentary, hoisted for your consideration, explores how statistical models find the Platonic forms beneath our words. Yet, the leap from correlation to causation and from plausible verbiage to actual intelligent insight remains a chasm even the cleverest algorithm struggles to cross. Tracing the lineage from ancient philosophy to modern machine learning, we try to grasp the promise—and the peril—of machines that claim that their stochastic parrotage is on the edge of true understanding.
But the leap from successful representation of the shadows on the walls of the cave to true understanding of Nature’s Forms remains to be made. Learning causal models—discerning which states of the world cause which others—remains a far harder technical nut to crack, requiring intervention and experimentation rather than mere observation. Kaufmann notes that while LLMs can interpolate between the “little stories” humanity tells to explain the world, their ability to extrapolate reliably is tightly bound to the quality of their training data and the absence of direct grounding in reality.
The automation of representation is a monumental achievement, explaining the leap from “just autocomplete on steroids” to the breathless claims of AGI and ASI superenthusiasts. Yet, the absence of grounding in direct experimentation and the sensitivity to training data quality mean that LLMs, for all their prowess, remain fundamentally limited. The Platonic ideal is within reach, but the Promethean fire of true understanding has yet to be stolen.
Apropos of: J. Bradford DeLong: Spreadsheet, Not Skynet: Microdoses, Not Microprocessors <https://braddelong.substack.com/p/spreadsheet-not-skynet-microdoses>
Rafael Kaufmann of Gaia Network <https://engineeringideas.substack.com/p/gaia-network-an-illustrated-primer> <https://engineeringideas.substack.com/p/gaia-network-a-practical-incremental> writes:
Rafael Kaufmann: <https://braddelong.substack.com/p/spreadsheet-not-skynet-microdoses/comment/129169221>: ‘What Brad/Cosma’s explanation of LLM misses (along with most others) – the reason why Brad’s intuition “strongly leads [him] to think that they should not be able to do half as well as they do”—is the concept of faithful representation. Put simply: Reality allows itself to be mapped in a low-dimensional latent space, and some compression schemes are just good mappings. [Or at least (the instrumentalist view), of humanity’s currently-known shared ways to describe and predict our observations of reality.]
This is why our scientific and statistical models work in the first place. Brute-force ML works because (or to the extent that) it finds these mappings. LLMs work because their training process finds the same latent representations as (or equivalent to) the representations that internet sh*tposters have in their heads as they’re writing [1]. Therefore, they can quite accurately predict what a sh*tposter would have said, in pretty much any context.
Yes, this is just Plato’s concept of “carving nature at its joints” from ~2400 years ago. Quite surprisingly, it turns out that we’re all living at the unique moment in humanity’s history where we are discovering that such a carving is an objective possibility, and that we can, for the first time, automate this carving [2, 3]. And this does not just apply to language, BTW: if the success of multimodal models weren’t enough, tabular foundation models [4] demonstrate that good old statistics has the same property.
There is still the crucial question of whether/when, beyond “just” the latent representations, ML can learn the true causal world model: which latent states at t cause which latent states at t’ > t. This is, in a technical sense, much harder than learning temporal correlation, and at least in some cases, it’s impossible to learn just from the data, requiring the capability to intervene/experiment [5]. However, it has recently been proven that learning a good causal model is a strong requirement for robust behavior—ie, reliable extrapolation beyond the data [6].
We’ve been lucky so far that the Internet already has a lot of natural-language descriptions of causal models of just about everything. These can be compressed into “meta-representations” that let LLMs “interpolate extrapolations”. This is similar at some level to how humans learn much of their own extrapolation capability—not by experimenting themselves, but by learning theory from other people and representing it in their heads as “little stories” that they can interpolate.
However, because these meta-representations can’t be directly grounded in observational data, the way LLMs learn them is very sensitive to the quality of the training corpus. That is why stuff like embodied learning and synthetic data are hot topics in AI: This is stuff you want to get exactly right.
Regardless, “just” getting the representation right is a huge step in the direction, and it goes a long way to explain/justify the conceptual leap from “just statistics on steroids” to “AGI” made by LLM stans.
[1] https://arxiv.org/abs/2405.07987
[2] https://arxiv.org/abs/2505.12540v2
[3] https://aiprospects.substack.com/p/llms-and-beyond-all-roads-lead-to
[4] https://www.nature.com/articles/s41586-024-08328-6
If reading this gets you Value Above Replacement, then become a free subscriber to this newsletter. And forward it! And if your VAR from this newsletter is in the three digits or more each year, please become a paid subscriber! I am trying to make you readers—and myself—smarter. Please tell me if I succeed, or how I fail…
#carving-nature-at-the-joints