Paul Kedrosky Presents Us with the ECI—the Epoch Capabilities Index for "AI" Frontier Models: CHART OF THE DAY
When every LLM can closely approximate the typical internet s***poster, the money flows not to the competitive model-builders but to those who can build systems that usefully digest your data to reproduce and improve your SOPs. Or so Satya Nadella claims…
If the appropriate metric is the ECI, and the top model’s edge over the pack really has shrunk from 26% to 6%, we are not in “winner-take-all” land any more. We are are, instead, in commodity-silicon-with-fine-tuning land. That’s where pricing power migrates to whoever can bottle organizational procdures and judgment, and make it portable across interchangeable generalist models.
Narrowing in, as models get better but also closer to the same level at any moment in time:
Paul Kedrosky has a gloss:
Paul Kedrosky: Why .400 Hitters Disappeared—& What It Means for AI <https://paulkedrosky.com/why-400-hitters-disappeared-and-what-it-means-for-ai/>: ‘The trend is linear, not exponential, a steady gain of roughly 16 capability units per period, with an R² of 0.73. The exponential fit is worse than the linear one. Progress that gets told as relentless acceleration is, in the data, a straight line (at best). The second thing… is what happens to the spread…. In the early years of Epoch's Index… a top model could score almost fifty points above the mean…. This is the equivalent of Ted Williams, a .400 hitter…. [But today] the best model's premium over the 90th-percentile model, which was 26% in the early era, is now a mere 6%….
When the variance is wide, being the frontier model meant something…. The distance between frontier and good-enough was wide enough that it looked like an early moat…. This is what maturing commodity markets look like…. Price will become the main differentiator… [with] huge implications…. DeepSeek's massive pricing advantage…. Margin pressure on frontier companies, and the implications for said companies' post-IPO performance…
Ben Thompson <https://stratechery.com/aggregation-theory/> talks about the aggregator flywheel: offer a better value proposition, watch demand flow to you, use that demand to learn about how to offer an even better value proposition to your customers, and watch more demand flow to you until the only reason that your competitors are still around is because you are using market power to jack up your margins. (Note: not your prices, but your margins.) But this requires that economies of current scale and economies of learning-by-doing—cumulative scale—are both now and remain strong. In that world, however, OpenAI took its capabilities lead as of the summer of 2023 and its enormous advantage with respect to the number of users running its models to not have its model recursively self-improve itself but rather to give its programmers the insights they could use to pull further and further ahead of their challengers.
It simply did not happen.
And this is why Satya Nadella of Microsoft is confident that market value will flow to companies that can help users curate their own useful data rather than companies that build frontier models:
Satya Nadella: A Frontier without an Ecosystem Is Not Stable <[twitter.com/satyanade...](https://twitter.com/satyanadella/article/2066182223213293753/)>: ‘We can create a real cognitive loop between people and digital systems…. How [are] organizations [to] continue to learn, build IP, differentiate, and thrive in a world where AI models can continuously absorb the expertise of humans and organizations and commoditize it[?]… Human capital comprises the knowledge, judgment, relationships, ingenuity, and pattern recognition of its people…. Token capital is the firm’s AI capability it builds and owns…. The real opportunity is not in picking the best model but… in building a learning loop… where human capital and token capital compound…. A company should be able to switch out a “generalist” model without losing the “company veteran” expertise built into their learning system. This is the key “test” of your control and sovereignty in the era ahead. Companies need to turn their workflows, domain knowledge, and accumulated judgment into AI systems that improve with each use…. This loop becomes the new IP of the firm…. The last thing any of us want is a world where every company across every sector is ceding value to a few models that eat everything they see…. Employees… [ought to] see their expertise amplified and their judgment become part of systems that make it replicable and scalable and the benefits accrue to the companies and communities around them…
Yes: he is talking his book. But it seems to me that this is a good book to be talking these days. Building bureacracies around knowledge systems of SOPs that are hard to implement and thus to replicate then becomes the defensible IP. Yes, there will be a recursive-improvement loop. But it will be the firm‑specific learning loop, with the underlying model just a swappable, ever‑cheaper input.