By my own, completely arbitrary split of this into 16 predictions, I would say that this hits on maybe 8 of 16 if we are being a bit generous. Which is good, but certainly not "Inevitable" (the title of the book).
With that said, maybe he meant inevitable as N approaches infinity... So there's still time!
Aside: I think the biggest interesting prediction he had was that AI would obey a user driven network effect. It seems like a very clean prediction, and yet it has failed to materialize? I wonder if this has more to do with just how scrape-able the whole internet was or with distilation or what? There is certainly an advantage to having good user data, but it seems like it is largely something that can be worked around (by hook or crook)?
In addition to writing about current (and future) technology, Kevin Kelly also put together Vanishing Asia, a photo book documenting disappearing communities and traditions across Asia
Possibly the best photography book I've come across. It's out of print, but I managed to snag a copy on eBay last year
I would say that KK's "The bulk of AI will be special purpose software brains" has turned out closer to the truth than "The Bitter Lesson" has. LLM's do not drive cars, or vice versa. Stockfish still beats the newest, largest LLM at chess. Etc. Most of the truly productive "AI" is customized for a specific purpose, or class of purposes anyway.
This is a weird point because the Bitter Lesson spends a meaningful chunk of its short length talking about chess specifically.
People sort of generally refer to the Bitter Lesson as if its key argument is "Large Transformer models are best", but this is directly from the essay:
"One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning."
The second paragraph of the essay is specifically about chess:
"In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. At the time, this was looked upon with dismay by the majority of computer-chess researchers who had pursued methods that leveraged human understanding of the special structure of chess. When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers. They said that ``brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. These researchers wanted methods based on human input to win and were disappointed when they did not."
The "brute force search" he's talking about (Deep Blue) used tree search with alpha-beta pruning on top of a ton of human crafted heuristics and a database of game-specific information it could access. Stockfish has historically been an even more "brute force" approach in that it is much more focused on optimizing search--which is one of "arbitrarily scalable" techniques listed in the Bitter Lesson essay--and it has increasingly replaced handwritten human heuristics with neural networks over the life of its development.
At no point in the Bitter Lesson is there a mention of LLMs driving cars, or a suggestion that large transformer models will be the dominant class of model for every single task. It is specifically reflecting on the fact that approaches that leverage search and learning, combined with large amounts of compute, consistently prove to be most effective. Stockfish would qualify as evidence of that, under his criteria.
For now. But the trend remains that bigger, smarter, general purpose AI beats smaller specific purpose models. This is why GPT 5 and 6 remain better at OCR than anything Mistral has put out.
That's sort of a loaded question depending on what you mean by "smaller sub expert models inside".
At the most abstract level, you can observe patterns in a model's activations and theorize that particular collections of components contribute to particular kinds of operations. There are many flavors of this intuition. In that sense, you can think of there being "submodels" within a model, insofar as different parts of the network can participate in specialized computations. Some reading that might interest you in this area would include things like mechanistic interpretability.
There are also Mixture of Expert models, which are a dominant architecture for LLMs now, in which you essentially have a bunch of smaller FFNs that get routed over/aggregated together when the LLM performs a forward pass. It would be a mistake to treat any single expert as a better task specific model though. They are trained as part of an ensemble, and they're almost always being combined with other experts in the forward pass, so they're more like (to use a loose analogy) regions of the brain that tend to correspond more towards particular actions than others. You couldn't just extract one and say "This is now our Chess model that performs as well as the larger LLM we took it from". You could potentially use the relevant experts as a basis for training a smaller task-specific model, and there is work around expert pruning and other distillation techniques, but it's not as simple as "This smaller inner model is a complete task-specific model as is". And more importantly, that ensemble of experts *is* the LLM you're comparing it to, at least in major part. It doesn't really make sense to treat the LLM as some foreign model that calls the expert. The experts themselves don't often correlate to easily interpretable task domains, as they've been learned via joint computation with the other experts and broader LLM.
Then there's tool calling. Today LLMs are trained to make tool calls (basically execute functions) for particular tasks. Those tools can be smaller specialized models, or deterministic code, or anything really that can be fired from a host environment.
If you really want your mind blown look at how accurate Ray Kurzweil was from his 1999 book Age of Spiritual Machines
The most striking thing is that almost all of his missed predictions were missed not because the technology was not available, but because society rejected or didn’t adopt that technology
> but because society rejected or didn’t adopt that technology
OK, so imagine that in 1980, I predicted that we'll colonize Mars by the year 2020. Today, I'm arguing that I was right in my prediction - in broad strokes, we have the technology! Nothing is stopping us. It's just that politicians have different priorities.
Of course, that's a load of bull. You can't decouple predictions about the future of mankind from what said mankind actually decides.
I looked up the predictions on Wikipedia and I'd say that most of them are general enough that they'd hard to disagree with back in 1999 (there will be bioengineering, there will be portable computers, there will be robots, there will be AI), but the timing or the specifics are off on almost every single one of them.
Go to Dan Luu's evaluation (https://danluu.com/futurist-predictions/), search for "Appendix: detailed information on predictions". Seems like Kurzweil's has a huge number of awful predictions. This is from four years ago, so a few things are closer to being real now. But many things he predicted for the early 2000s still haven't arrived.
Another prediction I remember from this book was that AI would be supplied much like electricity as a kind of utility into many different appliances. This is basically how it has turned out with tokens and APIs.
By my own, completely arbitrary split of this into 16 predictions, I would say that this hits on maybe 8 of 16 if we are being a bit generous. Which is good, but certainly not "Inevitable" (the title of the book).
With that said, maybe he meant inevitable as N approaches infinity... So there's still time!
Aside: I think the biggest interesting prediction he had was that AI would obey a user driven network effect. It seems like a very clean prediction, and yet it has failed to materialize? I wonder if this has more to do with just how scrape-able the whole internet was or with distilation or what? There is certainly an advantage to having good user data, but it seems like it is largely something that can be worked around (by hook or crook)?
Is 50% good? It's what you'd expect by posing "will AI do X by 2026?" and then flipping a coin to get an answer.
Google’s main product was and continues to be ads
Are they integrating it well with AI overviews then? It looks like they haven't found a way to monetize it yet.
That is to say, its main product is its human users.
It's a bit nit picky because what most users use is theirnai search , well I'm guessing that's what most of us interact with
They are an ads business my friend - it’s what generate the revenues to cross subsidies the freebies.
Google is a ChatGPT for quick answers, and ChatGPT is a Google for deep web search.
Yes, thank you.
In addition to writing about current (and future) technology, Kevin Kelly also put together Vanishing Asia, a photo book documenting disappearing communities and traditions across Asia
Possibly the best photography book I've come across. It's out of print, but I managed to snag a copy on eBay last year
I would say that KK's "The bulk of AI will be special purpose software brains" has turned out closer to the truth than "The Bitter Lesson" has. LLM's do not drive cars, or vice versa. Stockfish still beats the newest, largest LLM at chess. Etc. Most of the truly productive "AI" is customized for a specific purpose, or class of purposes anyway.
This is a weird point because the Bitter Lesson spends a meaningful chunk of its short length talking about chess specifically.
People sort of generally refer to the Bitter Lesson as if its key argument is "Large Transformer models are best", but this is directly from the essay:
"One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning."
The second paragraph of the essay is specifically about chess:
"In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. At the time, this was looked upon with dismay by the majority of computer-chess researchers who had pursued methods that leveraged human understanding of the special structure of chess. When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers. They said that ``brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. These researchers wanted methods based on human input to win and were disappointed when they did not."
The "brute force search" he's talking about (Deep Blue) used tree search with alpha-beta pruning on top of a ton of human crafted heuristics and a database of game-specific information it could access. Stockfish has historically been an even more "brute force" approach in that it is much more focused on optimizing search--which is one of "arbitrarily scalable" techniques listed in the Bitter Lesson essay--and it has increasingly replaced handwritten human heuristics with neural networks over the life of its development.
At no point in the Bitter Lesson is there a mention of LLMs driving cars, or a suggestion that large transformer models will be the dominant class of model for every single task. It is specifically reflecting on the fact that approaches that leverage search and learning, combined with large amounts of compute, consistently prove to be most effective. Stockfish would qualify as evidence of that, under his criteria.
For now. But the trend remains that bigger, smarter, general purpose AI beats smaller specific purpose models. This is why GPT 5 and 6 remain better at OCR than anything Mistral has put out.
Dont many of the LLMs have smaller sub expert models inside?
That's sort of a loaded question depending on what you mean by "smaller sub expert models inside".
At the most abstract level, you can observe patterns in a model's activations and theorize that particular collections of components contribute to particular kinds of operations. There are many flavors of this intuition. In that sense, you can think of there being "submodels" within a model, insofar as different parts of the network can participate in specialized computations. Some reading that might interest you in this area would include things like mechanistic interpretability.
There are also Mixture of Expert models, which are a dominant architecture for LLMs now, in which you essentially have a bunch of smaller FFNs that get routed over/aggregated together when the LLM performs a forward pass. It would be a mistake to treat any single expert as a better task specific model though. They are trained as part of an ensemble, and they're almost always being combined with other experts in the forward pass, so they're more like (to use a loose analogy) regions of the brain that tend to correspond more towards particular actions than others. You couldn't just extract one and say "This is now our Chess model that performs as well as the larger LLM we took it from". You could potentially use the relevant experts as a basis for training a smaller task-specific model, and there is work around expert pruning and other distillation techniques, but it's not as simple as "This smaller inner model is a complete task-specific model as is". And more importantly, that ensemble of experts *is* the LLM you're comparing it to, at least in major part. It doesn't really make sense to treat the LLM as some foreign model that calls the expert. The experts themselves don't often correlate to easily interpretable task domains, as they've been learned via joint computation with the other experts and broader LLM.
Then there's tool calling. Today LLMs are trained to make tool calls (basically execute functions) for particular tasks. Those tools can be smaller specialized models, or deterministic code, or anything really that can be fired from a host environment.
these are good / accurate, but what's the error-rate? assuming these have been cherry picked, and he made more predictions
If you really want your mind blown look at how accurate Ray Kurzweil was from his 1999 book Age of Spiritual Machines
The most striking thing is that almost all of his missed predictions were missed not because the technology was not available, but because society rejected or didn’t adopt that technology
> but because society rejected or didn’t adopt that technology
OK, so imagine that in 1980, I predicted that we'll colonize Mars by the year 2020. Today, I'm arguing that I was right in my prediction - in broad strokes, we have the technology! Nothing is stopping us. It's just that politicians have different priorities.
Of course, that's a load of bull. You can't decouple predictions about the future of mankind from what said mankind actually decides.
I looked up the predictions on Wikipedia and I'd say that most of them are general enough that they'd hard to disagree with back in 1999 (there will be bioengineering, there will be portable computers, there will be robots, there will be AI), but the timing or the specifics are off on almost every single one of them.
Except, we don't have the technology to even get to Mars. Forget about colonizing. [0]
[0]: https://idlewords.com/2023/1/why_not_mars.htm
Explain what you mean by not being able to get to Mars?
Go to Dan Luu's evaluation (https://danluu.com/futurist-predictions/), search for "Appendix: detailed information on predictions". Seems like Kurzweil's has a huge number of awful predictions. This is from four years ago, so a few things are closer to being real now. But many things he predicted for the early 2000s still haven't arrived.
To be clear I’m not saying he got everything right but if you look at the missed ones, many were/are possible technically
What’s the value of predictions if they’re not good enough to make decisions on?
I think it is largely domain dependent. He's had solid predictions around computing, but for biology (and longevity) he seems to be way off.
For instance, by his prediction, we're already many years past a 100 year average life expectancy (with 120 years expected by the end of the decade).
I think his mistake was tightly coupling exponential compute with exponential progress in biology - which has so far not panned out.
> but because society rejected or didn’t adopt that technology
I would argue that society would likely have been glad to adopt, but the profit-seeking motive got in the way (again).
Actually I think it’s the inverse
I was focused on augmented reality largely in part because it was such a big enabler of all other AI technologies
However people have almost uniformly rejected AR glasses my entire career for social reasons and that is only increasing
To the point where in 2016 I was debating with Jessica Alba and she was firm that they won’t take off cause most people don’t want to wear glasses
I don't like wearing glasses. I should wear my glasses more often, but I don't. But that isn't going to stop me from buying the Even Reality G2
All right, let's see his 2026 predictions for the next ten.
Another prediction I remember from this book was that AI would be supplied much like electricity as a kind of utility into many different appliances. This is basically how it has turned out with tokens and APIs.
OK, but which ones of his predictions did NOT come true? We don’t want survivorship bias do we?
The real survivorship bias is the selection of forecasters.