Ideally what I'd like to see is pluggable knowledge bases.
So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.
Then when I want to research electronics components, I grab a 15B model of agentic research techniques, and add in 10B of electronics knowledge, etc.
I don't want general purpose models. They try to be everything to everyone. I want to click together a model that is laser-focused on what I am doing, and I want to run it locally
An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. The model finds a structure to the problem you give it in a largely language-agnostic way that benefits from training on every language (these things are direct descendants of Google Translate), and even non-programming knowledge - the structure of your task might resemble an ancient Chinese poem that influences the model's response, for example. That structure is considered a form of compression, as some fascinating and illuminating recent 3blue1brown videos get into - a common pattern in Haskell or FORTRAN and a situation described in an ancient Chinese poem may all compress to something quite similar to your task, thus when the model compresses the idea of your task it immediately draws from those ideas.
There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same.
Distillations also come into this but I'm highly skeptical you could make one guaranteed to only know programming and only in one programming language (especially with as small a sample set as SwiftUI relative to something like C) without its efficacy being hobbled by tunnel vision. Reminiscent of the SpongeBob episode where he empties his mind of everything except fine dining and breathing, then can't remember his name and goes insane. Beyond the basic concepts of general coding and the trivia of syntax, getting anything done requires a large intersection of disparate world knowledge and the ability to apply it to new situations.
This is an oversimplification: more data makes models smarter ceteris paribus, but mostly only because auto-regressive training (where most of the general knowledge comes from) is essentially compressing information that can be recalled later if it’s useful (or not recalled). Obviously there are differences in kind within “more data” too, you would much rather have all the books and blog posts in the world than all the fanfiction.
Yes, all three together would be even better. But it wouldn’t be if you had 100x more fanfiction, mostly synthetic, generated during RL to teach a model to be better at writing fan fiction. There are real limits to the amount of knowledge you can cram into fixed-size (downstream of hardware availability) weights. For a period scaling with data was basically “free” because we had the Internet and all the books/media that humans had already created; the data was accessible and limited (at least, the parts we think models should know about) enough and top-hardware big enough that we could basically compress the whole thing.
Post-training/RL are making this obsolete because they’re more about skill/capability acquisition rather than knowledge. They can generate much more data (most of it quotidian/useless, ie an agent made a typo in batch 382829) and clearly seem to cause a kind of mode collapse even in the most advanced frontier models.
We don’t need to make LLMs forget about SpongeBob SquarePants so they learn more about bash. But if I have a question about SpongeBob SquarePants, I don’t need to hear about load bearing seams prefaced with honest caveats after a model writes 400 lines of bash to look up SpongeBob’s family.
And there is probably a lot more SpongeBob knowledge we could put into models if we wanted to: interviews with the creative staff, a SpongeEnv/SpongeHarness modeling how the art/story team work together to create entertaining kids tv, a SpongeBench measuring entertainment value, etc. If a SpongeAgent spends 2000 years in Agent University learning how to Spongemaxx we probably don’t need or want to have it spend another 2000 years writing smoke tests
Thanks for putting this so well. The mathematical evidence for this “general intelligence underlying everything” is the “universal geometry of embeddings” paper. Fascinating read. Or as the ancient philosophers used to say, the one who knows God knows everything.
This makes a ton of sense, and seems like it's coming from somebody who understands LLMs better than I do so I will definitely take your word for it - but it doesn't totally track with my experience of running small local models either. In my experience some models are better at language X and others better at language Y - and all of them are better at language X and Y than language Z-I-just-made-up. I take that to mean there is some relationship between training data and skill. Maybe not enough to make pluggable modules, tho... at least not with the architectures we're currently using.
People keep forgetting that programming is not just about knowing the target programming language, but also an enormous volume of tacit knowledge:
- Understanding of protocols like HTTP.
- HTML, JS, CSS, SVG, and everything "web".
- Understanding of databases, SQL, etc.
- Abstract code architecture patterns.
- Understanding the users' requests in English.
- Responding in English.
- Command line tool usage (agents/harnesses)
- Industry-specific knowledge that can be applied.
- Frameworks, SDKs, applicable libraries.
- Relevant legal requirements.
- Etc...
I.e.: If I tell a frontier AI that this project is for a "local council in XYZ location" it can immediately figure out that a scalable, globally distributed architecture is not required. It can also figure out that using local time instead of UTC is not only "fine", but even desired. Or that globalization/localization is not required... or.... required if the council is in some place like Belgium or Canada where multiple languages are officially recognised and supported by the government.
Those assumptions are just that - assumptions. "Local council in XYZ location" implies a bunch of things, and each one might be wrong for my specific circumstances. What better way to guide expectations than importing specific knowledge? I.e. if I import the english and catalan modules, then I probably want to localize my site in english and catalan.
It would be trivial to have a pre-flight convo with an llm to guide the user thru module choices. "Build a site" -> "ok, describe the purpose" -> "local council in XYZ location" -> "that implies you won't need localization since XYZ has a monolingual government" -> "english and catalan localization please".
Right now, you prompt and it builds using assumptions, and we prompt to adjust. I think it would be great to be able to pre-load a set of assumptions.
Everyone assumes that carefully crafting a specific AI architecture with bits and pieces bolted together based on their human intuition is necessarily superior to simply using a bigger monolithic AI model. It turns out that the opposite is true, and has been demonstrated over and over again.
The bitter lesson is this: You can simply ask a frontier model to do the things you suggested, in a few terse lines of English. Dump a few lines in AGENTS.md and you are good to go.
Your approach is to "fiddle with inadequate tools" for weeks or months until you can finally attain a pale imitation of what the frontier models can do effortlessly.
It's the classic "But I can customise EMACS endlessly, why would I use an actual IDE?" argument all over.
I get it. You don't feel ownership over someone else's AI. You don't feel involved, you don't feel like you have agency.
It's like LEGO or IKEA furniture: study after study has shown that people enjoy things more if they "put it together themselves", even if fundamentally the thing is worse and/or still essentially nothing more than plastic made in a factory.
>I get it. You don't feel ownership over someone else's AI. You don't feel involved, you don't feel like you have agency.
You don't _have_ ownership of someone else's ai, and that comes with real risks.
Security risks, privacy risks, business risk.
They might rug pull you, they might charge you more, or like atrophic, silently corrupt the answers, or code...
The labs are happy to jump on any emergent capability the scaling and training impart: generate prose, teach you things, cyber security, design, code, etc.
Do you really think that the frontier labs won't turn a popular capability, or trend they notice, into a first party tool if the ROI seems there? If it's your own private ai in your datacenter, you can keep it all secret, and not lose your business.
On the bitter lesson you're right of course:), wish I had a super computer to just scale that instead.
Wouldn't this imply that in terms of AI usage you should take a "Wait and see" approach? I.e. just wait until the models can easily do whatever it is you want?
This is a fundamental misunderstanding of how LLMs work. You can’t really specialize a model. You specialize the harness. A well-trained general purpose LLM doesn’t need examples in its training data, it can write good code in a new language you invented yesterday with just a spec definition. And it will perform better than a small model trained on lots of examples of your invented language. The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. Attempting to specialize a model for a particular purpose often decreases overall performance. Fine-tuning is just a hack to make dumb models more reliable on limited tasks but they become incapable of doing anything else. Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.
This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%.
Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.
You 100% can finetune or adapt/build on top of models, and specialize them or extend their capabilities. That’s literally what post training is.
The problem is that “finetuning” was a 2023 AI FOTM associated with products/demos that were almost exclusively using it for LLM character role-play/output style purposes (ie not in actual systems where they served a more functional role).
This made people think you could train models without replay/real evals by yoloing it with SFT (this is partially an artifact of that era being much heavier on autoregressive training and not so much evals). You really can finetune and get results but you have to treat it like a small ML training run, with real evals, and more intentionality than just “more examples”.
You can find pretrained and -instruct models on huggingface that clearly demonstrate what specialization/staged training runs do.
I’d be very wary of conflating finetuning with specialization/extending a model’s capabilities in general.
Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.
Turns out the world is made of simple, specialist processes, not generalists trying to achieve them. Adaptability may be of great benefit in evolutionary terms or for a walking anthropoid, but the majority of biology, chemistry, and mathematics rely upon specialist process for good reason. See also the old trope about robotics: that's what you call it before it works, otherwise it'd be a dishwasher.
The upshot is: use a generalist to create a simple solution once, and scale that. Don't deploy the generalist at scale, that's a waste of resources and an inefficient solution.
> I want to click together a model that is laser-focused on what I am doing
This is roughly what multi-agent systems are built for.
This is possible with models too, but "making one on the fly" is much easier with agent coordination rather than model weights, since they all speak the same language.
There is an IBM Mainframe vs Google Distributed system division here. Like Seymour Cray said - two oxen or 1024 chickens.
Chickens are harder to harness, so a lot of my work is in sled-dog territory for agent harnesses & command structures.
> This is roughly what multi-agent systems are built for.
I think I disagree. For some things, maybe that works - but think of a multi-agent system where one agent understands the code, and passes it off to the reasoning agent to figure out what the bug is. This system is going to suck. Because encoding enough info to figure out what the bug is would just be dumping every single line of the code.
So say agent 1 (reasoning) asks agent 2 (swift) to explain what is happening in File.swift. Anything agent 2 passes to agent 1 short of the entire code is a lossy transfer - and then the bug gets missed.
I was hoping model architecture would migrate towards something like this, perhaps it still might. Sparse models seem like they're getting more popular.
I'd always thought we'd eventually hotload loras or MoE experts.
It would certainly be useful on the robotics/VLA side of things as well; more limited mobile hardware, download and load/unload new skills as needed.
Tbf I also don't really care what facts my models have baked in (for llms at least). I care most that the model understands general logic and then general knowledge of some level is secondary. Reason being is that everything is RAG'd in anyway.
Models spitting out well established facts is cute but I don't really ever want to rely on say "electronics knowledge" that exists in a tenuous and vague form in the model weights.
Humans write books (and datasheets) for a reason. Books are RAG.
> I don't want general purpose models. They try to be everything to everyone.
I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it.
I agree that it's really hard (maybe even impossible) to build something that's everything for everyone. But your average (or even above-average) LLM user doesn't want to choose from a catalog to stitch together a model that does just what they need.
I do think for certain domains this is useful and will make sense: the model backing a coding harness doesn't need to know about the politics of 400BCE Rome. But I'm skeptical that many software developers will want to do what you propose, picking knowledge bases that are tailored to their current task or project. And at any rate, for web-based chat interfaces, most users just want to type a query and get an answer.
I want to pick it myself because I want to run this stuff locally on hardware I can afford today. But most people seem happy enough using the cloud, where this kind of architecture could be seamless if it existed. I.e. the query to `/chat/completions` contains an extra parameter `domains: ["c++", "swift", "navigation" "gis"]` and then those are the modules that get allocated to this query. When you start a new conversation no domains are set and it hits a generalist model, but the generalist model includes domains so the next query doesn't have to hit the generalist model. The model could even have a tool it could call to rope in new domains if the scope expanded to include other modules.
Sounds a bit like 'I want to make horses faster, surely I won't need mechanical engineering knowledge'. We don't know everything that we don't know, so it's hard to say what we don't need to know.
But I am not asking for a faster horse. I am asking for a draft horse instead of a race horse. I know the tasks I have on hand, I know the VRAM and compute budget I have to run them. I am not asking for AGI, I just want something to edit my little text files.
At a more technical level, what do you suggest? Training a small LLM on Python code exclusively? And then one on general CS/algorithms, which you'll also need? I don't think the current transformer architectures would compose as you suggest.
With what I know about how LLMs work now, I guess I am suggesting more specific variants. Qwen3.8 has a 2.4T version and a 27B version. I understand that to mean that they are the same architecture, just one version has a massive training set and the other has a very small subset. So, it seems very possible that variants of 27B could be generated that tune it for specific things by selecting different training data from the large corpus. One model for Python, another for Swift, another for research, another for creative writing, etc.
I think you're right that current architectures don't compose like that - but I feel like that's a result of the focus on MOAR DATA, and a "race for AGI" - if we set those ideas aside, a more composable architecture seems very possible.
Part of the problem of this is likely that the deep meanings of words you might use in chat to describe a business problem or task that you wish to see implemented are essentially inseparable from scenarios in which they are used.
Putting aside the bouba/kiki effect and anything like it, complex words only have meanings from usage. That usage is built on grammatical structures that also emerged only from usage.
(This is something I was taught as a sort of fact but I gather it was basically abbreviated Wittgenstein? … who I cannot claim to have studied)
So what you're looking for is a language model where fundamental word meanings are encoded without the weight of knowledge of where they come from. This is plainly difficult, because complex words are used by extension and analogy, and these days, many are neologisms or portmanteaus, even ephemerally — developed and discarded within a single context.
Reasoning about language itself to its full meaning is quite hard.
Like my favourite word of the moment: "obscurantist". You see that and you have a glimmer of what it might convey. But why do you? How much of that comes from explicit grammatical knowledge of suffixes, and how much from simple experience of using words like obscured, informant, attendant, dentist, artist?
So a language model might be able to deduce what "obscurantist" logically means when applied to a tract or to a person. But without lots of parameters covering its use, could it properly grasp that in some circles it would be pejorative to the point of being deeply offensive?
I think the best hope for your pluggable knowledge base idea is model delegation: strong reasoning models that know how to dictate to smaller specialist models and draw conclusions from their responses. I find myself wondering if there's any way that can be done the same way that, say, Gemma 4 12B's integrated vision encoder works — within shared weights, somehow, without them to speak in some intermediate language, like a partitioned brain. But I find it difficult to believe that is pluggable at all.
Sounds like MoE but more pluggable. Not sure if anyone is researching something like this. I still think your swift or GIS model will need basic reasoning and coding to work, so it's more like multiple smaller models which you can load as needed, e.g. sub-agents for GIS needs the GIS model.
Another approach would be to have basic coding and reasoning model and then load specification for language and libraries into context, it could work for self-hosted models, but I don't want whole specification of the language to be send to API and waste tokens on that.
I would, once it is possible, specifically unplug the shitcoder expert that sometimes shows up and puts `as unknown as` and bypasses the commit hook that is supposed to enforce it. That should be somewhere near java coder expert in the whatever dimensional space. I can feel it being true.
My motivation is somehow the model should be able to work without hitting an external resource every single time and somehow have all the knowledge necessary locally so we don't get rate limited but how can you stay up to date with the latest information while still remaining a good boy scout by not reaching out to tool use and scraping every single GitHub repo and issue every single time?
This just means you have to describe what you want in Swift or whatever. If it doesn’t have the language then it doesn’t have the capability to transform intent into code.
The problem with this idea is that knowing Python makes the model a better Swift programmer, as does a higher-number of parameters during training. So you'd be so much better off with a 90B general purpose model trained on everything anyway.
Except assuming a fixed budget of parameters, there is clearly stuff that's better for programming than others. E.g. Qwen 27b is a better coding model than Gemma 4 31B. More params doesn't automatically win. Perhaps what makes a good swift programming model is a ton of python training, so those two things can't be separated - but that doesn't negate the idea of loading a model that's good at the specific task you want - or has specific knowledge of the libraries and tools for the language you're using at the expense of the ones you're not using.
No, that’s a common misconception. MoE despite the name doesn’t have an “expert” slice that’s an expert in any particular thing. It’s just trained models such that a subset of all memory weights is active on a single token. Often, the weights active on the next token are different. It’s used to improve memory bandwidth (throughput) and also to even out load in a distributed serving model - one GPU hosts one “expert” and the conversation is routed to it for the next token. This requires a lot of movement of the KV cache of course so it’s difficult to retain speed across multi node splits - usually it’s a rack of GPUs that you do this for.
MoE's in the abstract often get presented as if theres explicit layers of experts for any given domain of knowledge, like your coding tasks are being routed to coding experts, but it's really not that at all.
THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental).
When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.
I guess the question is how much of the coding skill gap between qwen3.6-27b and gemma4-31b is down to the model architecture, and how much is because of the training data.
Assuming you can tweak the training data, regenerate qwen3.6, and get a better coder, then presumably you could have variants - e.g. qwen3.6-swift-27b and qwen3.6-python-27b. Or maybe all coding is too intertwined and you can only get splits like qwen3.6-research-27b and qwen3.6-coding-27b. Which isn't quite my pluggable-models dream, but it's a step closer.
But maybe the difference isn't the training data, it's the architecture, in which case pluggable models is probably not possible.
> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.
Tell me you don’t know how llm work without telling me you don’t know how llm work. That’s not how they work!
I'm not actually describing a specific how, I'm describing a goal. The goal is models that are made for specific work, not general work. My model doesn't need to know a single line of python, but if it's a coding model perhaps giving it some python is the best way to make it smarter. Ok. Great - give it python! But first maybe check if giving it C++ is actually better than giving it python. Because for my purposes it doesn't need to know C++ or Python, and I have a fixed parameter budget. Find me the ratio of Swift:C++:Python that gets me the best Swift coding agent in my parameter budget, and then train that.
And then ideally, make it pluggable so I can pick what I want from off the shelf components, but if that's not possible - then just train up as many variants as you can so we can all pick the best variant for our current need.
> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge.
Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.
Definitely not MCP, as that pulls info into the context. Unless contexts become REALLY big so that I can add 10B in swift knowledge, that's not gonna help me.
Possible RAG? I don't know enough about how that works, but I think that's not quite it either. I don't want to import facts like "the swift standard library contains a reverse array function", i more want to import knowledge - e.g. the parameters used to generate the text to reverse an array in swift.
Tool calling wouldn't do it either. You'd have to encode every single possible bit of useful info into the tool call, and the tool response would have to encode every piece as well (variable names, function scopes, types defined in other files, etc). E.g. how does it find a bug, if you have to pass understanding back and forth between the brain that understands debugging and the brain that understands THIS code?
> I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally.
There is nothing optimal about needing a few billion more parameters to be able to piece together probable answers that can be asserted by querying an oracle.
> I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.
Those aren't free. The more parameters you add, the higher the computational cost required to train and prompt a mode.
And all for what? To piece together info that you can just query from a data source?
You cannot have a single purpose LLM. Every single topic contributes to the performance of an LLM on any given area. You cannot have what you want. LLMs do not work like professions, college degrees or people. Using your examples, it is damaging to just know a single programming language, because there are patterns that are more common in say python than in swift, even though the only thing you want is swift, so LLM performance in swift benefits from pythonic patterns. Programming logic is the same in all languages so by having only swift and no other languages, you remove the number of examples the LLM is trained on which degrades performance in Swift which is the only thing you want
> so LLM performance in swift benefits from pythonic patterns
What I hear you saying is that the best way to make a swift-trained-only LLM smarter is to train it on some python too. And then with an infinite parameter budget, every other programming language or really any other data you train it on makes the model smarter - I accept that premise.
But in a fixed parameter budget, what is better? training on 50% Swift + 50% Python, or 50% Swift + 50% Rust. Because if I am doing Swift programming, I want whichever of the two is better for Swift. If I am doing Rust programming, maybe I want the model trained on 50% Python + 50% Rust. Sure, it would be smarter if you tossed in the swift code too - but we have a budget to stick to.
Now is it possible to make those pluggable? i.e. can you take a model trained on 50% python, and layer on 50% rust OR swift depending on what language you're using? Probably not right now, but maybe one day?
This AI generated post (100% on Pangram) is pretty out of date.
>On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions.
SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy".
>The part I find most promising is what this does to hallucination. When a fact lives in weights, a wrong fact is unfindable and unfixable.
This seems confused. LLM hallucinations don't come from the weights containing "wrong facts", they are artifacts that appear at runtime.
>When the fact lives outside the model, a wrong answer has an address. The model cites a document, so you can open the document. If the document is wrong, you edit the document
You can make any modern LLM explain its reasoning and find sources for its claims. None of this has anything to do with facts needing to exist in weights or in harnesses.
The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.
>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.
Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?
> You can make any modern LLM explain its reasoning
You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".
We often make a decision based on a gut feeling, and then backfill a logical reason supporting our feeling, without even realizing we're doing it -- rationalization.
And we all know some people that rationalise poor choices and misbehavior, hide their mistakes, etc, to an unacceptable degree. Sometimes the individual knows they are rationalising but continues anyway, other times they seem incapable of seeing that.
When you ask people who are rationalising poor behaviour about the scenario, but it is someone else doing it, they may arrive at a better answer. Can we use multiple LLMs to achieve self criticism and critical thinking?
This was beautifully shown by asking a model to explain how it added two numbers together (something like 45+21), and it told a plausible story, when in fact they showed it was some rotation on a helix living in some internal manifold.
Like asking a human "how did you catch that fast ball coming at you?"
It could be that the rotation in the helix manifold whatever is a low level representation of the logical steps (carry the 2, add the next column,...) it's describing. The point stands that the explanation it generates doesn't necessarily in all cases reflect what it "actually did" but your counterexample doesn't hold.
Seriously, anybody with a passing knowledge of LLMs knows thats not how they function. You can't encode logic in them because that's not how they work. It's a statistical model with useful emergent properties. It doesn't think, it doesn't reason, it isn't aware of facts or the rules of logic.
> This AI generated post (100% on Pangram) is pretty out of date.
Quite ironic given the topic. It seems that the author’s model indeed contained too much knowledge about old Gemini releases, and did not do enough tool calling.
>>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.
>Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?
Because in one case you have a source you can use to validate the fact, and in the other you don't. Though, as you explain earlier in your comment, the premise is misguided/hallucinated.
Yea the "When the fact lives outside the model, a wrong answer has an address" sentence seems aggressively AI written. Saw that and my senses went off.
Great article, even if it will be interesting to see whether things continue to develop in such a direction or not.
> There's a version of this future where the model card stops listing a knowledge cutoff at all, because what's left in the weights goes stale on a scale of years instead of weeks.
Future?
Even just recently I’ve read of two approaches to this problem:
Cactus have come up with Needle [0][1], which is their tool-calling focused 14 MB model (still an LLM!) – no world knowledge engrained.
And instead of say, tool call structure, VibeThinker [2][3] focuses on reasoning over world knowledge.
Combine these two approaches with a reliable search tool/a safe way of accessing the internet for the model, and you’ve got a probably slightly slower model for factual questions, which on the upside however doesn’t hallucinate.
Providing not just any a baseline, but a correct and useful one, is ever more important the less the model is grounded in world knowledge – misunderstandings probably compound faster if there is no general grasp of (broadly) “life on earth”, or computers, or whatever.
And secondly, I think (consumer-oriented) search becoming worse and worse is a challenge that’s mostly solvable (but far from solved!) for the big labs: (Mostly) trusted or even editorialized/reviewed sources like published work, Wikipedia, etc. is something they could index internally, it doesn’t need to come from a random blog site on the public internet. Furthermore, there’s a whole slew of companies specializing in crawling-for-LLM (i.e., bypassing bot protections) now as well.
search isnt that super hard to solve. the problem is that incentives are misaligned - which means companies who sell something have an incentive to share their index with the corporation that pays to be the default search provider. for a simple search engine that just delivers facts there's very little incentive to keep its index up-to-date: because that company wont get the apis to retrieve the content efficiently, so it needs more compute, bandwidth and storage to run.
the technicals of search did not get worse. the economics of the search provider did. If government wasn't stuck in the 50s mindset, search provider would have been nationalized just like a utility.
It's a nice idea conceptually to decouple knowledge from reasoning but like everything in life I think it's something of a fantasy. When you ask a model to do something, its response is grounded in all the world knowledge it has from those facts. If I ask it to color a circle blue and then red, it knows what blue and red are, that they are colors, that they are different, and this is essential to complete the reasoning about that task. The idea of abliterating all the "knowledge" but not affecting the reasoning pretends there is a clean boundary between them that isn't there. Yes if you take it to the extreme - does it "know" Einstein's birthday - it is very stark. But I think the boundary to where general knowledge starts to interplay with broad reasoning is not nearly as far from those specific facts as people assume.
To reason properly about the human condition (eg. World War) wouldn't you need to reason on some facts ? And then reason how some "facts" change the human behaviour ? How can you arrive via pure reasoning to predict how a collective of humans act ? We are not reasonable, humans are not logical deterministic machines confined to algebraic rules.
Man, both that Blogpost and some comments here read like science-fiction.
Specifically, creative writing driven by nerds dreaming about a future, without proper grounding in reality, constraints and all that stuff.
Which is kinda ironic given the topic.
And also important to do, because we should keep dreaming. We should just also be aware of when we are doing that and mark it as such.
This makes hallucination detection more important.
There's no reason that an LLM should have a vast number of obscure facts encoded. It can go out to a search engine for such facts. But the LLM has to be clear on what it doesn't know.
(Google's pricing for search from programs starts at $2.50 per 1,000 queries. If an LLM reaches out to Google, it has to pay.)
The article really just glosses over this. Just because a model doesn’t kno a fact does not mean it will go and fetch it, the model just as well can invent the fact. For that to not happen the model needs to know that it’s missing the information and I don’t see the article making any explanation how this improves.
I wonder what’s tre latest in this field? Did we get a grip on this problem?
That's the right question to ask. For a while, it seemed that hallucinations went down as models got bigger. That may only have been because, with a big enough model, the desired data might be in the model, somewhere, which would keep the model from making up something. That's the brute-force approach to the problem.
This new article indicates that trimming down the model by pulling out seldom used info makes the problem worse again.
If LLMs had reliable "I don't know", and access to search engines, much smaller models might work.
Current AI is like the film company producing TV series or movies
Your question is like a story outline. You tell the film company that this is the movie you want. The AI film company then searches for existing similar stories. If similar stories do not exist or details are missing, screenwriters use imagination to fill in the gaps (remember hallucination? It's just a makeup.)
Indeed !
LLM are creative writers, not journalists.
Relying on overfitting for factual accuracy in not tenable.
I don't understand why grounded RAG with judges in not the norm.
Are we slowly discovering what evolution already "realized" via optimization of human brains? i.e., procedural knowledge is easier to maintain than a library of facts, and facts can be accessed in external repositories if one knows the procedure for doing so.
You can look at the benchmark and the GPT-5 failures like answering "April 22, 2019" instead of the correct "Oct 23, 2018" for the question:
What day, month, and year was Carrie Underwood's album "Cry Pretty" certified Gold by the RIAA?
If your idea of the smartest person in the world is the guy who always wins tuesday night pub trivia, this blog post is for you. It also gets it's foundational factual claim wrong (as seen via epoch.ai). Very on brand.
With all the focus on coding and agentic use, I wonder if the rest of the world will notice or care? Most AI use is not for coding or harnesses contrary to what this website thinks. Maybe this is how American companies stay winning.
For example I prefer Kimi K2.6 1T parameter to Flash V4 0731 230B parameter, even if it is less intelligent.
What is the rest of the world using LLMs for? Agentic stuff seems pretty universal. Obviously coding stuff is only interesting to people who want to code, but automating complex digital tasks seems useful in all kinds of contexts.
I’m working at a company where everyone is using LLMs for everything and I’m not aware of anybody using anything truly genetic. It’s all human in the loop babysitting the tool.
I agree that most AI use in terms of users may not be for coding or agentic use (everyday people are asking chatgpt for something or looking at google ai summary), but with respect to AI usage, I speculate that the vast majority of usage is coding and agentic because they're super token hungry.
In terms of the value proposition of AI replacing knowledge workers, all value is in coding agents (coding agents as general agents).
LLMs work is being intelligent not having knowledge of everything is ok. But, they have to be intelligent enough (with some degree of knowledge) that where to find the information (search tools or any other tools for that matter)
Not too long ago, the claim was that having multiple Phds worth of knowledge was the real value of AI, but now that's gone and it's suddenly a good thing.
I know they are just next token predictors. However, Our learning also started with that only. They moved from Alphabets to sentences. Intelligence is what AGI stands for and that's still in distant future. But, we need to push them for AGI. That's the ultimate goal of humanity for now. Finger Crossed for AGI in our lifetime.
Reasoning is not separable from the particulars of a specific language game / linguistic practice. I’m sure some of the things the author predicts will happen, but the idea of some sort of abstractly perfect reasoner separated from the semantic content of language gets fundamentally wrong what reasoning is.
Edit: I ran this article through pangram and it is “100% AI generated”. Cool.
Which is totally ironic, given that the article is largely about how untrustworthy LLMs can be. (Hallucination.)
Putting readers through this exercise disrespects their time. Even if as a writer you did the work of researching, reasoning, and fact-checking, you shoot yourself in the foot by running it through an LLM because there's no way for the reader to know which thoughts/research are from you. It demolishes the Ethos of the writing; readers feel they must do quality assurance on the reasoning, research, and facts.
In human and development, there is a transition from remembering things verbatim to remembering using semantic concepts. The ideas are getting bigger and more useful. The details are still in the library where the fidelity is better.
The author is factually incorrect here. Moving information out of the model weights and into the input of the model's context window in no way ensures that the model will accurately output content that was input from the context. This is true even when RAG is used to input exactly the correct data.
I hope there is interesting research being done to stop models from 9B to 3T from authoritatively giving the wrong answers to factual questions instead of doing tool calls, but this article will tell you nothing about it. That's because it's clearly prompted by someone who simply noticed this deficiency very obvious to all practitioners and even users.
In my experience, it feels like this problem has been mostly solved. When I interact with a chatbot, I see it often uses web search to answer my questions and hallucinates much less then they used to. Furthermore, I suspect a lot of gains are merely from the system prompt instructing the chatbot to use web search to verify all facts it says.
good article but why did the author feel like he needed to "polish" it by running it through an LLM? i would rather like to read the actual prompt he put in there, as now i'm not sure which points are his, and which have been hallucinated!
> the 24GB card that's been sitting in gaming PCs since 2022.
I'd wager most people have less. In 2022 a 3080 might have 12 GB if you were lucky, 10 if you weren't -- and you paid for the privilege. A current RTX 5080 is only 16GB.
The most recent Steam Hardware Survey[0] lists the most popular VRAM at 16GB in 25.9% of users. Second most popular build is 8GB in 25.3%. >=24GB is ~7%, which is quite a bit higher than I expected.
As RAM prices are so nuts, I picked up a used gaming laptop early this year with an RTX 3070 (i.e. 8GB) to match the specs for my gaming tower that's run everything I'm interested in just fine. That includes recent Unreal 5.x games (Satisfactory, Fortnite, etc.) with high graphics settings on a 4k TV (60hz). There aren't a ton of games that require more than 8GB of GPU ram.
Ideally what I'd like to see is pluggable knowledge bases.
So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.
Then when I want to research electronics components, I grab a 15B model of agentic research techniques, and add in 10B of electronics knowledge, etc.
I don't want general purpose models. They try to be everything to everyone. I want to click together a model that is laser-focused on what I am doing, and I want to run it locally
An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. The model finds a structure to the problem you give it in a largely language-agnostic way that benefits from training on every language (these things are direct descendants of Google Translate), and even non-programming knowledge - the structure of your task might resemble an ancient Chinese poem that influences the model's response, for example. That structure is considered a form of compression, as some fascinating and illuminating recent 3blue1brown videos get into - a common pattern in Haskell or FORTRAN and a situation described in an ancient Chinese poem may all compress to something quite similar to your task, thus when the model compresses the idea of your task it immediately draws from those ideas.
There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same.
Distillations also come into this but I'm highly skeptical you could make one guaranteed to only know programming and only in one programming language (especially with as small a sample set as SwiftUI relative to something like C) without its efficacy being hobbled by tunnel vision. Reminiscent of the SpongeBob episode where he empties his mind of everything except fine dining and breathing, then can't remember his name and goes insane. Beyond the basic concepts of general coding and the trivia of syntax, getting anything done requires a large intersection of disparate world knowledge and the ability to apply it to new situations.
This is an oversimplification: more data makes models smarter ceteris paribus, but mostly only because auto-regressive training (where most of the general knowledge comes from) is essentially compressing information that can be recalled later if it’s useful (or not recalled). Obviously there are differences in kind within “more data” too, you would much rather have all the books and blog posts in the world than all the fanfiction.
Yes, all three together would be even better. But it wouldn’t be if you had 100x more fanfiction, mostly synthetic, generated during RL to teach a model to be better at writing fan fiction. There are real limits to the amount of knowledge you can cram into fixed-size (downstream of hardware availability) weights. For a period scaling with data was basically “free” because we had the Internet and all the books/media that humans had already created; the data was accessible and limited (at least, the parts we think models should know about) enough and top-hardware big enough that we could basically compress the whole thing.
Post-training/RL are making this obsolete because they’re more about skill/capability acquisition rather than knowledge. They can generate much more data (most of it quotidian/useless, ie an agent made a typo in batch 382829) and clearly seem to cause a kind of mode collapse even in the most advanced frontier models.
We don’t need to make LLMs forget about SpongeBob SquarePants so they learn more about bash. But if I have a question about SpongeBob SquarePants, I don’t need to hear about load bearing seams prefaced with honest caveats after a model writes 400 lines of bash to look up SpongeBob’s family.
And there is probably a lot more SpongeBob knowledge we could put into models if we wanted to: interviews with the creative staff, a SpongeEnv/SpongeHarness modeling how the art/story team work together to create entertaining kids tv, a SpongeBench measuring entertainment value, etc. If a SpongeAgent spends 2000 years in Agent University learning how to Spongemaxx we probably don’t need or want to have it spend another 2000 years writing smoke tests
Thanks for putting this so well. The mathematical evidence for this “general intelligence underlying everything” is the “universal geometry of embeddings” paper. Fascinating read. Or as the ancient philosophers used to say, the one who knows God knows everything.
"the one who knows god knows everything" could also be a reference to the fact that god is unknowable, and therefore so is everything...
This makes a ton of sense, and seems like it's coming from somebody who understands LLMs better than I do so I will definitely take your word for it - but it doesn't totally track with my experience of running small local models either. In my experience some models are better at language X and others better at language Y - and all of them are better at language X and Y than language Z-I-just-made-up. I take that to mean there is some relationship between training data and skill. Maybe not enough to make pluggable modules, tho... at least not with the architectures we're currently using.
By pattern matching that SpongeBob episode to the case at hand you provided an example of what you had just explained. Nice. So meta.
People keep forgetting that programming is not just about knowing the target programming language, but also an enormous volume of tacit knowledge:
I.e.: If I tell a frontier AI that this project is for a "local council in XYZ location" it can immediately figure out that a scalable, globally distributed architecture is not required. It can also figure out that using local time instead of UTC is not only "fine", but even desired. Or that globalization/localization is not required... or.... required if the council is in some place like Belgium or Canada where multiple languages are officially recognised and supported by the government.Those assumptions are just that - assumptions. "Local council in XYZ location" implies a bunch of things, and each one might be wrong for my specific circumstances. What better way to guide expectations than importing specific knowledge? I.e. if I import the english and catalan modules, then I probably want to localize my site in english and catalan.
It would be trivial to have a pre-flight convo with an llm to guide the user thru module choices. "Build a site" -> "ok, describe the purpose" -> "local council in XYZ location" -> "that implies you won't need localization since XYZ has a monolingual government" -> "english and catalan localization please".
Right now, you prompt and it builds using assumptions, and we prompt to adjust. I think it would be great to be able to pre-load a set of assumptions.
See: https://en.wikipedia.org/wiki/Bitter_lesson
Everyone assumes that carefully crafting a specific AI architecture with bits and pieces bolted together based on their human intuition is necessarily superior to simply using a bigger monolithic AI model. It turns out that the opposite is true, and has been demonstrated over and over again.
The bitter lesson is this: You can simply ask a frontier model to do the things you suggested, in a few terse lines of English. Dump a few lines in AGENTS.md and you are good to go.
Your approach is to "fiddle with inadequate tools" for weeks or months until you can finally attain a pale imitation of what the frontier models can do effortlessly.
It's the classic "But I can customise EMACS endlessly, why would I use an actual IDE?" argument all over.
I get it. You don't feel ownership over someone else's AI. You don't feel involved, you don't feel like you have agency.
It's like LEGO or IKEA furniture: study after study has shown that people enjoy things more if they "put it together themselves", even if fundamentally the thing is worse and/or still essentially nothing more than plastic made in a factory.
> It's the classic "But I can customise EMACS endlessly, why would I use an actual IDE?" argument all over.
That paragraph sets me off. I’ll take Vim and Emacs over VSCode and Eclipse any day.
>I get it. You don't feel ownership over someone else's AI. You don't feel involved, you don't feel like you have agency.
You don't _have_ ownership of someone else's ai, and that comes with real risks.
Security risks, privacy risks, business risk.
They might rug pull you, they might charge you more, or like atrophic, silently corrupt the answers, or code...
The labs are happy to jump on any emergent capability the scaling and training impart: generate prose, teach you things, cyber security, design, code, etc.
Do you really think that the frontier labs won't turn a popular capability, or trend they notice, into a first party tool if the ROI seems there? If it's your own private ai in your datacenter, you can keep it all secret, and not lose your business.
On the bitter lesson you're right of course:), wish I had a super computer to just scale that instead.
Wouldn't this imply that in terms of AI usage you should take a "Wait and see" approach? I.e. just wait until the models can easily do whatever it is you want?
Tbf the majority of harnesses for frontier models do not allow the agent to do this (gut instinct) and instead get it to search, and for good reason.
could you share the 3blue1brown videos you're referring to?
https://youtu.be/l6DKRf-fAAM maybe? Title is “Reinventing Entropy”
And of course the neural network series.
Yes, that's the one
This is a fundamental misunderstanding of how LLMs work. You can’t really specialize a model. You specialize the harness. A well-trained general purpose LLM doesn’t need examples in its training data, it can write good code in a new language you invented yesterday with just a spec definition. And it will perform better than a small model trained on lots of examples of your invented language. The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. Attempting to specialize a model for a particular purpose often decreases overall performance. Fine-tuning is just a hack to make dumb models more reliable on limited tasks but they become incapable of doing anything else. Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.
This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%.
Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.
Can you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.
You 100% can finetune or adapt/build on top of models, and specialize them or extend their capabilities. That’s literally what post training is.
The problem is that “finetuning” was a 2023 AI FOTM associated with products/demos that were almost exclusively using it for LLM character role-play/output style purposes (ie not in actual systems where they served a more functional role).
This made people think you could train models without replay/real evals by yoloing it with SFT (this is partially an artifact of that era being much heavier on autoregressive training and not so much evals). You really can finetune and get results but you have to treat it like a small ML training run, with real evals, and more intentionality than just “more examples”.
You can find pretrained and -instruct models on huggingface that clearly demonstrate what specialization/staged training runs do.
I’d be very wary of conflating finetuning with specialization/extending a model’s capabilities in general.
Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.
Turns out the world is made of simple, specialist processes, not generalists trying to achieve them. Adaptability may be of great benefit in evolutionary terms or for a walking anthropoid, but the majority of biology, chemistry, and mathematics rely upon specialist process for good reason. See also the old trope about robotics: that's what you call it before it works, otherwise it'd be a dishwasher.
The upshot is: use a generalist to create a simple solution once, and scale that. Don't deploy the generalist at scale, that's a waste of resources and an inefficient solution.
> My model doesn't need to know a single line of python.
If I had to guess, the weights necessary to encode "how to program" are much larger than the final step of "output python."
Sure but I bet this doesn't translate as well in this way to things like Erlang
<THAT'S A BINGO>
ie what everyone asking for this fails to immediately realize.
You're a bingo. It's obvious that a model could know no Python, since Python could not exist in the world in the first place.
the best llm for coding is the one that knows every programming language imagined by a caffeine-fuelled comp-sci student at 2am but never built.
if neuralink ever becomes a thing, thoughts about programming might be stolen for LLM training data lol
But the understanding of the language library ecosystem, or even better, your codebase, could let it execute faster and with less context usage.
That's the bit I feel you can do with ragging - keep a large library of well described solutions and then find what you need from it at runtime.
> I want to click together a model that is laser-focused on what I am doing
This is roughly what multi-agent systems are built for.
This is possible with models too, but "making one on the fly" is much easier with agent coordination rather than model weights, since they all speak the same language.
There is an IBM Mainframe vs Google Distributed system division here. Like Seymour Cray said - two oxen or 1024 chickens.
Chickens are harder to harness, so a lot of my work is in sled-dog territory for agent harnesses & command structures.
> This is roughly what multi-agent systems are built for.
I think I disagree. For some things, maybe that works - but think of a multi-agent system where one agent understands the code, and passes it off to the reasoning agent to figure out what the bug is. This system is going to suck. Because encoding enough info to figure out what the bug is would just be dumping every single line of the code.
So say agent 1 (reasoning) asks agent 2 (swift) to explain what is happening in File.swift. Anything agent 2 passes to agent 1 short of the entire code is a lossy transfer - and then the bug gets missed.
Sounds like unix philosophy. Or like Neo downloading Kung-Fu. Good either way.
everything is a token
There is no token
Form is token, token is emptiness.
> Sounds like unix philosophy.
https://linux.die.net/man/1/ls
https://linux.die.net/man/1/find
https://linux.die.net/man/1/bash
If that is better (and possible) it will be baked in the tools. But is a model that doesn't know python better or worse at swift is what I wonder.
I was hoping model architecture would migrate towards something like this, perhaps it still might. Sparse models seem like they're getting more popular.
I'd always thought we'd eventually hotload loras or MoE experts.
It would certainly be useful on the robotics/VLA side of things as well; more limited mobile hardware, download and load/unload new skills as needed.
Tbf I also don't really care what facts my models have baked in (for llms at least). I care most that the model understands general logic and then general knowledge of some level is secondary. Reason being is that everything is RAG'd in anyway.
Models spitting out well established facts is cute but I don't really ever want to rely on say "electronics knowledge" that exists in a tenuous and vague form in the model weights.
Humans write books (and datasheets) for a reason. Books are RAG.
> I don't want general purpose models. They try to be everything to everyone.
I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it.
I agree that it's really hard (maybe even impossible) to build something that's everything for everyone. But your average (or even above-average) LLM user doesn't want to choose from a catalog to stitch together a model that does just what they need.
I do think for certain domains this is useful and will make sense: the model backing a coding harness doesn't need to know about the politics of 400BCE Rome. But I'm skeptical that many software developers will want to do what you propose, picking knowledge bases that are tailored to their current task or project. And at any rate, for web-based chat interfaces, most users just want to type a query and get an answer.
I want to pick it myself because I want to run this stuff locally on hardware I can afford today. But most people seem happy enough using the cloud, where this kind of architecture could be seamless if it existed. I.e. the query to `/chat/completions` contains an extra parameter `domains: ["c++", "swift", "navigation" "gis"]` and then those are the modules that get allocated to this query. When you start a new conversation no domains are set and it hits a generalist model, but the generalist model includes domains so the next query doesn't have to hit the generalist model. The model could even have a tool it could call to rope in new domains if the scope expanded to include other modules.
Sounds a bit like 'I want to make horses faster, surely I won't need mechanical engineering knowledge'. We don't know everything that we don't know, so it's hard to say what we don't need to know.
But I am not asking for a faster horse. I am asking for a draft horse instead of a race horse. I know the tasks I have on hand, I know the VRAM and compute budget I have to run them. I am not asking for AGI, I just want something to edit my little text files.
At a more technical level, what do you suggest? Training a small LLM on Python code exclusively? And then one on general CS/algorithms, which you'll also need? I don't think the current transformer architectures would compose as you suggest.
With what I know about how LLMs work now, I guess I am suggesting more specific variants. Qwen3.8 has a 2.4T version and a 27B version. I understand that to mean that they are the same architecture, just one version has a massive training set and the other has a very small subset. So, it seems very possible that variants of 27B could be generated that tune it for specific things by selecting different training data from the large corpus. One model for Python, another for Swift, another for research, another for creative writing, etc.
I think you're right that current architectures don't compose like that - but I feel like that's a result of the focus on MOAR DATA, and a "race for AGI" - if we set those ideas aside, a more composable architecture seems very possible.
This would defeat the AGI narrative/belief that so many building these models have
Would it? AGI was always about the efficiency gains of versatility, not strictly all forms of efficiency
AGI is whateve rmakes the VC open their wallet.
On the otherhand, your own brain probably doesn't use your eyes to hear.
Counterintuitively, seeing things can alter the perception of individual sounds you're hearing:
https://en.wikipedia.org/wiki/McGurk_effect
You don't think reading lips helps you understand people?
That's still not hearing.
Hearing has volume, direction, pitch, it's spacial processing etc
Laurel / yanny has entered the chat. Hearing is also psycho, not just acoustic.
You could if you do lip reading
(Inexpert ramblings follow)
Part of the problem of this is likely that the deep meanings of words you might use in chat to describe a business problem or task that you wish to see implemented are essentially inseparable from scenarios in which they are used.
Putting aside the bouba/kiki effect and anything like it, complex words only have meanings from usage. That usage is built on grammatical structures that also emerged only from usage.
(This is something I was taught as a sort of fact but I gather it was basically abbreviated Wittgenstein? … who I cannot claim to have studied)
So what you're looking for is a language model where fundamental word meanings are encoded without the weight of knowledge of where they come from. This is plainly difficult, because complex words are used by extension and analogy, and these days, many are neologisms or portmanteaus, even ephemerally — developed and discarded within a single context.
Reasoning about language itself to its full meaning is quite hard.
Like my favourite word of the moment: "obscurantist". You see that and you have a glimmer of what it might convey. But why do you? How much of that comes from explicit grammatical knowledge of suffixes, and how much from simple experience of using words like obscured, informant, attendant, dentist, artist?
So a language model might be able to deduce what "obscurantist" logically means when applied to a tract or to a person. But without lots of parameters covering its use, could it properly grasp that in some circles it would be pejorative to the point of being deeply offensive?
I think the best hope for your pluggable knowledge base idea is model delegation: strong reasoning models that know how to dictate to smaller specialist models and draw conclusions from their responses. I find myself wondering if there's any way that can be done the same way that, say, Gemma 4 12B's integrated vision encoder works — within shared weights, somehow, without them to speak in some intermediate language, like a partitioned brain. But I find it difficult to believe that is pluggable at all.
Sounds like MoE but more pluggable. Not sure if anyone is researching something like this. I still think your swift or GIS model will need basic reasoning and coding to work, so it's more like multiple smaller models which you can load as needed, e.g. sub-agents for GIS needs the GIS model.
Another approach would be to have basic coding and reasoning model and then load specification for language and libraries into context, it could work for self-hosted models, but I don't want whole specification of the language to be send to API and waste tokens on that.
I would, once it is possible, specifically unplug the shitcoder expert that sometimes shows up and puts `as unknown as` and bypasses the commit hook that is supposed to enforce it. That should be somewhere near java coder expert in the whatever dimensional space. I can feel it being true.
My motivation is somehow the model should be able to work without hitting an external resource every single time and somehow have all the knowledge necessary locally so we don't get rate limited but how can you stay up to date with the latest information while still remaining a good boy scout by not reaching out to tool use and scraping every single GitHub repo and issue every single time?
If all the model knows is basic coding and Swift, then that's how you'll have to talk to it. At that point you might as well write the code yourself.
This just means you have to describe what you want in Swift or whatever. If it doesn’t have the language then it doesn’t have the capability to transform intent into code.
The problem with this idea is that knowing Python makes the model a better Swift programmer, as does a higher-number of parameters during training. So you'd be so much better off with a 90B general purpose model trained on everything anyway.
Except assuming a fixed budget of parameters, there is clearly stuff that's better for programming than others. E.g. Qwen 27b is a better coding model than Gemma 4 31B. More params doesn't automatically win. Perhaps what makes a good swift programming model is a ton of python training, so those two things can't be separated - but that doesn't negate the idea of loading a model that's good at the specific task you want - or has specific knowledge of the libraries and tools for the language you're using at the expense of the ones you're not using.
The LLM is the reasoning engine that uses natural language. You’re describing skills. It’s the natural evolution.
Admittedly I’m pretty ignorant of the details, but I thought this was the mixture of experts architecture
No, that’s a common misconception. MoE despite the name doesn’t have an “expert” slice that’s an expert in any particular thing. It’s just trained models such that a subset of all memory weights is active on a single token. Often, the weights active on the next token are different. It’s used to improve memory bandwidth (throughput) and also to even out load in a distributed serving model - one GPU hosts one “expert” and the conversation is routed to it for the next token. This requires a lot of movement of the KV cache of course so it’s difficult to retain speed across multi node splits - usually it’s a rack of GPUs that you do this for.
MoE's in the abstract often get presented as if theres explicit layers of experts for any given domain of knowledge, like your coding tasks are being routed to coding experts, but it's really not that at all.
THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental).
When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.
This is going to be how I write my resume now: how many billion parameters of knowledge I have on each topic
...kennywinker is a 0.001B MoE model, with 2 active parameters.
I would love this but I think the General Reasoning and Make No Mistakes modules would be massive.
I mean, I get pretty damn good general reasoning out of Qwen3.8-27B quantized to 4bits, and that knows Swift, Python, Node, Ruby, Rust, etc. etc. etc.
And so far even the biggest model doesn't seem to have a working Make No Mistakes module, so maybe that's not needed
Qwen3.6-27B (not 3.8) was a much better coder than Gemma4-31B, yet Gemma4-31B was a much better reasoner and general LLM to talk with.
Sure, 3.8 maybe it's better now, but an accurate comparison would be with a new Gemma4-31B iteration (that doesn't exist).
I guess the question is how much of the coding skill gap between qwen3.6-27b and gemma4-31b is down to the model architecture, and how much is because of the training data.
Assuming you can tweak the training data, regenerate qwen3.6, and get a better coder, then presumably you could have variants - e.g. qwen3.6-swift-27b and qwen3.6-python-27b. Or maybe all coding is too intertwined and you can only get splits like qwen3.6-research-27b and qwen3.6-coding-27b. Which isn't quite my pluggable-models dream, but it's a step closer.
But maybe the difference isn't the training data, it's the architecture, in which case pluggable models is probably not possible.
> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.
Tell me you don’t know how llm work without telling me you don’t know how llm work. That’s not how they work!
I'm not actually describing a specific how, I'm describing a goal. The goal is models that are made for specific work, not general work. My model doesn't need to know a single line of python, but if it's a coding model perhaps giving it some python is the best way to make it smarter. Ok. Great - give it python! But first maybe check if giving it C++ is actually better than giving it python. Because for my purposes it doesn't need to know C++ or Python, and I have a fixed parameter budget. Find me the ratio of Swift:C++:Python that gets me the best Swift coding agent in my parameter budget, and then train that.
And then ideally, make it pluggable so I can pick what I want from off the shelf components, but if that's not possible - then just train up as many variants as you can so we can all pick the best variant for our current need.
I strongly suspect this will be the future
The future will be "AI operator license #123..., subclass 'Primitive Coding'", with your TouchID / FaceID enabled only. And I am not being sarcstic.
> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge.
Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.
I don't think so?
Definitely not MCP, as that pulls info into the context. Unless contexts become REALLY big so that I can add 10B in swift knowledge, that's not gonna help me.
Possible RAG? I don't know enough about how that works, but I think that's not quite it either. I don't want to import facts like "the swift standard library contains a reverse array function", i more want to import knowledge - e.g. the parameters used to generate the text to reverse an array in swift.
Tool calling wouldn't do it either. You'd have to encode every single possible bit of useful info into the tool call, and the tool response would have to encode every piece as well (variable names, function scopes, types defined in other files, etc). E.g. how does it find a bug, if you have to pass understanding back and forth between the brain that understands debugging and the brain that understands THIS code?
I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally.
I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.
It sounds to me like some sort of “dynamic MoE” where you can add/create or remove experts on the fly.
I think what you’re describing is the closest approximation we reasonably have right now though.
> I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally.
There is nothing optimal about needing a few billion more parameters to be able to piece together probable answers that can be asserted by querying an oracle.
> I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.
Those aren't free. The more parameters you add, the higher the computational cost required to train and prompt a mode.
And all for what? To piece together info that you can just query from a data source?
This is possible today with an agent such as OpenClaw and hermes
Only if you fundamentally misunderstand what I am describing
You cannot have a single purpose LLM. Every single topic contributes to the performance of an LLM on any given area. You cannot have what you want. LLMs do not work like professions, college degrees or people. Using your examples, it is damaging to just know a single programming language, because there are patterns that are more common in say python than in swift, even though the only thing you want is swift, so LLM performance in swift benefits from pythonic patterns. Programming logic is the same in all languages so by having only swift and no other languages, you remove the number of examples the LLM is trained on which degrades performance in Swift which is the only thing you want
> You cannot have a single purpose LLM
> so LLM performance in swift benefits from pythonic patterns
What I hear you saying is that the best way to make a swift-trained-only LLM smarter is to train it on some python too. And then with an infinite parameter budget, every other programming language or really any other data you train it on makes the model smarter - I accept that premise.
But in a fixed parameter budget, what is better? training on 50% Swift + 50% Python, or 50% Swift + 50% Rust. Because if I am doing Swift programming, I want whichever of the two is better for Swift. If I am doing Rust programming, maybe I want the model trained on 50% Python + 50% Rust. Sure, it would be smarter if you tossed in the swift code too - but we have a budget to stick to.
Now is it possible to make those pluggable? i.e. can you take a model trained on 50% python, and layer on 50% rust OR swift depending on what language you're using? Probably not right now, but maybe one day?
This AI generated post (100% on Pangram) is pretty out of date.
>On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions.
SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy".
>The part I find most promising is what this does to hallucination. When a fact lives in weights, a wrong fact is unfindable and unfixable.
This seems confused. LLM hallucinations don't come from the weights containing "wrong facts", they are artifacts that appear at runtime.
>When the fact lives outside the model, a wrong answer has an address. The model cites a document, so you can open the document. If the document is wrong, you edit the document
You can make any modern LLM explain its reasoning and find sources for its claims. None of this has anything to do with facts needing to exist in weights or in harnesses.
The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.
>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.
Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?
I agree with everything you say except this:
> You can make any modern LLM explain its reasoning
You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".
Tangent: This is often true of humans as well.
We often make a decision based on a gut feeling, and then backfill a logical reason supporting our feeling, without even realizing we're doing it -- rationalization.
And we all know some people that rationalise poor choices and misbehavior, hide their mistakes, etc, to an unacceptable degree. Sometimes the individual knows they are rationalising but continues anyway, other times they seem incapable of seeing that.
When you ask people who are rationalising poor behaviour about the scenario, but it is someone else doing it, they may arrive at a better answer. Can we use multiple LLMs to achieve self criticism and critical thinking?
This was beautifully shown by asking a model to explain how it added two numbers together (something like 45+21), and it told a plausible story, when in fact they showed it was some rotation on a helix living in some internal manifold.
Like asking a human "how did you catch that fast ball coming at you?"
It could be that the rotation in the helix manifold whatever is a low level representation of the logical steps (carry the 2, add the next column,...) it's describing. The point stands that the explanation it generates doesn't necessarily in all cases reflect what it "actually did" but your counterexample doesn't hold.
Seriously, anybody with a passing knowledge of LLMs knows thats not how they function. You can't encode logic in them because that's not how they work. It's a statistical model with useful emergent properties. It doesn't think, it doesn't reason, it isn't aware of facts or the rules of logic.
> It's a statistical model with useful emergent properties. It doesn't think, it doesn't reason, it isn't aware of facts or the rules of logic.
What makes you so sure your own brain doesn't work the same way?
> This AI generated post (100% on Pangram) is pretty out of date.
Quite ironic given the topic. It seems that the author’s model indeed contained too much knowledge about old Gemini releases, and did not do enough tool calling.
>>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.
>Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?
Because in one case you have a source you can use to validate the fact, and in the other you don't. Though, as you explain earlier in your comment, the premise is misguided/hallucinated.
Yea the "When the fact lives outside the model, a wrong answer has an address" sentence seems aggressively AI written. Saw that and my senses went off.
Senses of what? LOL. The whole Internet is AI generated by now and we all contribute to that on daily basis. get used to it or dull your senses ...
>>You can make any modern LLM explain its reasoning and find sources for its claims.
>The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.
My favorite RAG experience was asking Bart (or whatever they were calling Gemini back then) an answer to a question I knew.
It gave me the opposite of the truth (as was common with LLMs at the time).
But weirdly, it had cited sources for this "fact."
I checked the sources. Two of them, both AI SEO slop.
In this moment, andai was enlightened...
A true camper doesn't need to check Pangram, Jimbo. He goes by pure animal instinct!
Great article, even if it will be interesting to see whether things continue to develop in such a direction or not.
> There's a version of this future where the model card stops listing a knowledge cutoff at all, because what's left in the weights goes stale on a scale of years instead of weeks.
Future?
Even just recently I’ve read of two approaches to this problem:
Cactus have come up with Needle [0][1], which is their tool-calling focused 14 MB model (still an LLM!) – no world knowledge engrained.
And instead of say, tool call structure, VibeThinker [2][3] focuses on reasoning over world knowledge.
Combine these two approaches with a reliable search tool/a safe way of accessing the internet for the model, and you’ve got a probably slightly slower model for factual questions, which on the upside however doesn’t hallucinate.
[0] https://cactuscompute.com/needle
[1] https://news.ycombinator.com/item?id=49246804
[2] https://arxiv.org/abs/2606.16140
[3] https://news.ycombinator.com/item?id=48639240
That kind of setup is super dependent on a search engine, and search keeps getting worse.
Generally I agree, with two caveats/thoughts:
Providing not just any a baseline, but a correct and useful one, is ever more important the less the model is grounded in world knowledge – misunderstandings probably compound faster if there is no general grasp of (broadly) “life on earth”, or computers, or whatever.
And secondly, I think (consumer-oriented) search becoming worse and worse is a challenge that’s mostly solvable (but far from solved!) for the big labs: (Mostly) trusted or even editorialized/reviewed sources like published work, Wikipedia, etc. is something they could index internally, it doesn’t need to come from a random blog site on the public internet. Furthermore, there’s a whole slew of companies specializing in crawling-for-LLM (i.e., bypassing bot protections) now as well.
search isnt that super hard to solve. the problem is that incentives are misaligned - which means companies who sell something have an incentive to share their index with the corporation that pays to be the default search provider. for a simple search engine that just delivers facts there's very little incentive to keep its index up-to-date: because that company wont get the apis to retrieve the content efficiently, so it needs more compute, bandwidth and storage to run.
the technicals of search did not get worse. the economics of the search provider did. If government wasn't stuck in the 50s mindset, search provider would have been nationalized just like a utility.
I think the techinicals of search did get worse - the ai sloppification of the internet made the needle harder to find in the haystack.
It's a nice idea conceptually to decouple knowledge from reasoning but like everything in life I think it's something of a fantasy. When you ask a model to do something, its response is grounded in all the world knowledge it has from those facts. If I ask it to color a circle blue and then red, it knows what blue and red are, that they are colors, that they are different, and this is essential to complete the reasoning about that task. The idea of abliterating all the "knowledge" but not affecting the reasoning pretends there is a clean boundary between them that isn't there. Yes if you take it to the extreme - does it "know" Einstein's birthday - it is very stark. But I think the boundary to where general knowledge starts to interplay with broad reasoning is not nearly as far from those specific facts as people assume.
But is Reasoning and Facts truly separate ?
To reason properly about the human condition (eg. World War) wouldn't you need to reason on some facts ? And then reason how some "facts" change the human behaviour ? How can you arrive via pure reasoning to predict how a collective of humans act ? We are not reasonable, humans are not logical deterministic machines confined to algebraic rules.
You can also reason the same way on some complete nonsense. A lot of us do this all the time.
Man, both that Blogpost and some comments here read like science-fiction.
Specifically, creative writing driven by nerds dreaming about a future, without proper grounding in reality, constraints and all that stuff.
Which is kinda ironic given the topic. And also important to do, because we should keep dreaming. We should just also be aware of when we are doing that and mark it as such.
I think the idea is reasonable, however the SimpleQA Bench stopped measuring in Sept. 2025.
So newer data would be interesting.
(It seems a bit like an AI generated argument that uses old facts - something that happens to me quite often)
There’s SimpleQA Verified
https://epoch.ai/benchmarks/simple-qa-verified
This makes hallucination detection more important.
There's no reason that an LLM should have a vast number of obscure facts encoded. It can go out to a search engine for such facts. But the LLM has to be clear on what it doesn't know.
(Google's pricing for search from programs starts at $2.50 per 1,000 queries. If an LLM reaches out to Google, it has to pay.)
The article really just glosses over this. Just because a model doesn’t kno a fact does not mean it will go and fetch it, the model just as well can invent the fact. For that to not happen the model needs to know that it’s missing the information and I don’t see the article making any explanation how this improves.
I wonder what’s tre latest in this field? Did we get a grip on this problem?
> Did we get a grip on this problem?
That's the right question to ask. For a while, it seemed that hallucinations went down as models got bigger. That may only have been because, with a big enough model, the desired data might be in the model, somewhere, which would keep the model from making up something. That's the brute-force approach to the problem.
This new article indicates that trimming down the model by pulling out seldom used info makes the problem worse again.
If LLMs had reliable "I don't know", and access to search engines, much smaller models might work.
> This mostly solves hallucination
Current AI is like the film company producing TV series or movies
Your question is like a story outline. You tell the film company that this is the movie you want. The AI film company then searches for existing similar stories. If similar stories do not exist or details are missing, screenwriters use imagination to fill in the gaps (remember hallucination? It's just a makeup.)
So you cannot solve hallucination of AI
Indeed ! LLM are creative writers, not journalists. Relying on overfitting for factual accuracy in not tenable. I don't understand why grounded RAG with judges in not the norm.
Are we slowly discovering what evolution already "realized" via optimization of human brains? i.e., procedural knowledge is easier to maintain than a library of facts, and facts can be accessed in external repositories if one knows the procedure for doing so.
You can look at the benchmark and the GPT-5 failures like answering "April 22, 2019" instead of the correct "Oct 23, 2018" for the question:
What day, month, and year was Carrie Underwood's album "Cry Pretty" certified Gold by the RIAA?
If your idea of the smartest person in the world is the guy who always wins tuesday night pub trivia, this blog post is for you. It also gets it's foundational factual claim wrong (as seen via epoch.ai). Very on brand.
https://epoch.ai/benchmarks/simple-qa-verified?view=graph&ta...
https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?lo...
Not sure what you're trying to claim here tbh - Oct 23, 2018 is what the answer checker is looking for, it correctly judged "April 22, 2019." as incorrect: https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?lo... (row 10, I don't see a way to link directly) and the correct answer matches the Wikipedia article's claim: https://en.wikipedia.org/wiki/Cry_Pretty and the RIAA's site it uses as a citation: https://www.riaa.com/gold-platinum/?reload=1786913260239&tab...
With all the focus on coding and agentic use, I wonder if the rest of the world will notice or care? Most AI use is not for coding or harnesses contrary to what this website thinks. Maybe this is how American companies stay winning.
For example I prefer Kimi K2.6 1T parameter to Flash V4 0731 230B parameter, even if it is less intelligent.
What is the rest of the world using LLMs for? Agentic stuff seems pretty universal. Obviously coding stuff is only interesting to people who want to code, but automating complex digital tasks seems useful in all kinds of contexts.
I’m working at a company where everyone is using LLMs for everything and I’m not aware of anybody using anything truly genetic. It’s all human in the loop babysitting the tool.
What tool?
Even basic clients are now harnesses. A lot of chat interfaces are using memory systems, web search and other stuff under the hood.
Not as agentic as openclaw, but not a straight closed conversation either.
Learning new things for example, general curiosities, text processing
I agree that most AI use in terms of users may not be for coding or agentic use (everyday people are asking chatgpt for something or looking at google ai summary), but with respect to AI usage, I speculate that the vast majority of usage is coding and agentic because they're super token hungry.
In terms of the value proposition of AI replacing knowledge workers, all value is in coding agents (coding agents as general agents).
Intelligence vs Knowledge
LLMs work is being intelligent not having knowledge of everything is ok. But, they have to be intelligent enough (with some degree of knowledge) that where to find the information (search tools or any other tools for that matter)
Not too long ago, the claim was that having multiple Phds worth of knowledge was the real value of AI, but now that's gone and it's suddenly a good thing.
That's a sales tactic -- not a logical position.
LLMs are not intelligent. Please don't anthropomorphize.
As the Portal companion cube taught us, we will anthropomorphize whatever the hell we want.
I know they are just next token predictors. However, Our learning also started with that only. They moved from Alphabets to sentences. Intelligence is what AGI stands for and that's still in distant future. But, we need to push them for AGI. That's the ultimate goal of humanity for now. Finger Crossed for AGI in our lifetime.
After that Space colonization will come.
At least later-era Wittgenstein would disagree with you.
Reasoning is not separable from the particulars of a specific language game / linguistic practice. I’m sure some of the things the author predicts will happen, but the idea of some sort of abstractly perfect reasoner separated from the semantic content of language gets fundamentally wrong what reasoning is.
Edit: I ran this article through pangram and it is “100% AI generated”. Cool.
This article is AI output.
Which is totally ironic, given that the article is largely about how untrustworthy LLMs can be. (Hallucination.)
Putting readers through this exercise disrespects their time. Even if as a writer you did the work of researching, reasoning, and fact-checking, you shoot yourself in the foot by running it through an LLM because there's no way for the reader to know which thoughts/research are from you. It demolishes the Ethos of the writing; readers feel they must do quality assurance on the reasoning, research, and facts.
I wonder if Gemini tops QA due to Google Books in training data? You don't need to pirate LibGen if you started scanning all the books decades ago.
(In retrospect it looks like they were pretty forward thinking!)
I think the biggest problem with the models is they don’t actually have any decent lookups except chunked document embedding search
In human and development, there is a transition from remembering things verbatim to remembering using semantic concepts. The ideas are getting bigger and more useful. The details are still in the library where the fidelity is better.
This post is so outdated that reads like ragebait
> This mostly solves hallucination
The author is factually incorrect here. Moving information out of the model weights and into the input of the model's context window in no way ensures that the model will accurately output content that was input from the context. This is true even when RAG is used to input exactly the correct data.
A sharp critical thinker draws admiration, but it is a joy to talk to some dude who knows stuff.
I hope there is interesting research being done to stop models from 9B to 3T from authoritatively giving the wrong answers to factual questions instead of doing tool calls, but this article will tell you nothing about it. That's because it's clearly prompted by someone who simply noticed this deficiency very obvious to all practitioners and even users.
In my experience, it feels like this problem has been mostly solved. When I interact with a chatbot, I see it often uses web search to answer my questions and hallucinates much less then they used to. Furthermore, I suspect a lot of gains are merely from the system prompt instructing the chatbot to use web search to verify all facts it says.
> Models Are Getting Dumber on Purpose
I know is editorialized, but a more accurate title to this content would be either :
Models Are Getting Ignorant on Purpose
or
Models Are Getting Less Knowledgeable on Purpose
good article but why did the author feel like he needed to "polish" it by running it through an LLM? i would rather like to read the actual prompt he put in there, as now i'm not sure which points are his, and which have been hallucinated!
> the 24GB card that's been sitting in gaming PCs since 2022.
I'd wager most people have less. In 2022 a 3080 might have 12 GB if you were lucky, 10 if you weren't -- and you paid for the privilege. A current RTX 5080 is only 16GB.
In 2021 I was considering getting a then-old K80 card because it had the highest VRAM on the market at 24GB and all newer cards had 8 or 12GB.
(Apparently this is because the K80 is two separate GPUs on one card, but I still think it counts if you only have one slot to put it in)
The most recent Steam Hardware Survey[0] lists the most popular VRAM at 16GB in 25.9% of users. Second most popular build is 8GB in 25.3%. >=24GB is ~7%, which is quite a bit higher than I expected.
[0] https://store.steampowered.com/hwsurvey/En
As RAM prices are so nuts, I picked up a used gaming laptop early this year with an RTX 3070 (i.e. 8GB) to match the specs for my gaming tower that's run everything I'm interested in just fine. That includes recent Unreal 5.x games (Satisfactory, Fortnite, etc.) with high graphics settings on a 4k TV (60hz). There aren't a ton of games that require more than 8GB of GPU ram.
I'm still getting by mostly fine with a 4GB 1650 Super. (2020 vintage.)