It solves for the same thing a "train of thought" (internal vocalization) solves for in the human brain: there's a lot of stuff you "know" as episodic memories of words you've heard/read, but haven't yet internalized via practice into reflex reactions. It's a lot faster and cheaper to acquire a memory of some words, than it is to acquire an entire new skill, so you end up picking up far more memories-of-words than concrete skills from a given training corpus.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.
(An especially useful "trick" of this system is planning, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
Can you explain why your view, which is the opposite of the OP's, is correct?
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
Not mutually exclusive. LLMs are kind of like a self-fulfilling prophecy. Once some technique is discovered, it'd be part of the training process and becomes more efficient.
I get that this is the goal, but having looked at reasoning traces they kinda read like baby babble. The model is just kinda sitting there restating the question and itself over and over. It'll sometimes kinda state/clarify assumptions but it feels nothing like actual train of thought
I'm pretty sure that the current approach of baking in chain-of-thought reasoning into weights is not the end of the line. It just seems to accidental and ad-hoc.
> Consider the problem: “what’s 1 +2 +3 +⋯ +10”? One solution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to pair them from the outside in: 1 +10, 2 +9, 3 +8, 4 +7 and 5 +6 each make 11, and five pairs of 11 make 55. Both of these paths land on the correct answer, 55.
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
We might even formalize that concept and study how many different ways we can prove a claim, what they have in common, if two paths are mostly the same or very different, etc.
Much of modern foundations is about categorizing reasoning structures by such paths!
Wang's argument contradicts the view that intelligence necessarily improves through more reasoning and undermines any assumption that recursive self-improvement (RSI) will automatically converge toward greater intelligence.
Normalizes user voice to goal so the action has more chances to success you can see this on model that dont hide traces sending very badly spelled or incomplete sentences
That is how I have been thinking about LLM reasoning pretty much since it arrived.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
> Don’t we train the model to use a certain method to explore the space in search for an answer?
Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.
My understanding of the math is weak (to say the least), but intuitively what I'm getting from this is:
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
During training you're trying to find paths to success, and reinforcing those.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are luck the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
Yes. One point worth making though is that pre-training trains the model to produce high probability completions - in practice, most of what it's trained on is a good completion as a human would judge it, i.e. it's not factually false data, so you get "creates a bias _towards_ "correct answers" with each inference step", but the model isn't really learning what correct or incorrect means in any way a human would consciously think about it.
It depends on what you mean by “inside the box thinking”. In one sense, you are always mostly sampling high probability completions and they are going to be close to things that have been observed during training. Low probability completions will likely be low quality too. But, in a case where the context is forcing the model to generalize, I.e. interpolate in the latent space, the most probable completion may appear novel.
> The mathematics on this page is being rendered by MathJax, which generates both the text spoken by screen readers, as well as the visual layout for sighted users.
It solves for the same thing a "train of thought" (internal vocalization) solves for in the human brain: there's a lot of stuff you "know" as episodic memories of words you've heard/read, but haven't yet internalized via practice into reflex reactions. It's a lot faster and cheaper to acquire a memory of some words, than it is to acquire an entire new skill, so you end up picking up far more memories-of-words than concrete skills from a given training corpus.
Reasoning (i.e. bubbling those remembered words up into your mental context, and then using a special "trigger effectively arbitrary synaptic activations dependent on the content of the attention-masked context and an ongoing state" bridging reflex on them, a.k.a. "system-2 thinking") is how you leverage memories-of-words into useful behavior. Rather than executing native skills embedded into your synaptic weights, you're running a single very fancy native skill that acts as an interpreter for word-memory data, slowly and strenuously turning words you're holding in mind into more words (synthesis), or into associations to other memories (word-memories or otherwise), or into actions/outputs, etc.
(An especially useful "trick" of this system is planning, i.e. chaining the interpreter through "mere words" that conjure more "mere words" to mind until you end up at words that form a declaration of what you're about to do. You can then rely on a special part of the bridging reflex, namely the "instruction-following" reflex, to react in a way where your actions are constrained by the plan you've generated.)
Can you explain why your view, which is the opposite of the OP's, is correct?
"RL training directly raises the success probability , by spending a lot of compute on searching for successful paths and then training the model to reproduce them. The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference."
Not mutually exclusive. LLMs are kind of like a self-fulfilling prophecy. Once some technique is discovered, it'd be part of the training process and becomes more efficient.
I get that this is the goal, but having looked at reasoning traces they kinda read like baby babble. The model is just kinda sitting there restating the question and itself over and over. It'll sometimes kinda state/clarify assumptions but it feels nothing like actual train of thought
I'm pretty sure that the current approach of baking in chain-of-thought reasoning into weights is not the end of the line. It just seems to accidental and ad-hoc.
P. S. The animated bird is truly magical!
I just realized the bird turns it's head to follow the mouse pointer.
Yes, and the animation is made using an interesting technique that I can't immediately figure out.
For example, what is this [0] or this [1].
The code is not obfuscated and seems well documented [2]. Maybe, I'll read it later or have an agent explain it to me.
[0] https://alexwang.ai/images/kea/head-fill.webp
[1] https://alexwang.ai/images/kea/body-feathers.png
[2] https://alexwang.ai/js/kea3d.js
> Consider the problem: “what’s 1 +2 +3 +⋯ +10”? One solution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to pair them from the outside in: 1 +10, 2 +9, 3 +8, 4 +7 and 5 +6 each make 11, and five pairs of 11 make 55. Both of these paths land on the correct answer, 55.
Watching my son work through difficult math problems has forced me into this realization. In my head there is one "correct" path, and I often have to stop myself from interrupting his chain of thought. He often is just getting there a different way.
We might even formalize that concept and study how many different ways we can prove a claim, what they have in common, if two paths are mostly the same or very different, etc.
Much of modern foundations is about categorizing reasoning structures by such paths!
Wang's argument contradicts the view that intelligence necessarily improves through more reasoning and undermines any assumption that recursive self-improvement (RSI) will automatically converge toward greater intelligence.
Content aside, that's a beautiful webiste
Normalizes user voice to goal so the action has more chances to success you can see this on model that dont hide traces sending very badly spelled or incomplete sentences
That is how I have been thinking about LLM reasoning pretty much since it arrived.
I am not sure I support his statement about RL “searching for successful paths and then training the model to reproduce them”.
Don’t we train the model to use a certain method to explore the space in search for an answer?
As soon as you have tool calling and a way to verify your result, such as a theorem prover for example you are essentially running an optimization algorithm.
> Don’t we train the model to use a certain method to explore the space in search for an answer?
Not really - you just sample random high-probability completions and then the verifier tells you which ones to nudge the model towards and which to nudge away from. There's lots of clever ways you could improve the search process, but standard practice right now is just random sampling.
Lot of Alex Wangs out there
My understanding of the math is weak (to say the least), but intuitively what I'm getting from this is:
1) Pretraining creates a bias _towards_ "correct answers" with each inference step
2) A longer walk is more likely to land in the correct region, given it's easier to verify you have the correct answer than to create it
3) RL selects the subset of walks that lands in the correct region and tilts the distribution in favor of such walks
4) By 2, that trains towards longer walks
...Is that correct?
EDIT: Wait, but this contradicts the post's claim that "The gains we credit to “thinking longer” come from the search we paid for during training, rather than at inference.". This requires that thinking longer is not at all causal for correct answers, which contradicts 2.
During training you're trying to find paths to success, and reinforcing those.
There is going to be some minimum length path (assuming the model is generating some semblance of real reasoning), but more long ones given the possibilities for less optimal solves and backtracking after errors, so it stands to reason that if you think longer during training you'll find more paths to success.
At inference time you are hoping to follow one of the reasoning paths discovered during training. You need to generate enough tokens to follow the path (or one of the paths), but continuing to think past this point can only be detrimental - if you are luck the model will realize the error and circle back, but if you are unlucky it will snatch defeat from the jaws of victory and wander off who knows where.
There was a story recently linked here about one model "editing the weights" of another and thereby improving it's math performance, which was exactly about this - the need to stop generating at inference time at the right point.
IMO LLM's RL acquired reasoning, even if it gets the job done, sometimes spectacularly (math, coding successes), is really a very dumb cargo-cult process. At inference time the model is just trying to copy what it "saw" at training, without actually understanding it, which may even include duplicating errors and subsequent backtracking that had lead to success during training.
Yes. One point worth making though is that pre-training trains the model to produce high probability completions - in practice, most of what it's trained on is a good completion as a human would judge it, i.e. it's not factually false data, so you get "creates a bias _towards_ "correct answers" with each inference step", but the model isn't really learning what correct or incorrect means in any way a human would consciously think about it.
does that lead to much inside the box thinking by models? Do they try to follow the clearest path?
It depends on what you mean by “inside the box thinking”. In one sense, you are always mostly sampling high probability completions and they are going to be close to things that have been observed during training. Low probability completions will likely be low quality too. But, in a case where the context is forcing the model to generalize, I.e. interpolate in the latent space, the most probable completion may appear novel.
> reasoning as pathfinding through token space
Sounds like it's hillclimbing all the way down!
Cool framing. I'm excited for the parallel construction of ___ as pathfinding through weight space.
Mostly, it seems to be solving the problem of people thinking about things themselves.
What is the library, if any, used for the equation/formula rendering? Looks tastefully done.
> The mathematics on this page is being rendered by MathJax, which generates both the text spoken by screen readers, as well as the visual layout for sighted users.
Lovely. Thank you.
Misrenders on my browser. The bottom half of the characters in the equations are partly chopped off.
also very curious, visualization is great!