There’s a repeated theme here that I don’t understand. For example:
> (3) That's where I decided to build a harder test, where each task takes its start from one demonstration and its goal from another
> (4) I also built an interactive demo where I could place the block anywhere and pick any goal. [but]
The goals in the harder test are frames from expert demonstrations
Why lean so hard on the demonstration examples? Surely the actual task here is to move between any two arbitrary poses.
So if the demonstration set is statistically strange in any way, you keep banging into that. And even if you get great results, you still haven’t shown that it generalizes to the generic system. And the one thing you know 100% about a physical robot system is that you will never encounter exactly the same situation twice.
Is there actually a reason to pick eval examples from the demonstrations instead of independent samples from the underlying task?
I’m pretty sure that the T-shoving problem doesn’t have goal reachability issues. And if it did, test 4 could run into them since the start pose is arbitrary.
Agreed, I don't know much about JEPA but i wish this article had explained / motivated the memory graph more. "3. Fill gaps in memory with transformed memories" in particular seemed really really damning for the models—when you removed the examples from the demonstration that were near the goal state, the model completely broke. It seems like the only way you could make the model work again would be to manually flip and transform the memories it did have so there was something in the vicinity of the goal location.
That seems like a really massive failure of the model to generalize? This doesn't seem like a new technique, it seems like you've discovered a pretty big issue with the fundamental setup
JEPA is very similar to Model Predictive Control. Here, you swap the model of the system with black-box simulator (a neural net), apply some version of dynamic programming / Bellman equation to find the optimal action over planning horizon.
In the planning stage, by factoring out the optimization step from the rollout step, you can free the model to focus on learning the dynamics in latent space. (The other case, where the rollout+optimization happen in the learning loop, just becomes model-free RL.)
My background straddles classical & data-driven controls. This was the first similarity that popped out to me when I first started learning about world models.
Is JEPA just MPC but where the "M" is a latent-space model?
I think this is why it has a lot of potential. I'm also controls background, so I'm going to just throw stuff at the wall.
I'm thinking it's both M and P. M for the reasons you've mentioned. Then there is P where a forward horizon prediction of both plant and the world.
I think JEPA also touches on the perception tasks: What to do with sensor data? Classical control objectives are quite simple, drive the state to stable, follow a trajectory, stay safe. It's very deterministic: Sensor data is used to determine state. State is used to update dynamics, dynamics is used for the forward horizon.
So. MPC works for things like self-driving because the car model is fairly knowable and world model is predictable (rules of the road, knowable dynamics of other cars).
JEPA seems like a fusion of control, sensing, world model into an embedding space where the learning that occurs is a model free approach to how these are all related to each other.
Yeah not sure where the C comes from; models lack access to knobs and switches of the system they're on and taking over the OS isn't control over the hardware still since it can't upgrade its RAM, or GPU
Yeah it looks like humans, other animals, and life in general are much more willing to instinctively control things in the transient regime, away from equilibrium
I guess this is the new frontier. Rather than doing safe, predictable, boring control, push the envelope on what trajectories are possible, so that control becomes more lifelike and has a degree of autonomy to recover from error states
Yes absolutely. JEPA is far from the first approach to apply a learned dynamics model to MPC - it's just been well-marketed.
If you're interested, I'd recommend the TD-MPC series of papers, which also learn a latent world model, but achieves much better task performance in general. Unlike JEPA, TD-MPC learns to predict the task reward (what you'd call the cost in optimal control), so it's got a much better ability to rank trajectories during planning.
Hands-on learning is an awesome thing - especially in the age of AI!
But maybe worth reframing away from “discovering” - hierarchical planning, sub-goals, action chunking etc. are well understood - this is more of a… transcript of your personal learning journey?
Which is still cool to see. But I read through expecting there to be some kind of novel results. We probably need to categorize these kinds of posts a bit better.
Hm, I get your perspective, but completely disagree with your expectation. I'm not sure I would be entitled to expect a discovery that is novel with regard to current scientific knowledge unless I'm reading a paper. The fact that it is couched as "I discovered" and posted to a personal blog should be more than enough to convey that you might not find novel scientific discoveries within.
Fair. To clarify, I jumped in seeing this ranked #1 on HN which warped my expectations.
The root of my concern is the overall trend I'm seeing on HN: Someone spends time iterating with a coding agent exploring a field that's new to them. The agent's suggestions come from existing literature, but it rarely says "this is Search on the Replay Buffer, here's the paper." So each new concept feels like a discovery and gets written up that way. The enthusiastic tone of AI-assisted writing adds to this sensationalized effect.
So my concrete feedback would be a title like "Learnings from 600 planning experiments with LeWM on Push-T" since the current one suggests something novel about JEPA world models in general. And cite techniques they've learned along the way so others can learn from source material too.
I knew close to nothing about JEPA before reading your post, but i was under the impression this was supposed to be a contender for LLM's limitations, a way to do things "properly".
However, based on this article, i have the feeling that on the contrary, JEPA is filling a super niche problem, and is pretty much only trying to improve robotic motions in space.
Which of the two ideas is it ? Can JEPA technics be applied to a much more wide range of problems ? (like LLM and transformers)
JEPA can be applied to text as well in the sense you can use it to build text embedding models, freeze your JEPA text embedder, then train a decoder over the top of that.
OF course your decoder model could end up doing a lot of the heavy lifting if your decoder loss was based on text reconstruction methods (which almost all LLMs use for eval purposes)
For certain tasks, you do need pixel/token level reconstruction fidelity to do a lot of things. Like image generation or text generation.
For other tasks, having a coherent representation of the world that can be computed quickly and intuitively is more salient. I suspect the human mind works more like JEPA than like an autoregressive LLM. Human children train on far less data to achieve extreme fidelity on a lot of tasks generative reconstruction loss based AI/ML techniques struggle with.
In a world where I'm increasingly being triggered by the fonts default Claude uses, I really liked the font used for the title here - it's Faculty Glyphic FYI.
We are doomed if most of the comments here are just talking about the website layout and not the contents of the article.
Anyway:
> The physics of pushing a block doesn't change if you shift, rotate or mirror the whole table. So when the graph has no states near a goal, it can take demonstration segments from elsewhere, transform them to fit the gap, and plan through those. With these transformed segments, success came back to 85%
This is an interesting technique to recover incomplete states or lost memories after demonstration. I'm guessing that the original LeWM does not have technique of transformative memories if I am correct?
If you’re going to use AI to generate text, keep it short and concise and meaty. I don’t hate reading AI, but I can’t read large walls of text for presumably some useful nugget of information in it.
There’s a repeated theme here that I don’t understand. For example:
> (3) That's where I decided to build a harder test, where each task takes its start from one demonstration and its goal from another
> (4) I also built an interactive demo where I could place the block anywhere and pick any goal. [but] The goals in the harder test are frames from expert demonstrations
Why lean so hard on the demonstration examples? Surely the actual task here is to move between any two arbitrary poses.
So if the demonstration set is statistically strange in any way, you keep banging into that. And even if you get great results, you still haven’t shown that it generalizes to the generic system. And the one thing you know 100% about a physical robot system is that you will never encounter exactly the same situation twice.
Is there actually a reason to pick eval examples from the demonstrations instead of independent samples from the underlying task?
I’m pretty sure that the T-shoving problem doesn’t have goal reachability issues. And if it did, test 4 could run into them since the start pose is arbitrary.
Agreed, I don't know much about JEPA but i wish this article had explained / motivated the memory graph more. "3. Fill gaps in memory with transformed memories" in particular seemed really really damning for the models—when you removed the examples from the demonstration that were near the goal state, the model completely broke. It seems like the only way you could make the model work again would be to manually flip and transform the memories it did have so there was something in the vicinity of the goal location.
That seems like a really massive failure of the model to generalize? This doesn't seem like a new technique, it seems like you've discovered a pretty big issue with the fundamental setup
JEPA is very similar to Model Predictive Control. Here, you swap the model of the system with black-box simulator (a neural net), apply some version of dynamic programming / Bellman equation to find the optimal action over planning horizon.
In the planning stage, by factoring out the optimization step from the rollout step, you can free the model to focus on learning the dynamics in latent space. (The other case, where the rollout+optimization happen in the learning loop, just becomes model-free RL.)
My background straddles classical & data-driven controls. This was the first similarity that popped out to me when I first started learning about world models.
Is JEPA just MPC but where the "M" is a latent-space model?
I think this is why it has a lot of potential. I'm also controls background, so I'm going to just throw stuff at the wall.
I'm thinking it's both M and P. M for the reasons you've mentioned. Then there is P where a forward horizon prediction of both plant and the world.
I think JEPA also touches on the perception tasks: What to do with sensor data? Classical control objectives are quite simple, drive the state to stable, follow a trajectory, stay safe. It's very deterministic: Sensor data is used to determine state. State is used to update dynamics, dynamics is used for the forward horizon.
So. MPC works for things like self-driving because the car model is fairly knowable and world model is predictable (rules of the road, knowable dynamics of other cars).
JEPA seems like a fusion of control, sensing, world model into an embedding space where the learning that occurs is a model free approach to how these are all related to each other.
Yeah not sure where the C comes from; models lack access to knobs and switches of the system they're on and taking over the OS isn't control over the hardware still since it can't upgrade its RAM, or GPU
But the M+P aspect def fits.
Yeah it looks like humans, other animals, and life in general are much more willing to instinctively control things in the transient regime, away from equilibrium
I guess this is the new frontier. Rather than doing safe, predictable, boring control, push the envelope on what trajectories are possible, so that control becomes more lifelike and has a degree of autonomy to recover from error states
Yes absolutely. JEPA is far from the first approach to apply a learned dynamics model to MPC - it's just been well-marketed.
If you're interested, I'd recommend the TD-MPC series of papers, which also learn a latent world model, but achieves much better task performance in general. Unlike JEPA, TD-MPC learns to predict the task reward (what you'd call the cost in optimal control), so it's got a much better ability to rank trajectories during planning.
In LLM parlance, I guess you could say MPC rollout and planning is the "harness" around a JEPA style world model.
Hands-on learning is an awesome thing - especially in the age of AI!
But maybe worth reframing away from “discovering” - hierarchical planning, sub-goals, action chunking etc. are well understood - this is more of a… transcript of your personal learning journey?
Which is still cool to see. But I read through expecting there to be some kind of novel results. We probably need to categorize these kinds of posts a bit better.
Hm, I get your perspective, but completely disagree with your expectation. I'm not sure I would be entitled to expect a discovery that is novel with regard to current scientific knowledge unless I'm reading a paper. The fact that it is couched as "I discovered" and posted to a personal blog should be more than enough to convey that you might not find novel scientific discoveries within.
Fair. To clarify, I jumped in seeing this ranked #1 on HN which warped my expectations.
The root of my concern is the overall trend I'm seeing on HN: Someone spends time iterating with a coding agent exploring a field that's new to them. The agent's suggestions come from existing literature, but it rarely says "this is Search on the Replay Buffer, here's the paper." So each new concept feels like a discovery and gets written up that way. The enthusiastic tone of AI-assisted writing adds to this sensationalized effect.
So my concrete feedback would be a title like "Learnings from 600 planning experiments with LeWM on Push-T" since the current one suggests something novel about JEPA world models in general. And cite techniques they've learned along the way so others can learn from source material too.
Let me know if you have any questions!
I knew close to nothing about JEPA before reading your post, but i was under the impression this was supposed to be a contender for LLM's limitations, a way to do things "properly".
However, based on this article, i have the feeling that on the contrary, JEPA is filling a super niche problem, and is pretty much only trying to improve robotic motions in space.
Which of the two ideas is it ? Can JEPA technics be applied to a much more wide range of problems ? (like LLM and transformers)
JEPA can be applied to text as well in the sense you can use it to build text embedding models, freeze your JEPA text embedder, then train a decoder over the top of that.
OF course your decoder model could end up doing a lot of the heavy lifting if your decoder loss was based on text reconstruction methods (which almost all LLMs use for eval purposes)
For certain tasks, you do need pixel/token level reconstruction fidelity to do a lot of things. Like image generation or text generation.
For other tasks, having a coherent representation of the world that can be computed quickly and intuitively is more salient. I suspect the human mind works more like JEPA than like an autoregressive LLM. Human children train on far less data to achieve extreme fidelity on a lot of tasks generative reconstruction loss based AI/ML techniques struggle with.
I wanted to know how you made these animations, they look gorgeous!
Seconding that, gorgeous animations
javascript and webgl magic ;)
there's no webgl on the page though afaict?
The animations use custom vanilla JS, Canvas 2D, SVG, and a WebGL shader.
https://haddyalchaer.com/stories/jepa/figures.js
Based on the colors and how it looks, it seems like Astra generated them.
Thanks for doing this!
Basic question: couldn't something like probabilistic roadmaps [1] trivially solve this? Or am I missing something about the limitations of PRM's?
And if so, is that the best test for JEPA?
[1]:https://en.wikipedia.org/wiki/Probabilistic_roadmap
The theme is really hard to read. Dont mean to be a dick, I love the article. I tried a few browsers and my eyes really strained.
MCTS also solves push-T problems without training: https://github.com/olivkoch/gumbel-mcts
Whenever I read stuff like this, I wonder how could I get into this field?
Awesome read!
There's weird subtle moving rainbow color hues on this webpage on mobile
In a world where I'm increasingly being triggered by the fonts default Claude uses, I really liked the font used for the title here - it's Faculty Glyphic FYI.
as someone with an oled monitor that background effect gave me a small heart attack
We are doomed if most of the comments here are just talking about the website layout and not the contents of the article.
Anyway:
> The physics of pushing a block doesn't change if you shift, rotate or mirror the whole table. So when the graph has no states near a goal, it can take demonstration segments from elsewhere, transform them to fit the gap, and plan through those. With these transformed segments, success came back to 85%
This is an interesting technique to recover incomplete states or lost memories after demonstration. I'm guessing that the original LeWM does not have technique of transformative memories if I am correct?
AI; DR
If you’re going to use AI to generate text, keep it short and concise and meaty. I don’t hate reading AI, but I can’t read large walls of text for presumably some useful nugget of information in it.
https://www.pangram.com/history/2dd2b019-221a-4bee-bb6a-6b7e...
Cool
Website is borderline unusable on my phone. I suspect because of the animations? Either way not a great state of affairs
For anyone considering skipping because of this, it works flawlessly on my several year old iPhone.
There is a faint somewhat distracting background animation though I think…
Thanks! Glad to know it works for most people. I reduced the bg animation intensity.
Works perfectly fine for me on iOS 27.3 (with the exception of some glitching that was solved by disabling adaptive dark mode)
Sorry about that- will investigate. What OS/browser are you using?
Older intel macs initially have quite large blank areas of scroll while the figures later in the page render.
Much better after scrolling through and then reloading.
Looks perfectly fine for me
I'm calling it: JEPA is what unlocks general robotics. In a autonomy stack, LLMs are only useful at the planning level and above.