I'm thinking a lot about this recently as I'm building AQ.dev (Multiplayer coding harness). My instinct is to keep it simple and not trying to do too many shenanigans with these files and let engineers maintain control.
I feel like the Big Insight with Claude Code was 'give the AI access to tools, in fact to your actual laptop'. And of course 1.5 years later 'a harness to enable an agent to use my computer' has turned into the flagship product we expect from frontier labs
However. I don't think all the other 'Ideas' these harness-building guys have are all that universally applicable. Heck they are naysaying their own gearhead stack of skills.md and claude.md lately including in this video. But also I'm suspicious of this whole thing of spawning agents. For example "find every function in this codebase" is probably better done deterministically using a script that extracts function names rather than by spawning 20 agents to 'read' chunks of the code in token space. But it certainly racks up the token usage which is good for the person selling you inference...
And there is a lot missing in the actual harnesses frankly that isn't about more parallel agent ninjutsu. Like why isn't there a 'move this function from this file to this file' tool (copy/paste char range) and we have to see Claude/Codex/etc flail around rewriting huge chunks of code in token space
LSPs don't really do this. They're like a better way to grep for something (so they do address the previous thing I said about listing all function names etc.) But they often don't say (depending on the particular LSP setup etc) this function ends at this char and they definitely don't then provide the text editing tool
This is not accurate. Most LSPs do provide text editing tools, which the Language Server Protocol calls "code actions". These actions do things like: reorganizing imports, expanding macros, extracting a selection into a function, or converting one type of control flow into another (e.g. if to match). Common LSPs, like rust-analyzer and Ruff (Python) support code actions. Editors also expose code actions: Zed, for instance, exposes actions under the default keymap with the binding Command + .
Come on what are we talking about. Ruff is a linter. Can you show me any LSP server that can move an arbitrary function from file1.ts to file2.ts based on calling it on the command line?
They're not command line applications, they use JSON-RPC inline, and it's an interactive flow, so you need the client to actually persist state.
So what you'd see in the LLM traces, for example, is a couple back-and-forths using JSON-RPC. It absolutely works at the moment, and claude code will happily use it if everything is set up correctly, just tested it using https://github.com/typescript-language-server/typescript-lan...
I was just using it as an example of how dropping claude / gpt in a Linux shell and saying good luck and then giving interviews about spawning sub agents seems to overlook basic text editing primitives
But if you managed to move an arbitrary char range from one file to another using Claude Code talking to the TS LSP server let me know that would be enlightening.
I've had decent results telling agents to use emacs for structural editing operations. Pretty rare that I need to move a function across files verbatim but that should only be a few extra lines of elisp.
I don't get why Anthropic keeps saying how cool it is that they were able to rewrite bun in rust, I feel like if the model is really that good you wouldn't need a rewrite, you'd be able to create the runtime from scratch.
Code is a fantastic way to capture requirements, particularly iterative product discoveries over time. It's much easier to start with a manifestation of the requirements (and tests of invariants) than starting from base principles.
2 weeks rebuilding in Swift is absolutely crazy. Wonder if they're using production Opus 5 for this or their internal Mythos (if they even use it internally)
The TL;DR of the fragment at around 6:57 in the video, for those who don't want to watch, is that Boris recommends to delete all the customizations, CLAUDE.md, skills, and other stuff you had for older models, and try using the new models without all that, because the new models can surprise you.
Steering/corrective instructions are now supposed to go into memories - which are non-portable? How do we manage these documents for team members without everyone repeating themselves? And if you remove all these documents, how do you work with other models which might use them (nevermind that claude ignored AGENTS.md for the longest time).
I am on board with not putting stuff like "write clean code" into an agent file, or using plugins for tools that are now built into the harness. I don't see enough evidence to support models being significantly better at figuring out intent, or getting the assumption correct. I've always gotten better results (as ever) with very constrained instructions, vs "fix the install".
> Steering/corrective instructions are now supposed to go into memories - which are non-portable?
I’m sure this just intended to steer you to vendor lock-in. Remember, these are the same people who are so insecure/petty about their product that they won’t make it recognize the .agents/AGENTS.md standard.
Yes, that is the interesting part to me. Was anyone else familiar with the LLM-related terms "ablation" and "product overhang," prior to this? I was not.
If you were, please share the knowledge with us. Where can I learn more?
Yes, they keep surprising me that they still keep doing all of this: https://news.ycombinator.com/item?id=48962703 with no improvement despite all the marketing assurances that "hey you don't need to read code anymore"
Yeah, I'm going over a bunch of components a frontier model generated and while they "work" the code is quite shocking and will create significant maintenance burden. Not to mention they don't use any of our shared utilities and duplicate so much code.
Even for older models I was (and am) of the opinion that a lot of the context-overstuffing cruft people have been wrapping around their LLM usage is more of an RNG-impacting (but not always for the better) lucky charm rather than it being universally helpful.
And a lot of people latched on to it as a form of self-soothing.
"I may not write much code anymore, but I can still be an expert prompt engineer!"
Not really. The big change in newer models is the amount of RL relative to pretraining. RL makes the models stable by collapsing distributions around desired outcomes. The old tweaks like giving the agent a role ("You are an expert XYZ engineer") or stuffing the context with engineering idioms have effectively been RL'd into negligence, but they still waste tokens.
You can see this visually in older image generation models. Stable Diffusion 1.5 would produce wildly different images based on slight variations in prompt and seed, but the latest image gen models are nearly seed indifferent and can tolerate a decent amount of prompt tweaking while staying "consistent"
Salesman "recommends" you to continue wasting more tokens and on top of that, he "recommends" you to use only recursive loops with the latest and greatest models.
Finally he also "recommends" that you do not look at the code, or even understand it.
His "recommendations" are designed to get you to spend even more tokens and get you hooked on the Opus / Fable slot machine in order to extract as much money as possible from your wallets.
I've spent the last year building an app with Claude Code without knowing how to code. I'm still surprised at how much it gets wrong, even on the newest models. What makes it work for me is describing exactly what I want and then actually checking everything it gives back.
For me, I just don't trust Claude Code to write first-class professional code. I have used all frontier claude models and it always does sloppy job that needs to be fixed by codex. I have claude/codex setup and any execution must go to codex and all reviews as well.
30:08 "This means engineers can talk to users... stuff that's actually fun". Wow, he totally gets us. /s
31:09 He flubs the questioning. His first says "Raise your hand if 100% of your code uses agents." Then he says, "What about more than 50%". You can see the same audience members who raised the first time shrug and also raise the second time. He then says "Slightly less hands," which just means he sees what he wants to see. Don't believe this man.
So maybe SKILLS.md and CLAUDE.md should have model version pinned?
Somewhat like AWS CloudFormation templates start with something like TemplateVersion: 2017-10-01
But maybe that's old school thinking still.
I'm thinking a lot about this recently as I'm building AQ.dev (Multiplayer coding harness). My instinct is to keep it simple and not trying to do too many shenanigans with these files and let engineers maintain control.
I feel like the Big Insight with Claude Code was 'give the AI access to tools, in fact to your actual laptop'. And of course 1.5 years later 'a harness to enable an agent to use my computer' has turned into the flagship product we expect from frontier labs
However. I don't think all the other 'Ideas' these harness-building guys have are all that universally applicable. Heck they are naysaying their own gearhead stack of skills.md and claude.md lately including in this video. But also I'm suspicious of this whole thing of spawning agents. For example "find every function in this codebase" is probably better done deterministically using a script that extracts function names rather than by spawning 20 agents to 'read' chunks of the code in token space. But it certainly racks up the token usage which is good for the person selling you inference...
And there is a lot missing in the actual harnesses frankly that isn't about more parallel agent ninjutsu. Like why isn't there a 'move this function from this file to this file' tool (copy/paste char range) and we have to see Claude/Codex/etc flail around rewriting huge chunks of code in token space
> Like why isn't there a 'move this function from this file to this file' tool (copy/paste char range)
There is, you can install LSPs for a given language which act as just another tool the model can use to more efficiently manipulate code.
LSPs don't really do this. They're like a better way to grep for something (so they do address the previous thing I said about listing all function names etc.) But they often don't say (depending on the particular LSP setup etc) this function ends at this char and they definitely don't then provide the text editing tool
This is not accurate. Most LSPs do provide text editing tools, which the Language Server Protocol calls "code actions". These actions do things like: reorganizing imports, expanding macros, extracting a selection into a function, or converting one type of control flow into another (e.g. if to match). Common LSPs, like rust-analyzer and Ruff (Python) support code actions. Editors also expose code actions: Zed, for instance, exposes actions under the default keymap with the binding Command + .
source: https://microsoft.github.io/language-server-protocol/specifi... and https://microsoft.github.io/language-server-protocol/specifi...
Come on what are we talking about. Ruff is a linter. Can you show me any LSP server that can move an arbitrary function from file1.ts to file2.ts based on calling it on the command line?
They're not command line applications, they use JSON-RPC inline, and it's an interactive flow, so you need the client to actually persist state.
So what you'd see in the LLM traces, for example, is a couple back-and-forths using JSON-RPC. It absolutely works at the moment, and claude code will happily use it if everything is set up correctly, just tested it using https://github.com/typescript-language-server/typescript-lan...
What did you test exactly?
Look what I’m asking for is not complicated
Node cutpaste.js inputpath startcharnum endcharnum outputpath startcharnum
I was just using it as an example of how dropping claude / gpt in a Linux shell and saying good luck and then giving interviews about spawning sub agents seems to overlook basic text editing primitives
But if you managed to move an arbitrary char range from one file to another using Claude Code talking to the TS LSP server let me know that would be enlightening.
I've had decent results telling agents to use emacs for structural editing operations. Pretty rare that I need to move a function across files verbatim but that should only be a few extra lines of elisp.
I don't get why Anthropic keeps saying how cool it is that they were able to rewrite bun in rust, I feel like if the model is really that good you wouldn't need a rewrite, you'd be able to create the runtime from scratch.
Code is a fantastic way to capture requirements, particularly iterative product discoveries over time. It's much easier to start with a manifestation of the requirements (and tests of invariants) than starting from base principles.
Productivity theatre is when activity becomes celebrated over outcomes. It's absolutely everywhere in US tech culture.
Rust is a great tool for what they did with Bun. Why make a whole new tool when existing off the shelf tools can do the job?
I like this, it will be a really exciting benchmark to refer back to in a few years
2 weeks rebuilding in Swift is absolutely crazy. Wonder if they're using production Opus 5 for this or their internal Mythos (if they even use it internally)
If it's on the house (and they can probably turn up the token speed) why use anything other than the most powerful models?
The TL;DR of the fragment at around 6:57 in the video, for those who don't want to watch, is that Boris recommends to delete all the customizations, CLAUDE.md, skills, and other stuff you had for older models, and try using the new models without all that, because the new models can surprise you.
All of these harnesses should support pinning config files and tools to specific models or families of models.
It's really tiring to have to tweak everything with each model release and then watch those changes mess up cheaper models in the process.
Yup. That's pretty much the most interesting part.
Steering/corrective instructions are now supposed to go into memories - which are non-portable? How do we manage these documents for team members without everyone repeating themselves? And if you remove all these documents, how do you work with other models which might use them (nevermind that claude ignored AGENTS.md for the longest time).
I am on board with not putting stuff like "write clean code" into an agent file, or using plugins for tools that are now built into the harness. I don't see enough evidence to support models being significantly better at figuring out intent, or getting the assumption correct. I've always gotten better results (as ever) with very constrained instructions, vs "fix the install".
> Steering/corrective instructions are now supposed to go into memories - which are non-portable?
I’m sure this just intended to steer you to vendor lock-in. Remember, these are the same people who are so insecure/petty about their product that they won’t make it recognize the .agents/AGENTS.md standard.
Yes, that is the interesting part to me. Was anyone else familiar with the LLM-related terms "ablation" and "product overhang," prior to this? I was not.
If you were, please share the knowledge with us. Where can I learn more?
> because the new models can surprise you.
Yes, they keep surprising me that they still keep doing all of this: https://news.ycombinator.com/item?id=48962703 with no improvement despite all the marketing assurances that "hey you don't need to read code anymore"
Yeah, I'm going over a bunch of components a frontier model generated and while they "work" the code is quite shocking and will create significant maintenance burden. Not to mention they don't use any of our shared utilities and duplicate so much code.
I was using Opus 4.6 until 2 days ago with no CLAUDE.md or anything and it was great.
Tried out Opus 5 and it's been a super annoying experience out of the box.
Even for older models I was (and am) of the opinion that a lot of the context-overstuffing cruft people have been wrapping around their LLM usage is more of an RNG-impacting (but not always for the better) lucky charm rather than it being universally helpful.
And a lot of people latched on to it as a form of self-soothing.
"I may not write much code anymore, but I can still be an expert prompt engineer!"
Not really. The big change in newer models is the amount of RL relative to pretraining. RL makes the models stable by collapsing distributions around desired outcomes. The old tweaks like giving the agent a role ("You are an expert XYZ engineer") or stuffing the context with engineering idioms have effectively been RL'd into negligence, but they still waste tokens.
You can see this visually in older image generation models. Stable Diffusion 1.5 would produce wildly different images based on slight variations in prompt and seed, but the latest image gen models are nearly seed indifferent and can tolerate a decent amount of prompt tweaking while staying "consistent"
> is more of an RNG-impacting (but not always for the better) lucky charm rather than it being universally helpful.
I wrote this last year, it's still true:
--- start quote ---
https://dmitriid.com/prompting-llms-is-not-engineering
In reality these are just shamanic rituals with outcomes based on faith, fear, or excitement. Engineering it is not.
--- end quote ---
There are some prompts useful for the user like brainstorming [1] but on the whole it's nothing but lucky charms
[1] brainstorming from superpowers: https://github.com/obra/superpowers
CHANGELOG: surprise me
Salesman "recommends" you to continue wasting more tokens and on top of that, he "recommends" you to use only recursive loops with the latest and greatest models.
Finally he also "recommends" that you do not look at the code, or even understand it.
His "recommendations" are designed to get you to spend even more tokens and get you hooked on the Opus / Fable slot machine in order to extract as much money as possible from your wallets.
News at 10.
I've spent the last year building an app with Claude Code without knowing how to code. I'm still surprised at how much it gets wrong, even on the newest models. What makes it work for me is describing exactly what I want and then actually checking everything it gives back.
For me, I just don't trust Claude Code to write first-class professional code. I have used all frontier claude models and it always does sloppy job that needs to be fixed by codex. I have claude/codex setup and any execution must go to codex and all reviews as well.
Never create it in the first place
30:08 "This means engineers can talk to users... stuff that's actually fun". Wow, he totally gets us. /s
31:09 He flubs the questioning. His first says "Raise your hand if 100% of your code uses agents." Then he says, "What about more than 50%". You can see the same audience members who raised the first time shrug and also raise the second time. He then says "Slightly less hands," which just means he sees what he wants to see. Don't believe this man.