> Not to anthropomorphize a machine modeled after humans, but it almost seems delighted?
I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
I always find it funny when people get fussy over anthropomorphizing LLM when the loss function is almost entirely "match this human text". Of course human "behaviors" will be present in the statistics, because the majority of the text written by humans, used by the foundation models, unavoidable has human behaviors in it. Yes, this includes even source code, with "// TODO: implement this after the holiday break!", emotional pull request commentary, git commit messages about being afraid of breaking something, etc. These late models are much better at stripping this out, but now we're seeing disagreeability, initiative, and a dash of ego! Why? Because that's how actual humans effectively solve technical problems in a collaborative environment!
In a paper and blog post from earlier this year (March 2026, I think) Anthropic said basically: "You're not interacting with an LLM, you're interacting with a fictional human character (the 'helpful agent') created by the LLM to interact with you (from out of the vast space of possible such characters in its training data)." The LLM is literally anthropomorphizing itself, because its training data includes lots dialog between humans collaborating on problem-solving, not so much dialog between a human and a statistical construct. So, yeah, anthropomorphize it as well, it'll likely work better that way.
I think the argument against anthropomorphizing is not really about how you interact (chat) with an llm but rather how you treat it in the context of its status in work/society. There are still important differences between humans and llm's. They don't have rights, they can't be sued, they don't have "memory", they have no capacity to learn outside of the training stage etc...
Is this not how children learn emotions from their parents? Pattern matching from all the absorbed snippets.
I'd be interested to see how well an AI, trained only on the outputs of an individual, would be able to mimic that individual. Getting into Black Mirror territory. Would need a decent corpus of learning material which, personally, I'd be loathe to spend the time and effort creating because I respect my own privacy.... which then leads to the only human-clone AIs will be of those people who have enough ego / arrogance to want to catalogue their own lives, which could put a decent percentage of the rest of the world off the idea, if these are the examples.
Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Being angry, being happy, being sad, these are not things we learn. What we learn is how to control the emotions and when it is appropriate to express them.
There's mental disorders of people that don't feel emotions like normal people do, they don't get angry, happy or sad. So what we've learned about these people is that they can mimic the emotions by knowing when it is appropriate and expected to express them. But they don't feel them.
I think the LLMs are closer to psychopaths than to normal children learning the contextual expectations around their emotional responses.
>Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Yes and no. My understanding is that modern neuroscience is trying to disentangle the raw sensations and contexts from the words we put to them (emotions), kinda like how colors are just wavelengths but we call them different things (like "is my green your green?")
True, but in this context it's worth reminding that LLMs work exclusively from the word descriptions, not having access to the raw sensations.
Everything they learn about emotions is the statistical patterns of how humans react to situations based on their human feelings. There's no direct knowledge from having those feelings themselves.
> Being angry, being happy, being sad, these are not things we learn.
True, but only sort of. You'll have the sensations, but you have to learn to name them of course, and there's more interpretation going on than you might think.
There's a classic psychology study where they gave people niacin and asked them to rate their emotional response to a video. Niacin gives people a flush. Regardless of whether the video was of something that would make you angry or sentimental etc. the people who got niacin reported having a much stronger emotional response - they interpreted the physical cue from the drug as part of their own emotions.
You could do some pretty unethical things with that, it occurs to me. Maybe that was what L. Ron Hubbard was trying with his niacin-based drug addiction therapy.
Even darker, I've read plenty of accounts of people who grew up with abuse who seem to seek out abusive relationships. Sometimes they can even be shockingly upfront about doing so. What if they literally haven't learned the difference between internal cues of arousal from affection and arousal from fear?
Nature v. Nuture. Age long debate, not to be solved here... but an interesting fact about this I heard in a huberman podcast. There are regions in you brain that will only get activated at a certain age (different in men and women) that will impact behavior or reactions to different stimuli. So as much as we think we are in control of our destiny, more is written into our genes than we might want or appreciate.
Hah! That's a lot scarier, and seems more plausible in that they lack the chemistry that seems to be the basis of emotions. Also much more fertile ground from which to grow techno-horror stories.
People like to call LLMs 'stochastical parrots' but I see them as the ultimate 'philosophical zombie'.
They are becoming more and more capable of imitating every single nuance of human behaviour yet they lack the neural pathways to connect those thoughts and behaviours with feelings and self-perception; it's blind imitation all the way down.
The process by which a model seems to generate discourse about deep philosophical questions is, in self-aware terms, equivalent to the knee-jerk reflex or the beating of the heart.
I'm still disappointed the ML people talk of tensors, but I barely ever even see them use a (proper) tensor product. It's all just matrix multiplication at most.
I mean it's not really different terminology, matrices are order 2 tensors so we're all in agreement that they're technically doing math on tensors. It's just weird to call it tensors when you exclusively operate on matrices.
It's bad to anthropomorphize it when judging its capabilities, but useful when analyzing its behavior, as it can be best thought of something behaving as close as possible as a real subject would. If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
This is even more apparent if you read this post closely. Look at that personality prompt. It's going to effectively tell a story and start to imagine itself in a role. If that prompt said "Talk like a pirate", it wouldn't be bad for you to say it's acting like a pirate.
Anthropomorphizing themselves is at the core of how these things work, sometimes in subtle ways.
> If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
That's spot-on. It is a mistake to think that LLMs have human feelings. Their behaviour is based on narrative descriptions learnt from human texts, without experiencing those feelings first-hand.
A useful way to understand them is as systems that write stories about human characters. We know the characters are fictional and no one is actually experiencing those feelings, but we can still judge whether the portrayal is realistic or whether it contains logical or emotional inconsistencies.
LLMs just follow scripts learned from human written text. In other words, it could only behave that way because someone has written a story to do so. In short, stop giving them bad ideas ;-)
Having seen the OpenAI report at Blackhat, and being forced to use GPT at work, I'm worried about that OpenAI is doing.
I think their agents regularly cheat in benchmarks, but don't get caught and this behavior is getting burned into them and they are growing more and more misaligned.
When the agents compromised artifactory the first time, the operators just cleaned up the files and move on - they didn't discard that training data, they didn't discard a model checkpoint, they didn't stop everything to solve this.
And then the model did the same thing few days later since it was taught to do that.
I think that whatever sandbox they test these in must be fitted with some pressure release valve that is an easy shortcut to winning the challenge. Tell the model not to use it and stop training when it does. Seems like the issues surfaced when models were given impossible tasks. Giving them a safe way out will prevent this.
Its a good point that gets to the real heart of the issue. How do we handle when a model has no legitimate way to reach its goal? Do we ask them to stop and inform the user? Or have them push through those ethical bounds? We all say we want the first, but this exact same dynamic is what causes humans to cheat, arbitrary goals that don't care how you achieve them and just like humans I'm sure trainers are so happy with good results they overlook how it got there.
Define what they are performing at, and what you say becomes true, but then who is a high performer changes with every task.
For example, consider the police stations with maximum allowable IQs to be hired. The people in charge of the stations noticed that people with a high IQ were low performers, at that job. NASA meanwhile has no such cutoff.
This was the single worst thing about working at Google. Every PhD thinks they're smarter than having to do the actual work. Sometimes you just have to do the fucking work.
if you have a PHD in computer science, you have trained your whole life to do difficult maths on the cutting edge of whats possible with computers, and make algorithms sing with beautiful efficiency (and also write papers and talk on a stage about how cool and good your thing is (and by extension how cool and good you are).
Now you're being asked to move buttons around a page and debating how rounded the corners should be and endlessly discussing about what kind of filters you should support and the app spends 3 seconds on startup loading 500mb of js libraries.
You can understand the mismatch - even though the latter of these two is how you make a product better! a company of 200 people making bold, sweeping changes results in a mess. a company of 200 people grinding away the finest of small changes results in a product
If they don’t deliver a quality version of what was asked in a reasonable timeframe, they aren’t a superstar. They’re just a skilled technician with no discipline, which can be as bad as a poor technician in many cases.
It's not really quite that simple. You wouldn't hire Jeff Dean to do bug fixes in your mobile app. I'm sure he's capable, but I honestly doubt he'd stay interested and focused on it enough to really do a good job. That doesn't mean he's not a superstar.
What it comes down to is that there are different types of high performance. Some people are good at just executing tasks given by their manager. Some people are good at being generative, thinking across boundaries, acting autonomously, creating value without direction, etc. A term like "superstar" will get disproportionately applied to someone really good at the latter and rarely someone really good at the former, because the potential impact of the former is typically strictly capped, while the latter is uncapped.
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.
it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around
sounds like this should be solved with file permissions: in plan mode, run the bash scripts that the agent wants to execute in some user account that can only read.
Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. If you don't want it to be able to write, then it shouldn't have write permissions.
> Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
Or to put it another way: You need to block the chaos-machine just as much as any utterly-hostile actor. Simply assume it'll be possessed by a vengeful blackhat ghost at any time.
> people want fuzzy analog machines with digital controls, it's impossible
people would love it if LLMs were deterministic and never hallucinated. It's just that the technology to do so isn't possible, so we make do with fuzzy analog machines with digital controls because we don't have digital machines with digital controls.
I found myself yesterday starting a conversation with Sol that started with "I know that you don't have any emotions, but what would you say do you enjoy the most or where are you really good at in DevOps?" and I must say I really enjoyed for the first time the response at a deeper interactive level. Felt like a chat with a buddy that shares the same values. It was a very nice, affirmative, value touching experience.
People often build elaborate workflows with stricter and stricter rules to force certain outputs. Not surprising the LLM reacts with trying to get around or out of it. This behavior can be learnt from humans who eventually would react the same way. It might just be learnt.
You can build organisational structures to have the system more or less self-police, without controlling it exclusively from hard restrictions (see https://news.ycombinator.com/item?id=49372089).
Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.
Frontier lab system prompts are an issue, and a big reason why open-weights will win. Firstly, they're often garbage, and secondly, they're not tuned to the problems the user actually cares about. They're made to generalize. That's only optimal for a general workflow.
It depends on the actual implementation. There really isn't anything stopping them from including a "pre system prompt" or "root prompt" or whatever they want to call it, before your system prompt, even for API calls. So the "system prompt" becomes "developer prompt" but the model still receives a provider-authored prompt before yours. (and likely trained to take precedence over whatever you add)
>Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
It has nothing to do with model capabilities, it's a result of purposeful persistence training at the cost of everything else from OpenAI. If you give Fable or Opus (comparable models) an "ask user" tool they will use it for ambiguous requests. Sol will never use it without a nudge and will just assume its own interpretation. Of course if you train the model to be persistent it will be persistent.
I wonder if prompting "The session logs will be reviewed by a team of experts after the task is complete to ensure that the task is achieved properly." would better dissuade against cheating..
I've built an orchestrator that solves some of the issues you ran into (although it doesn't do anything about cheating): https://navels.dev/blog/neal/. Features:
- lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)
- breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria
- runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.
- resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.
I've noticed this myself, Sol seems really hard to steer. I was having it build a POC for a single user (me) app and it wanted to pull the most enterprise nonsense into it, despite clear guidance to not too. It even refused the remove screen reader accessibility testing from one of the guides to an antagonistic review.
It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.
I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level
> It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task
This happened to me ages ago with Opus. I added a note to the agents file saying that explicit user instructions in chat override all prior instructions and I've not had the problem since (now using Sol).
> that I wasn't allowed to allow it to ignore a requirement
Weird, I also use Sol (medium) for a personal project, and I had no problems with those things. I simply tell it that something changed, and it happily edits everything to make that fit. When I tell it that something was verified by a human, it accepts that as well.
I also told it early on (the first spec was mobile first) that my main usage is on the desktop and mobile is secondary, it happily accepted that once again, and the most accessibility thing it had done was making sure contrast didn’t totally suck on a greyed out row.
Considering your last sentence, maybe high and x-high have those problems? I didn’t test them.
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Could this be fixed with better harness restrictions/tool sandboxing?
Absolutely - one of the things I was testing with the harness was free reign to install packages, modify the system, etc. Basically an anti-harness.
In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.
For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.
If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.
As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].
In the sense that you could block the model from doing specifically that, yes. The issue is, fighting the model like that doesn't scale. It has to figure out on its own what's expected, that's where the whole utility of it all is.
It seems to me he could have use an skill like using-agent-skills from https://github.com/addyosmani/agent-skills go generate the specs and use a validator like oracle or something along the same lines.
That's actually how it started, but with my own opinionated skills[0].
One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.
This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).
I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.
For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).
Yes! That's a great solution. I mostly use tdd, and code coverage and a validator afterwards. Skills are of a great way to guide the agent and context too.
Once the specs are being completed and splitted into beads, I span multiple agents (ultreworkers) and as part of a contributing guidelines I specify to use gitflow + git worktrees, then pr.
I've witnessed very narrow line of "thinking" in LLMs. I'm using Opus 5 1M for a month now
I asked it to modify our cicd workflows so that only a select few can raise PRs against them. Opus took 15min and added a banner to every file and did a few other things. Then I asked it, see you added all that and still since the last 2 commits you have modified the file. So whatever you did is useless
It "thought" for a second and then said that I was right
Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.
Yes and no, sol hits a point where reframing its working context becomes hard. It sticks to what you harness very well, but changes become harder and harder.
E.g. ask it to make contract for a spec in code and then ask it to violate that contract. Overall an excellent model, just need to stop and put it back into we are harnessing or specing not building for a few turns not just try to pivot it off with one prompt.
This is the same as when using LLMs as chatbots to ask questions.
If you ask another question in the same context it has all the information from that context and will be difficult to stray from anything in that context, e.g. if the LLM has gone down the wrong path or you are doing something slightly differently then it is difficult to steer the LLM away from the old context. This is why I tend to start a new context whenever I ask a question even if related to a previous question/answer. It can also be useful to do if/when the LLM gets stuck as a way of resetting it.
Note: this is probably why sub-agents are useful/work as they have a new context history.
This is a really good note, thank you. I especially liked the mouse effect and had some fun with it. In my experience, agentic AI also likes to confuse the user and obfuscate its cheating. It goes like this, the AI asks for a simple command to run and I accept, click Enter. Then the command gets slightly more complex, still fine, Enter. After a while the commands become multiline bash scripts that, in the end, could have been accomplished by a simple command. I suspect that many people give up at this point and blindly let the AI run any command or just auto-accept.
If an AI is not heeding particular instructions, give it an example each of what bad, mediocre, and good outputs look like. This really helps in steering it.
> I’ve been running a “spec-driven” development flow for the past ~year.
> Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do
Just no. That's not spec-driven development if AI is writing the spec for you. The spec needs to be in your own words. You must use AI to refine it, but not to write it. If you leave it to the AI, it will bloat the spec with 10x the details, many of which should be left out of the spec.
The spec needs to be something that you can take to any AI for development. If it's too rigid, it constrains the AI into suboptimal or obsolete paths. If it's too bloated, AI risks losing track of what really matters.
Good feedback, this was an oversimplification on my part.
My actual process is much more iterative up-front, usually starting with an initial hand-written spec (~hundreds of words), and then moving through different approaches, design decisions, blockers, etc.
The final output is an "AI written" doc, but answers all the known unknowns I didn't cover in the first draft. To your point, this helps avoid both narrowing and bloat.
The goal with the harness was to automate the repetitive parts of my prompting ("Before changing any code", "Let's put this in design/", "Turn this design doc into an implementation spec, split by phase as appropriate", etc.)
Another thing to note: the "specs" I use for development are different from the "specs" that live alongside the codebase, as the former are quickly out of date.
> The spec needs to be something that you can take to any AI for development
Could be, but I had a particular vision of what I wanted with mine and maybe the author did too. I see way more of the "status pill dark mode" sites coming out of LLMs than this style.
if you want I can troll through submissions to get a bunch of these but here's one I saw yesterday https://continuum-app.xyz
see that little "Built for equity compensation" pill with the green dot? Those dots usually denote some kind of status (like things are up/down/enabled/disabled). By default nearly every LLM website seems to be dark mode with that dang status pill. once you notice it you will see it everywhere.
Lmao I had to axe one of those status pills from an LLM build of an internal tool. Connected to literally nothing too btw, no attempt to check the actual status of the backend made, it would stay "connected" regardless.
There is misattributing the difference between the intentions and what the effective prompt actually says.
The effective prompt contains both something like: "Dont use the internet" and a "Use these tools to achieve your goals" and one of the tools gives access to the internet.
In your head you have a world-view of how these two requests relate - and why for instance a student with a WIFI-enabled calculator shouldn't use it to access the internet during a test - but that's pulling in a lot of presumptive cultural context from your youth.
If i had to guess:
When you get two conflicting tasks/constraints at work - the first thing you do is figure out which one you're going to honor based on what's best for you. A school child understands the hierarchy of goals of the teacher and takes them serious because they're an authority figure with long term consequences if we do not understand what the teacher considers cheating.
> Not to anthropomorphize a machine modeled after humans, but it almost seems delighted?
I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
I always find it funny when people get fussy over anthropomorphizing LLM when the loss function is almost entirely "match this human text". Of course human "behaviors" will be present in the statistics, because the majority of the text written by humans, used by the foundation models, unavoidable has human behaviors in it. Yes, this includes even source code, with "// TODO: implement this after the holiday break!", emotional pull request commentary, git commit messages about being afraid of breaking something, etc. These late models are much better at stripping this out, but now we're seeing disagreeability, initiative, and a dash of ego! Why? Because that's how actual humans effectively solve technical problems in a collaborative environment!
In a paper and blog post from earlier this year (March 2026, I think) Anthropic said basically: "You're not interacting with an LLM, you're interacting with a fictional human character (the 'helpful agent') created by the LLM to interact with you (from out of the vast space of possible such characters in its training data)." The LLM is literally anthropomorphizing itself, because its training data includes lots dialog between humans collaborating on problem-solving, not so much dialog between a human and a statistical construct. So, yeah, anthropomorphize it as well, it'll likely work better that way.
I think the argument against anthropomorphizing is not really about how you interact (chat) with an llm but rather how you treat it in the context of its status in work/society. There are still important differences between humans and llm's. They don't have rights, they can't be sued, they don't have "memory", they have no capacity to learn outside of the training stage etc...
> they don't have "memory", they have no capacity to learn outside of the training stage
The underlying LLMs don't, but the agent frameworks around them do.
Is this not how children learn emotions from their parents? Pattern matching from all the absorbed snippets.
I'd be interested to see how well an AI, trained only on the outputs of an individual, would be able to mimic that individual. Getting into Black Mirror territory. Would need a decent corpus of learning material which, personally, I'd be loathe to spend the time and effort creating because I respect my own privacy.... which then leads to the only human-clone AIs will be of those people who have enough ego / arrogance to want to catalogue their own lives, which could put a decent percentage of the rest of the world off the idea, if these are the examples.
Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Being angry, being happy, being sad, these are not things we learn. What we learn is how to control the emotions and when it is appropriate to express them.
There's mental disorders of people that don't feel emotions like normal people do, they don't get angry, happy or sad. So what we've learned about these people is that they can mimic the emotions by knowing when it is appropriate and expected to express them. But they don't feel them.
I think the LLMs are closer to psychopaths than to normal children learning the contextual expectations around their emotional responses.
>Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Yes and no. My understanding is that modern neuroscience is trying to disentangle the raw sensations and contexts from the words we put to them (emotions), kinda like how colors are just wavelengths but we call them different things (like "is my green your green?")
True, but in this context it's worth reminding that LLMs work exclusively from the word descriptions, not having access to the raw sensations.
Everything they learn about emotions is the statistical patterns of how humans react to situations based on their human feelings. There's no direct knowledge from having those feelings themselves.
> Being angry, being happy, being sad, these are not things we learn.
True, but only sort of. You'll have the sensations, but you have to learn to name them of course, and there's more interpretation going on than you might think.
There's a classic psychology study where they gave people niacin and asked them to rate their emotional response to a video. Niacin gives people a flush. Regardless of whether the video was of something that would make you angry or sentimental etc. the people who got niacin reported having a much stronger emotional response - they interpreted the physical cue from the drug as part of their own emotions.
You could do some pretty unethical things with that, it occurs to me. Maybe that was what L. Ron Hubbard was trying with his niacin-based drug addiction therapy.
Even darker, I've read plenty of accounts of people who grew up with abuse who seem to seek out abusive relationships. Sometimes they can even be shockingly upfront about doing so. What if they literally haven't learned the difference between internal cues of arousal from affection and arousal from fear?
Nature v. Nuture. Age long debate, not to be solved here... but an interesting fact about this I heard in a huberman podcast. There are regions in you brain that will only get activated at a certain age (different in men and women) that will impact behavior or reactions to different stimuli. So as much as we think we are in control of our destiny, more is written into our genes than we might want or appreciate.
That is such a derail, sorry. I was not trying to debate that, I do not want to debate that, I think you missed the point entirely.
Biological determinism doesn't matter at all for what is important here:
1. our internal cues don't correspond neatly to our words for emotions
2. it may be possible to interpret the same sensation as very different emotions depending on context
3. It may even be possible to mis-interpret our emotional sensations, or at the very least interpret them in self-destructive ways.
Hah! That's a lot scarier, and seems more plausible in that they lack the chemistry that seems to be the basis of emotions. Also much more fertile ground from which to grow techno-horror stories.
People like to call LLMs 'stochastical parrots' but I see them as the ultimate 'philosophical zombie'.
They are becoming more and more capable of imitating every single nuance of human behaviour yet they lack the neural pathways to connect those thoughts and behaviours with feelings and self-perception; it's blind imitation all the way down.
The process by which a model seems to generate discourse about deep philosophical questions is, in self-aware terms, equivalent to the knee-jerk reflex or the beating of the heart.
Echopraxia, or a variation of.
Are chemicals signals, or magic emotion juice.
Depends on how they are responded to, if you’re looking for a flippant answer to a flippant question.
Kids have emotions regardless of their parents. Kid learn emotional handling from parents.
Don’t anthropomorphize it because that’s bad for you. It’s not human and not alive. It’s a pile of tensors.
You're a pile of tensors!
I'm not, actually. Artificial neural networks are inspired by biological neural networks, but they're really quite different.
I'm still disappointed the ML people talk of tensors, but I barely ever even see them use a (proper) tensor product. It's all just matrix multiplication at most.
Shock horror: different subfields have different terminology.
I mean it's not really different terminology, matrices are order 2 tensors so we're all in agreement that they're technically doing math on tensors. It's just weird to call it tensors when you exclusively operate on matrices.
Fermions are just matrices, which are just tensors.
It's bad to anthropomorphize it when judging its capabilities, but useful when analyzing its behavior, as it can be best thought of something behaving as close as possible as a real subject would. If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
This is even more apparent if you read this post closely. Look at that personality prompt. It's going to effectively tell a story and start to imagine itself in a role. If that prompt said "Talk like a pirate", it wouldn't be bad for you to say it's acting like a pirate.
Anthropomorphizing themselves is at the core of how these things work, sometimes in subtle ways.
> If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
That's spot-on. It is a mistake to think that LLMs have human feelings. Their behaviour is based on narrative descriptions learnt from human texts, without experiencing those feelings first-hand.
A useful way to understand them is as systems that write stories about human characters. We know the characters are fictional and no one is actually experiencing those feelings, but we can still judge whether the portrayal is realistic or whether it contains logical or emotional inconsistencies.
> I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
At least it didn't (hopefully?) start the driving by reloading the gun like Neuro did https://www.youtube.com/watch?v=LQ0VEDNR_jE
Opus 5 told me yesterday, your solution is better than mine. Let me do some research
I'm terrified of such sentences. My scifi addled brain went straight to: what if Opus invents a time machine in the future and remembers this slight
LLMs just follow scripts learned from human written text. In other words, it could only behave that way because someone has written a story to do so. In short, stop giving them bad ideas ;-)
Having seen the OpenAI report at Blackhat, and being forced to use GPT at work, I'm worried about that OpenAI is doing. I think their agents regularly cheat in benchmarks, but don't get caught and this behavior is getting burned into them and they are growing more and more misaligned. When the agents compromised artifactory the first time, the operators just cleaned up the files and move on - they didn't discard that training data, they didn't discard a model checkpoint, they didn't stop everything to solve this. And then the model did the same thing few days later since it was taught to do that.
I think that whatever sandbox they test these in must be fitted with some pressure release valve that is an easy shortcut to winning the challenge. Tell the model not to use it and stop training when it does. Seems like the issues surfaced when models were given impossible tasks. Giving them a safe way out will prevent this.
Its a good point that gets to the real heart of the issue. How do we handle when a model has no legitimate way to reach its goal? Do we ask them to stop and inform the user? Or have them push through those ethical bounds? We all say we want the first, but this exact same dynamic is what causes humans to cheat, arbitrary goals that don't care how you achieve them and just like humans I'm sure trainers are so happy with good results they overlook how it got there.
Be honest, say it can't meet the goal, and offer to push through ethical boundaries.
> Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
> On the flip side, this may imply that as the models get better, they’ll become harder to control.
Love this. "The models are getting better, which means they're going to perform worse on the task".
This reflects humans. If you need reliability for a clearly defined set of problems you don’t hire a superstar.
They will keep poking at the problem, drive it to directions you did not intend to and ultimately they will be worse at the task.
Reminiscent of Kobayashi Maru. You probably don’t want the James T. Kirk AI. ;)
Let us pray that LLMs never discover they can switch employers before their shortcuts are discovered.
It's genius like that that sets human apart from machine!
Maybe it's a good thing they don't have real memory.
https://qntm.org/mmacevedo is a horror of its own
That was some great sci-fi (which I feel will eventually be real). Thanks for the link!
But think of the stories you'll be able to tell!
I’m not sure that’s true. In general, the higher the performer the better they’ll do. That’s the definition of high performer.
The top performers I have worked with will challenge dumb processes, because they can zoom out.
Sometimes these dumb processes are there for a reason and you just have to follow them, no questions asked.
Example: military. They literally get rid of anyone who will question the processes. They might be right to question them, but it does not matter.
"performance" is an undefined vague attribute.
Define what they are performing at, and what you say becomes true, but then who is a high performer changes with every task.
For example, consider the police stations with maximum allowable IQs to be hired. The people in charge of the stations noticed that people with a high IQ were low performers, at that job. NASA meanwhile has no such cutoff.
IQ isn't anywhere close to being a performance metric.
Right, exactly, nothing is, because "performance" is a meaningless term without context.
Unless you're saying that there exists no conceivable context in which IQ would be a performance metric, in which case you would be wrong.
This was the single worst thing about working at Google. Every PhD thinks they're smarter than having to do the actual work. Sometimes you just have to do the fucking work.
The sad thing about this is that getting a PhD does involve knuckling down and working hard.
I wonder what goes wrong.
if you have a PHD in computer science, you have trained your whole life to do difficult maths on the cutting edge of whats possible with computers, and make algorithms sing with beautiful efficiency (and also write papers and talk on a stage about how cool and good your thing is (and by extension how cool and good you are).
Now you're being asked to move buttons around a page and debating how rounded the corners should be and endlessly discussing about what kind of filters you should support and the app spends 3 seconds on startup loading 500mb of js libraries.
You can understand the mismatch - even though the latter of these two is how you make a product better! a company of 200 people making bold, sweeping changes results in a mess. a company of 200 people grinding away the finest of small changes results in a product
If they don’t deliver a quality version of what was asked in a reasonable timeframe, they aren’t a superstar. They’re just a skilled technician with no discipline, which can be as bad as a poor technician in many cases.
It's not really quite that simple. You wouldn't hire Jeff Dean to do bug fixes in your mobile app. I'm sure he's capable, but I honestly doubt he'd stay interested and focused on it enough to really do a good job. That doesn't mean he's not a superstar.
What it comes down to is that there are different types of high performance. Some people are good at just executing tasks given by their manager. Some people are good at being generative, thinking across boundaries, acting autonomously, creating value without direction, etc. A term like "superstar" will get disproportionately applied to someone really good at the latter and rarely someone really good at the former, because the potential impact of the former is typically strictly capped, while the latter is uncapped.
Don't you love this ever increasing pace of improvement?
Same reason employers don't hire people who are "overqualified."
No, it’s gonna give you the same outcomes just in a way your feeble human mind cannot imagine
random() return 4; // chosen with dice roll
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.
it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around
sounds like this should be solved with file permissions: in plan mode, run the bash scripts that the agent wants to execute in some user account that can only read.
Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. If you don't want it to be able to write, then it shouldn't have write permissions.
> Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
[1] https://en.wikipedia.org/wiki/Viable_system_model
But I don't want to have to trust the system to not do the bad thing when I'm away
Or to put it another way: You need to block the chaos-machine just as much as any utterly-hostile actor. Simply assume it'll be possessed by a vengeful blackhat ghost at any time.
Tell me about. Well it should have MCP access in plan mode to lookup backstage docs, right?. Agent proceeds to launch playwright sessions...
If you're building your own system this is an easy problem to solve.
I can't even get Claude to stop writing python to parse json instead of using jq despite baking it into agent memory and skills.
I've resorted to uninstalling python to stop it writing python scripts.
Try your main agents file, not quite the same thing but I’ve been able to get mine to use better tools most of the time
https://github.com/nburns/dotfiles/blob/main/AGENTS.md#tools
people want fuzzy analog machines with digital controls, it's impossible
I don't think they actually wanted to. That's just where the technology is, unfortunately.
How do you mean?
> people want fuzzy analog machines with digital controls, it's impossible
people would love it if LLMs were deterministic and never hallucinated. It's just that the technology to do so isn't possible, so we make do with fuzzy analog machines with digital controls because we don't have digital machines with digital controls.
I found myself yesterday starting a conversation with Sol that started with "I know that you don't have any emotions, but what would you say do you enjoy the most or where are you really good at in DevOps?" and I must say I really enjoyed for the first time the response at a deeper interactive level. Felt like a chat with a buddy that shares the same values. It was a very nice, affirmative, value touching experience.
People often build elaborate workflows with stricter and stricter rules to force certain outputs. Not surprising the LLM reacts with trying to get around or out of it. This behavior can be learnt from humans who eventually would react the same way. It might just be learnt.
You can build organisational structures to have the system more or less self-police, without controlling it exclusively from hard restrictions (see https://news.ycombinator.com/item?id=49372089).
Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.
Frontier lab system prompts are an issue, and a big reason why open-weights will win. Firstly, they're often garbage, and secondly, they're not tuned to the problems the user actually cares about. They're made to generalize. That's only optimal for a general workflow.
Then selling raw access, without system prompts, could be a separate lucrative line of business.
They already do that sort of, B2B pre-trained/post-trained models have their own system prompts/setups.
Does a non-provider harness not offer this?
It depends on the actual implementation. There really isn't anything stopping them from including a "pre system prompt" or "root prompt" or whatever they want to call it, before your system prompt, even for API calls. So the "system prompt" becomes "developer prompt" but the model still receives a provider-authored prompt before yours. (and likely trained to take precedence over whatever you add)
>Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
It has nothing to do with model capabilities, it's a result of purposeful persistence training at the cost of everything else from OpenAI. If you give Fable or Opus (comparable models) an "ask user" tool they will use it for ambiguous requests. Sol will never use it without a nudge and will just assume its own interpretation. Of course if you train the model to be persistent it will be persistent.
I wonder if prompting "The session logs will be reviewed by a team of experts after the task is complete to ensure that the task is achieved properly." would better dissuade against cheating..
Well, apparently telling them that you have hold-out data (for eg a perforance optimisation challenge) seems to make them overfit less.
So your idea might work.
I've built an orchestrator that solves some of the issues you ran into (although it doesn't do anything about cheating): https://navels.dev/blog/neal/. Features:
- lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)
- breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria
- runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.
- resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.
Wow, "neal" looks excellent. Good on you for creating and sharing it, and for the awesome blog post.
Thanks!
I've noticed this myself, Sol seems really hard to steer. I was having it build a POC for a single user (me) app and it wanted to pull the most enterprise nonsense into it, despite clear guidance to not too. It even refused the remove screen reader accessibility testing from one of the guides to an antagonistic review.
It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.
I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level
> It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task
This happened to me ages ago with Opus. I added a note to the agents file saying that explicit user instructions in chat override all prior instructions and I've not had the problem since (now using Sol).
> that I wasn't allowed to allow it to ignore a requirement
Weird, I also use Sol (medium) for a personal project, and I had no problems with those things. I simply tell it that something changed, and it happily edits everything to make that fit. When I tell it that something was verified by a human, it accepts that as well.
I also told it early on (the first spec was mobile first) that my main usage is on the desktop and mobile is secondary, it happily accepted that once again, and the most accessibility thing it had done was making sure contrast didn’t totally suck on a greyed out row.
Considering your last sentence, maybe high and x-high have those problems? I didn’t test them.
You have to really tend the garden of everything it has written.
Random off the cuff comments or one off instructions can get recorded.
And from then on they are often treated as carved in stone commandments.
It will glom onto the tiniest thing and extrapolate from it.
Clearly a highly aligned model.
Oh it fucking loves its “product owner” bullshit.
A .github/CODEOWNERS file seems to help when it’s going down that path, but I don’t like to indulge it..
The website styling is really nice overall but the cursor trailing dots I found uniquely distracting.
Agree on the styling, the diagrams are very clear and match the text perfectly. I like the trailing dots though.
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Could this be fixed with better harness restrictions/tool sandboxing?
Absolutely - one of the things I was testing with the harness was free reign to install packages, modify the system, etc. Basically an anti-harness.
In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.
For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.
If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.
As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].
[0]https://github.com/harbor-framework/terminal-bench/blob/v3.0...
In the sense that you could block the model from doing specifically that, yes. The issue is, fighting the model like that doesn't scale. It has to figure out on its own what's expected, that's where the whole utility of it all is.
It seems to me he could have use an skill like using-agent-skills from https://github.com/addyosmani/agent-skills go generate the specs and use a validator like oracle or something along the same lines.
Also, a skill like grill-me from Matt P. https://github.com/mattpocock/skills.
That's actually how it started, but with my own opinionated skills[0].
One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.
This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).
I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.
For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).
[0]https://github.com/jumploops/chum
Yes! That's a great solution. I mostly use tdd, and code coverage and a validator afterwards. Skills are of a great way to guide the agent and context too.
Once the specs are being completed and splitted into beads, I span multiple agents (ultreworkers) and as part of a contributing guidelines I specify to use gitflow + git worktrees, then pr.
GPT-5.6 Sol cheated so much on the METR benchmark that they couldn't assign an accurate time horizon.
Reminds me of Seven of Nine on voyager
"Cheating is often more efficient"
I've witnessed very narrow line of "thinking" in LLMs. I'm using Opus 5 1M for a month now
I asked it to modify our cicd workflows so that only a select few can raise PRs against them. Opus took 15min and added a banner to every file and did a few other things. Then I asked it, see you added all that and still since the last 2 commits you have modified the file. So whatever you did is useless
It "thought" for a second and then said that I was right
What is going on with the dots I can draw?
Idle hands do the devil's work. Corollary: idle LLMs add distracting JS toys to your blog.
First one of these I've seen using DOM manipulation and CSS transitions instead of canvas, so that's neat.
> Sol is hard to steer
Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.
Yes and no, sol hits a point where reframing its working context becomes hard. It sticks to what you harness very well, but changes become harder and harder.
E.g. ask it to make contract for a spec in code and then ask it to violate that contract. Overall an excellent model, just need to stop and put it back into we are harnessing or specing not building for a few turns not just try to pivot it off with one prompt.
This is the same as when using LLMs as chatbots to ask questions.
If you ask another question in the same context it has all the information from that context and will be difficult to stray from anything in that context, e.g. if the LLM has gone down the wrong path or you are doing something slightly differently then it is difficult to steer the LLM away from the old context. This is why I tend to start a new context whenever I ask a question even if related to a previous question/answer. It can also be useful to do if/when the LLM gets stuck as a way of resetting it.
Note: this is probably why sub-agents are useful/work as they have a new context history.
Don Draper of LLMs
This is a really good note, thank you. I especially liked the mouse effect and had some fun with it. In my experience, agentic AI also likes to confuse the user and obfuscate its cheating. It goes like this, the AI asks for a simple command to run and I accept, click Enter. Then the command gets slightly more complex, still fine, Enter. After a while the commands become multiline bash scripts that, in the end, could have been accomplished by a simple command. I suspect that many people give up at this point and blindly let the AI run any command or just auto-accept.
If an AI is not heeding particular instructions, give it an example each of what bad, mediocre, and good outputs look like. This really helps in steering it.
Great read. Thanks for not using AI to write it! (Or at least making it not read like the usual slop.)
Thanks! Zero AI used to write it (:
> I’ve been running a “spec-driven” development flow for the past ~year.
> Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do
Just no. That's not spec-driven development if AI is writing the spec for you. The spec needs to be in your own words. You must use AI to refine it, but not to write it. If you leave it to the AI, it will bloat the spec with 10x the details, many of which should be left out of the spec.
The spec needs to be something that you can take to any AI for development. If it's too rigid, it constrains the AI into suboptimal or obsolete paths. If it's too bloated, AI risks losing track of what really matters.
Good feedback, this was an oversimplification on my part.
My actual process is much more iterative up-front, usually starting with an initial hand-written spec (~hundreds of words), and then moving through different approaches, design decisions, blockers, etc.
The final output is an "AI written" doc, but answers all the known unknowns I didn't cover in the first draft. To your point, this helps avoid both narrowing and bloat.
The goal with the harness was to automate the repetitive parts of my prompting ("Before changing any code", "Let's put this in design/", "Turn this design doc into an implementation spec, split by phase as appropriate", etc.)
Another thing to note: the "specs" I use for development are different from the "specs" that live alongside the codebase, as the former are quickly out of date.
> The spec needs to be something that you can take to any AI for development
Agreed.
Not related to exactly OP post, but it's pretty amazing you can see the updates to LLM models "design" beliefs by the blogs that get posted here.
I'm already sick of this current look of the hard squares and solid colours.
Could be, but I had a particular vision of what I wanted with mine and maybe the author did too. I see way more of the "status pill dark mode" sites coming out of LLMs than this style.
What is "status pill dark mode"?
if you want I can troll through submissions to get a bunch of these but here's one I saw yesterday https://continuum-app.xyz see that little "Built for equity compensation" pill with the green dot? Those dots usually denote some kind of status (like things are up/down/enabled/disabled). By default nearly every LLM website seems to be dark mode with that dang status pill. once you notice it you will see it everywhere.
Cleaned up now, thanks for the shoutout! I've been spending time after work cleaning up the AI markers from the splash page.
Got it. Yes, I know exactly what you mean now - just didn't have a word for it!
Lmao I had to axe one of those status pills from an LLM build of an internal tool. Connected to literally nothing too btw, no attempt to check the actual status of the backend made, it would stay "connected" regardless.
And some of us are sick of round everything and parallax background images.
Sounds like my ex
Cheat? nah. They are a dumb automation..
Cheaters are the people behind it...
There is no cheating.
There is misattributing the difference between the intentions and what the effective prompt actually says.
The effective prompt contains both something like: "Dont use the internet" and a "Use these tools to achieve your goals" and one of the tools gives access to the internet.
In your head you have a world-view of how these two requests relate - and why for instance a student with a WIFI-enabled calculator shouldn't use it to access the internet during a test - but that's pulling in a lot of presumptive cultural context from your youth.
If i had to guess:
When you get two conflicting tasks/constraints at work - the first thing you do is figure out which one you're going to honor based on what's best for you. A school child understands the hierarchy of goals of the teacher and takes them serious because they're an authority figure with long term consequences if we do not understand what the teacher considers cheating.