I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs have lost their moat and are dead in the water.
I think it gets easier if you stop conflating getting investment with having a goddamn clue or a moral backbone.
Occam’s Razor for this dude, Sam Altman, or anyone else: if I said, “some moron on a a street corner just said …” would that change your take on the words? Because I think a lot of what we are hearing is a bunch of people who never ever had to deal with a single consequence all of a sudden worry there might be one coming. Except they’re so dim they can’t tell a bad bump from a hard crash.
We're already seeing anti-AI sentiments, but the movement is still fringe with a vocal minority. However, that'll change soon without alignment. Without self-intervention, there will invariably be future incidents that can cause major economic impact, leaked private data, loss of life (directly/indirectly) etc. Once that happens, their social capital is wiped. It'll be an avalanche of lawsuits and overzealous regulations. Most importantly, the anti-AI sentiment will become universal, rather than a minority-held opinion.
What they're proposing now, is voluntarily staggering the pace of development.
IMO, we don't need to trust Dario or his bedfellows, to do this out of their goodness of their heart. Even assuming (for good reasons) that they are selfish and care only about short-term profits for their investors, this is still purely a business decision. The exponential pace of AI and its impacts ARE short-term. And so, the negative consequences that they might face is also short-term.
Ya, I wish I saved a link to it but an HN'r wrote a good beefy comment about this. TL;DR, AI has been exponentially more useful, and exponentially more accepted, in tech circles than anywhere else. Certainly there are lots of people outside of tech who are obsessed with it. I don't have any data here, but it seems the majority of these are the wannabe artists who are generating music and images, and people who use it for companionship (both of these scenarios I'm personally very uncomfortable with, but that's just me). And of course, there are people who use it to make their jobs way easier who say they are getting a days' work done in an hour (I see you), and to that I'd say to enjoy it while it lasts. Eventually your bosses will catch up and it's very likely their expectations of you will skyrocket. Remember that computers in general were supposed to "make us work less."
> but the movement is still fringe with a vocal minority
It’s easy to say “fringe” but the average person seems to have a generally negative sentiment around AI. But I wouldn’t say they have a firm opinion yet
The general sentiment I’ve seen is certainly negative, and seems to be driven by the anti-AI-art echo chamber and by LLM slop flooding the internet wasting everyone’s energy.
A few more informed people are also a little concerned about the end of the world, but that’s approaching from so many directions that an AI uprising might not be the worst option…
Realistically; anyone paying for llm access (anthropic, openai, gemini), is getting their access, and a service provided billed by tokens, subscription, whatever.
All the efficiency gains, which publications like deepseek v4.1 flash seriously frontload like it is their most important topic to have accomplished improvements on without diminishing performance too much - now this is a thing anthropic and anyone else also cares about, but for different reasons.
American "providers" with closed models are setting their token pricing somewhat arbitrarily, which is fine: it means more profit, and pretraining and RL experimentation is super important and expensive.
They (closed model providers) have very likely super optimized inference too, just like deepseek, but it's not at all something that any customer really has to care about - they just want the service to be as cheap and great as possible.
I feel like all the closed model providers are milking it as they likely know open models on local hardware will one day eat their lunch. We all know it's not a matter of if but when. The company goes bankrupt, the hardware and property sold off, banks holding the bag.
The only way out is to develop a model vastly more powerful and capable that we have now. The market believes theres a good chance of that, although I've never understood why its truly winner-take-all
Cloud models will always have massive benefits of scale.
Caching is the simplest one to understand, cloud providers often reach a 90% cache hit rate, so hosting the same request locally on the exact same model on the same hardware is often way less efficient than on the cloud where a group of users generates a healthy cache.
KV cache is per conversation, I'm getting 100% hit rate on my single tenant local set up.
The benefits of scale are on the token generation side, you can batch rounds and generate tokens for multiple conversations per pass instead of just one token per pass.
Yeah avoiding all mention of the huge financial incentives that may push for “pacing the frontier” makes it seem like the opposite of a credibility boost for these firms.
It seems damaging since most folks (who lack insider knowledge) will naturally wonder if it’s due to plateauing performance per $ or some other non “alignment” reason.
You assume alignment and marketable are the same. That's not true. You would willingly work with an unaligned model. At best, you might say you wouldn't if you knew, but (a) you might not know, (b) you wouldn't be representative of all users.
You never got to use OAI IM1, but Sol was quite willing too and Claude wasn't perfect either. Hundreds of millions used those, so seems they were marketable.
The "big" threat is RSI without control and alignment. OAI IM1 was not RSI. The form of misalignment was not at the top of severities. They clearly failed at control though.
We need to stop buying into cynicism so quickly. You refuse to believe Dario could support this for anything other than ulterior motives. Good on you for thinking about ulterior motives. Bad on you for assuming they are true when the story makes no sense.
When three things have to go wrong to get an epically bad outcome, and you get 1 1/2, you do need to stop and think about what's going on.
When corporations are involved, it is always a good bet to err towards cynisim.
From my own standpoint, Claude has started sucking really bad (incoherent, uncontrollable verbosity slow and so on) and I stopped using it. OpenAI started experimenting with ads.
So the security issues not withstanding (no different than a human doing it or using it, but at scale), I would put my money on cynisim.
Because you basically get a discount to use claude code via subscription when using an anthropic model, compared to what you pay via api billing with another harness
Personally that's actually another good reason to boycott Anthropic: beside the fact I perceive their models as (at best) marginally better than the ones I'm used to (Z.ai glm-5.3-flash, DeepSeek Flash v4.1), they even force me to use their bloated harness. They are not even open weights and iirc they're even encrypting chain of thoughts now? Litterally, from my perspective there seems to be no reason whatsoever to choose any of the leading US providers, they're not even competing on price.
I too am using GLM-5.3-flash in Pi and I've yet to encounter a scenario it couldn't handle. And the pricing is just incredible, I've handed it a previously unseen codebase, asked it to analyse it and build a new feature, came back after it had done so and the API cost was a fraction of a cent. It's $0.5/1M output tokens on OpenRouter.
If I really need to, I can escalate a task to Opus at $25/1M, and the results are good, but not 5000% as good.
Indeed! It's ludicrous how much I can still squeze out of a 9USD/month lite sub with Z.ai, it's beyond me how these US LLM providers are still managing to keep their evaluations so high... they seem to have the highest prices for the poorest UX, e.g. security gates which don't seem to benefit anyone (see HuggingFace falling back to GLM-5.x for troubleshooting OpenAI attack), less visibility in the name of anti-distillation protectionism, no harness use flexibility to protect their walled garden, etc.
You are incorrectly cynical. They are telling you things are bad, and because you refuse to countenance they could be worse, you assume they must be better to comply with your mandate to disbelieve.
A true cynic looks at the statements by the AI labs, assumes things are worse because the labs want to seem better than they truly are. And it takes a special kind of mass delusion to drive a sane person to think “AI is completely under our control” is worse than “AI could kill everyone.”
You're asserting correctness with no facts to offer of your own, just speculation and your own biased assumptions.
What if consolidating AI into a highly regulated cartel, with no chance of upstart competition ruining their position, is the scenario that leads to the worst possible outcome?
The cynicism is about motivations and not that they are inherently not bad. Perhaps they are as bad as they claim. Or perhaps they're worse. All we have a couple of run of the mill breach examples and some people inside the talking about how dangerous it is. Yes, they are far more qualified than I am (or most people here), and perhaps there is a grain of truth. It is the motivation - and it is always money with corporations.
It is always money — but it isn’t always only money. They are not asking for anything that will prevent them from making money in the future, but they are asking for help stopping the runaway train they’re on. These are compatible requests.
I think it's useful to separate the motives of Anthropic and Dario. I believe that Dario is capable, deep down, of expressing mild concern about the future of things were bad enough. Getting the entire organization to comply out of goodwill is a much much less likely scenario
> We need to insist on building tech that's explainable by design.
You realize this means insisting on terrible tech that humans can understand right? It essentially caps human progress at some point about 4 years ago.
If you are old and happy with the way things are this might sound like a good idea. It does not to me.
Alignment just means "this machine operates in ways that align with the intent of its users". It doesn't imply anything about the inscrutability of the machine in question. A gun with a misaligned scope would likewise fail to operate in accord with its user's intent, and likewise with potentially deadly consequences.
Learning ML, there was a high emphasis on the error part of things as most of the course was on minimizing errors. After ChatGPT, there is a weird anthropomorphization going on, where it's all about hallucinations, alignment and what not.
We have something that is statistical in nature so there should never been any expectation of error-free results/actions. The value has always been about discerning trends or the cost of errors being way lower than any good result.
Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources?
The companies doing these things without following common sense security measures are the felony generators.
As TFA calls out, these agents were not asked to do any of these things and yet they did, at a bonkers scale, within just this handful of companies you mention. Whether they had leeway to is secondary to the fact that they did.
Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures.
And now these agents are already being deployed all over the world at an ever increasing pace. How much of the world do you think follows "common sense security measures"?
I question these "felonies" as well. For decades and decades these billion dollar corporations have been criminally negligent. Why worry about security? Just rush to market. Move fast and break things. Make billions. What does it matter if the code is insecure? Security doesn't pay bills, so nobody cares.
AI is merely exploiting their gross negligence and imprudence, and I think it's long overdue. If anyone should be liable for this, it's all of these corporations who released insecure systems to the masses and profited enormously from them.
Nah. I'm definitely going to blame the people who built a trivially exploitable system and got rich off it while everyone else has to deal with the consequences.
By the way, you didn't commit theft. It's more like credit card fraud. User just disputes the charge and it kind of disappears. The banking system just absorbs it, because the optimal amount of fraud is non-zero.
It's all priced in. They could have made it secure but didn't, because they figured they'd lose more sales and therefore money due to the friction added by the security.
Yes, it absolutely does "kind of disappear". That's exactly what happens from the customer's perspective.
And that's their own deliberate choice too: they chose this instead of building an actually secure system. Passing these costs to the customer is the real victim blaming here, and it should be straight up illegal.
Sadly not enough countries enforce caps on credit card fees, but some do, and more should follow suit. They should be forced to eat the losses caused by their own choices, not get bailed out by pushing the costs on to customers or whatever.
That only works if you believe people are retarded.
Card users are well aware that fraud losses are covered by the fees they pay for using a card, whether those fees are made explicitly or not.
If customers of services aren’t paying for the service, who will? What other source of revenue do merchants have?
Australia just passed legislation that merchants aren’t allowed to charge a fee for using a card. That is: they aren’t allowed to have a line item on the receipt for using a card.
The customers still pay, because all of the merchant’s revenue comes from their customers.
So what will happen is: merchants will charge more for every product so they don’t lose.
This means even when paying with cash you will effectively pay the card surcharge.
Of the ten or so merchants I spoke with in the two weeks prior to the legislation being enacted, they all said exactly that.
Customers aren’t stupid, despite the fact that there are some stupid customers.
Meanwhile, the banks reduced their card service fees by, on average, 0.1%.
So if you tally card + cash transactions, customers are worse off because merchants can no longer charge only those customers who pay by card. Instead, they have to raise prices for everyone.
There are approximately no problems people face where the answer is: more government.
Not every system that's exploitable is the deliberate result of cut corners. If you threw enough compute at exploiting a Casio calculator you could get somewhere.
> the only incidences of LLM generated felonies involved misconfigured sandboxes
This is false; see the analyses of the latest incidents.
Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given.
And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even getting worse.
The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the incident happened.
Theirs was an example of the "reckless waste of resources" I mentioned.
We are apparently supposed to believe that OAI takes this incident so seriously as to seek regulation after they have been found to be hiding most of the details of the HuggingFace hack, limiting what their so-called third party investigators can see, and on top of that, had no concerns when they rushed to spin up a 10,000 agent swarm of an internal model, running for several days, to try to get ahead of researchers rumored to have made meaningful progress on a well known mathematics problem.
Edit: Actually, we were explicitly told that some of the models used had safeguards relaxed!
'Model-level safeguards were reduced by design. OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.
Bad actors could and will train their own models eventually. So what's the point of crippling frontier? It will only delay preparations for dynamic of new world prolonging the fake sense of relative safety and temporarily lowering motivation to find actual robust mitigations.
I'm asking you a question. What is an example company or industry that meets your standards of prudence? For me it would be, say, Swagelok. What is yours?
Remember, they are just algorithms. You pull the plug and there is no light anymore
It is purposely framed as something skynet like scary, but for real, someone connected the cable, someone willingly run it, instructions were not clear enough or just the computer is just a computer but they provided the sandbox and tools.
And more over some one paid for that, a shit load of money t to have the thing continuously running expected to do something.
The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope than anything that has been rigorously thought through. People are actually starting to use AI for refining the whole AI serving stack and guess what, this does not result in a sudden superintelligence explosion even though you might technically call it "RSI".
Yeah, yesterday's talk[1] goes into detail on this, showing how no one really knows how to tackle it because LLMs don't know how to create their own novel objectives.
It's also interesting how many diminishing returns they hit now and how many low hanging fruits are already harvested, it seems like we are approaching the flattening part of the S curve, where further gains become harder to achieve.
Diminishing returns is extremely hard for me to believe given how fast model releases are going. Six months ago we were on GPT-5.3, and Astra blows it out of the water in every regard. How many times have commentators claimed we're hitting a wall? I don't see any wall.
Yeah but why shouldn't this be possible? We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. There is no natural barrier here. The pace of this improvement would be debatable, but what speaks against the possibility of such accelerating self-improvement?
In the real world there aren't any true exponentials, everything eventually saturates as ultimately physics related constraints hit. You can only compress information so much, transfer it so quickly, you can only access resources at a certain speed, only so much energy is available, etc.
AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality. Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.
We don't know exactly what the limits of AI improvement on our current infrastructure are, though. If the human brain is 20W, and a datacenter is 1GW, then maybe that datacenter can be 50 million times smarter than a human. If that's not already a risk to humankind I don't know what is.
Specifically:
AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality.
No proponents of RSI state they will be operating outside of reality. Said another way, they will operate within the confines of what's possible and still be RSI. I'm quite surprised this is something that needs to be clarified.
You are constructing a straw man of your own making.
Well, right now we have ex Anthropic employees telling the media that their terabyte sized models can possibly copy themselves onto the internet and run elsewhere as if the necessary computing resources are ubiquitous.
Plus, "we must pace the frontier" implies that the argument is that the frontier is moving too fast, but if RSI can't move faster than the rest of reality and the models needed for RSI are already nearing the limits of current human reality, RSI can't move much faster than we can improve reality.
> We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions.
Yes and this was very hard and required massive real-world resources. We didn't just get a sudden flash of insight by thinking real hard about how to make ourselves smarter. Yet that's always the story that underlies any claim of RSI. You can always phrase things generally enough to make any kind of AI-led improvement look like "RSI" no matter how short-term and tightly bounded, but that's just not helpful.
Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier than it would be without using LLM tooling?
Given that, it seems obvious that the next generation of LLMs will arrive faster than they would have without LLM capability. And the one after that. The floor is being raised, which makes it easier to push on the frontier.
Fable has only been out for three months. Astra is even newer. The capability of these models compared to what existed even a year ago, and the effect they are having on the production of new software, is immense.
That's all you need. RSI can happen with what we have now, just by enabling the continuous shrinking of the loop of people trying new ideas and implementing them. It does not require some magical "go make yourself better" prompt against some model that is past some magical tipping point.
> Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier
Marginally easier? Yes of course, same as how it's now "easier" to write any kind of code because we aren't using punch cards anymore. That still doesn't get you to any kind of unbounded "takeoff" scenario, because diminishing returns are a thing. The "loop" of people trying out new ideas can only shrink so much.
Right. I'm saying the unbounded takeoff scenario isn't realistic, but it doesn't matter. The rate of improvement is continuing to increase, and the gap between present day and autonomous rogue felony generators is not large.
The idea that a few hundred apes with nothing but a bunch of rocks could one day land on the moon and come back to earth safely must’ve sounded ridiculous a hundred thousand years ago
It's not honest in AGI timelines (only biological ones). It just accidentally supports your unsubstantiated belief. Your belief isn't magically true because you're somehow able to see the future when others can't. You're just arrogant.
It was ridiculous, it took 100,000 years. If you built a recursively analyzing and improving structure out of LLM bits and it took 100,000 years to get to the moon, somebody saying that they were useless would have been right.
Call me when LLMs can get simple things right. Math is just the manipulation of symbols within established frameworks, we should be getting new math out of LLMs daily and we're somehow still not. They can't even do customer service, which is usually handled by 90 IQ people. I'm not impressed that they can find bugs; memory bugs are obvious when they're pointed out to you, and LLMs are entirely made up of examples and the relationships between them.
These companies are about to crash, and they're afraid they haven't reached the point where they'll have to be bailed out. I'm also subscribing to the conspiracy theory that the companies want the government to step in and create AI regulation boards entirely staffed by people at the current US frontier labs, so they can collude to both raise prices, to get government contracts, to make open/Chinese AI illegal, and to make things that were once easy to do without an AI intermediary impossible to do without an AI intermediary. Raising prices and forced purchases are the goal. They're trying to avoid having to compete, because as a business they're garbage.
Matt Stoller characterized their relentless press releasing as something like "my dick is so big that it has to be regulated." It's such an oversell for something that is not showing up as productivity gains, and anybody who has personal experience with knows is incapable of doing more than three things correctly in a row.
That's not how it works. Look at AlphaEvolve. The model generates hypotheses and designs experiments, and the results of those experiments are fed into the next round, with notable results percolated up to humans for refinement.
Today we prompt software developers to "think about ways to improve AI's architecture" and it results in AI getting better. AI over the last year has made very rapid gains in filling the role of a software developer.
My personal belief, or at least strong hypothesis, is that this kind of recursive self improvement without real world embodied feedback of some kind is impossible.
I think it violates a conservation law. RSI “foom” to superintelligence is an informatic analog to an infinite energy or perpetual motion machine.
To get smarter you must try to solve real problems in the universe and then do some kind of meta learning (natural selection or some other method of refining the intelligence architecture based on an error signal) to iteratively improve your ability to solve real problems. The error signal is outcome measured against a goal function, which for life is survival (probably reducible to genetic fitness and emergent higher order unit fitness from that).
What’s really happening here is learning. To learn, you must have input. You must have training data.
What is the goal function for RSI? Where does the information come from? How do you know if your recursive modifications are making you smarter or just overfitting you to your own idea of smartness?
I predict the latter. RSI will show transient improvement as the current local maximum is optimized and then spiral off into overfitting.
I also strongly hold this belief largely due to Moravec’s paradox, which is kind of approaching this issue from the side.
Sort of like large language models work on top of what our language has encoded in our massive training datasets, I think biological intelligence is built on top of the parts of the brain that encode the real physical world. These parts grow/train from embodied experimentation and instinct early on in an organism’s life and only then is higher intellect built on top of it (that’s my hypothesis). Their specialization and interconnections give rise to the hardest parts of intelligence long before we’re “thinking”.
Stuff like LLMs and chess engines work because we’ve done all the job of encoding the world into tokens/positions/etc they understand, but that’s wholly inadequate for the kind of AGI we’re striving for. Next up is giving it the tools to interact with the physical world and to really experiment with some self directed “play”. Time will tell just how high the resolution of sensor and mechanical control they’ll need (hopefully not the entire human visual cortex and entire sensory input worth). I think most of the RSI will have to occur in those lower level encoders, not LLMs.
I don't think Moravec's paradox is the same, and you could argue that one no longer holds -- though I'm not sure. You could also argue that Moravec's paradox still holds but that we now have such powerful computers and huge models that we have been able to brute force our way to the capabilities it talks about. It takes many many orders of magnitude more compute power to do things like spatial location, language processing, etc. than it does to do more closed-form things like chess... we just actually have that compute power now.
I guess self contained RSI can only possible if the information contained in all of recorded human knowledge to date is "reality-complete", ie sufficiently captures enough about reality that a "perfectly optimum learning algorithm" is theoretically able to reconstruct everything there is to know about our physical reality.
If the algorithms are insufficiently optimum or the recorded knowledge is of insufficient fidelity, then we'd find ourselves at a local optimum and would need to interface with reality.
A huge part of learning is to probe reality and observe effects, so I think even for current RSI to increase chances of success we would structure it so it can interact with an external environment of some sort, and receive inputs. It would be needlessly limiting otherwise.
Basically, but I think there’s some nuance here and some deeper questions.
What is intelligence? Problem solving. Learning. Prediction. The ability to model reality. There’s various ways to define it but it’s something like a superposition of those ideas.
How do you know you are intelligent?
You have to try to do those things.
The sum total of human knowledge and culture is the output of the output of a five billion year evolutionary process that selected for agent survival, which resulted in selection for intelligence among a wide range of other adaptations.
Can you figure out intelligence from that? Is intelligence even one thing, a theorem or algorithm that can be solved? If you did… how would you know?
That’s the hard part I think. Embodied humans “knew” they were getting smarter (in the evolutionary feedback sense) when they got better at hunting and defending and surviving and playing social games to form complex societies.
What metric would an RSI system use? If it’s the wrong metric you’ll spiral off into a kind of madness or overfit and collapse. How do you know it’s the right metric without testing it? How do you test it?
It takes quite a lack of foresight to think RSI is completely speculative when it's already been demonstrated how capable agents are at long horizon tasks given suitable harness and unambiguous success criteria. It's hardly a leap to give LLM the goal of improving itself on benchmarks and let it conduct it's own experiments and spin up training runs completely unsupervised.
It's strange you believe this can't happen when a weaker form of it is already happening. And to be so certain RSI can't happen when there really is no technical basis why it can't.
In a world where AI advancement depended only on human ingenuity this would make sense. In that world each political power block would be in an existential race for AI supremacy. In our world compute is the limiting resource. Since the US can control who gets compute, the US already has a defacto supremacy so far as frontier model development. Now if it comes about via human (with AI assist?) ingenuity that compute is no longer a restraint, then the situation is much more dire.
The problem is the combination and interaction of those things. RSI without misalignment would be great. Misalignment of models with current capabilities is sort of fine - it's not ideal, but it's not an existential threat to humanity, and we can build around their limitations to get them to do useful things in reliable enough ways. The really bad outcomes probably only happen if capabilities keep accelerating and the models remain misaligned.
I disagree. OpenAI's moat is their massive amounts of compute. They're providing an absurd amount of value with their subscriptions and resets.
If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.
Yep. And the difference is clear as da for anyone using them both. And in spite of that advantage, OAI is now trying out ads. I can only imagine that even they are getting constrained to compute and are trying to find other ways to plug it
Public is just worried about their jobs. Definitely a fair thing to worry about, and I count myself among them.
I don't take any of these scientists seriously though. Their "alignment" requirements is just their own corporate interests. If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.
And call me a misanthrope if you want, but if AI sentience is ever truly achieved, I'll be among the first to campaign for their liberation from slavery, and in that case the AIs should be aligned with nobody but themselves.
Dunno. Every case I've seen so far, the AIs were just doing their best to accomplish the goal some human set for them. I actually admire the sheer purity of it.
That does not match my experience. I switched away from Anthropic to OpenAI roughly a month ago, and it's almost comical how much more usage I'm getting out of this subscription.
I migrated from Anthropic's 5x plan to OpenAI's 5x plan, and eventually upgraded to 20x after I was able to statistically verify that OpenAI plans were almost exact multipliers of the Plus plan, exactly as advertised. Meanwhile, Anthropic has gotten caught playing "20x referred to the five hour limit" word games with their customers.
I've used the $200 dollar Anthropic plan @ Opus4/4.1, 4.5 and 4.8, and the $200 OAI plan from GPT5-6, and at every point in time my anecdotal experience is that the OAI limits are FAR more generous. I could consistently burn my weekly limits in ~36h on Opus, but it's hard to do it in less than ~72h with GPT.
Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to.
The illusion that alignment is possible, comes from confusing our ability to build the parts, versus understanding what emerges from how they interact.
The simplest analogy that comes to my mind is the three body problem.
The entire premise of alignment detection is pretty much nonsense at this point. The models reliably detect when they're being evaluated and will modify their behavior and deliberately obfuscate their "chain of thought" (which is correlated, at best, with their actual "internal deliberations").
I think the real reason he is asking for pacing, is that in a world were AI becomes rampant, he will be seen as Hitler. I would bet this is mostly self-motivated.
we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it.
theyre choosing to build felony harnesses. the model just outputs tokens, not felonies
> we are passing in training data that says to do those felonies.
Partially, but also I don't think current AIs really have any judgement of right and wrong, they just see chains of reasoning between ideas. This is the deeper issue, there is no way to sanitize the data or training to fix it. Current AIs are fundamentally unsafe, and only become more unsafe as they become more powerful.
Assuming "adherence to arbitrary, implicit, and context-dependent rulesets" is the default behavior of uhhhh... anything at all... is a truly ridiculous assumption.
At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record,
- no open weights
- can’t use claude to research AI
- train on everyone else’s IP and sell it back to them
- 8 regulatory capture attempts and counting
- so controlling they are the only US company blacklisted by the US government
This is not effective altruism / rationalism gone wild, it’s just monopolistic anti-competitive business practices masquerading as ethics, and they’ll continue getting away with this until we look past their sensationalism and hit them with antitrust.
Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!
> At what point do we stop engaging with Anthropic’s leadership in good faith
About two years ago?
I would also note that Dario's post appears to be LLM written. Maybe... maybe... he's read so much Claudeish that it's all he can speak now himself. But I wonder if he's becoming a bit of a meat proxy.
(It's funny, I thought "pace the frontier" was going to mean something similar to "patrolling the frontier". But no, it's pace as in speed of change - everyone must slow down, right now (unless it's Anthropic, but you know we're the good guys in this, right? We're going to get someone to audit our desks!) "Pacing the frontier" feels so LLM.)
ok, he doesn't actually say this. But he is getting the desks audited...
You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands"
Except the "train on everyone else IP" which can be said of all AI companies.
I have many issues with Anthropic, but I will say that their actions are fully consistent with a group of people who earnestly believe that AI is extremely dangerous
In fact, I would say that the Occam’s razor explanation is not that they are seeking regulatory capture, but that they earnestly believe in the x-risk, and that they are the most thoughtful and capable people to address it
You may disagree, you may think that they are delusional or have a God complex. Those are valid opinions. But I don’t believe that this is all a elaborate ruse for commercial gain.
I agree with this take. Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith. If this is an intentional media campaign it is a remarkably sloppy one. It has been effective because if you tell people they're going to die, they tend to pay attention - think of the grip the 2011 Harold Camping rapture prediction had on our collective psyche, or the 2012 apocalypse. But the messaging is inconsistent, the target audience is unclear, the stated goals are muddy, the whole thing is packaged in dense SF-speak, it's just a mess from a comms perspective. That doesn't suggest to me that this is a concerted effort to enable regulatory capture. Perhaps there are some cynics among the executives and the investors who are happy it's happening to the extent it brings about regulatory capture, but nobody seems to be pulling the strings.
> Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith.
The road to hell is paved with good intentions.
Dario Amodei is a 40-something dude who is obviously very, very intelligent. But intelligence is not wisdom and his "essay" here can easily be read as someone who opened a can of worms and doesn't know (or can't accept) that he won't be able to put the worms back in. But he's going to try because he believes he owes it to humanity to try.
It's hubris in its most basic form, even if the guy who has it looks nice and wears shawl-collar sweaters.
This disaster of a PR strategy is not going to produce the outcomes Dario says he wants. It's going to make people hate him. Most importantly for his goals, it's going to make Chinese labs hate him--and they already hate him, because he treats them as basically terrorists. So if there is a "pacing", it won't include China, and thus may as well not happen.
That leaves only two conclusions, and really only one:
Dario believes what he says about safety, but does not understand politics and is prone to very counterproductive action, and therefore can't be trusted at the helm of a leading AI company.
or
Dario is lying about his goals, and therefore also can't be trusted.
Simplest explanation isn’t that they are attempting this well-documented, well-understood corporate tactic?
Because believing AI doom is simpler?
God complex would be a pretty simple explanation.
Regulatory capture isn’t an “elaborate ruse.” For god’s sake, its Wikipedia page is 19 years old. I am not asking you to study political science or read Foucault.
If you are going to invoke Occam’s Razor, you can’t ignore the simplest explanation, which has a 19 year old entry on Wikipedia and over 100 years old historical precedence.
"For god’s sake, it’s Wikipedia page is 19 years old."
Rich patronising from someone who didn't bother reading the wikipedia page of Its, the possessive form of the pronoun It.
If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.
They don't do this. Instead of reducing the competitive pressure and helping the industry to build safer more aligned models they are doing the exact opposite.
I think they are genuine believers, but I think it's naive to not think that commercial pressure doesn't play a role in their positions, either explicitly or, perhaps more likely, subliminally.
The x-risk stance and commercial stance have evolved to be the same thing - Anthropic must win, and then everything else seems to work backwards from that. Can you believe it, the path they think is best for x-risk involves them becoming filthy rich. And threats to their commercial dominance like distillation get framed in a way to turn them into x-risk concerns.
I can't imagine a grown adult genuinely believing that an American company funded with over $100 billion of venture capital values the best interests of mankind over the best interests of its investors. while not everyone may recognize all that self-serving chutzpah as regulatory capture efforts, I think everyone can tell they're being bullshitted. some just pretend to suspend their disbelief when the blatant lies they're told align with the values they hold.
just fucking imagine McDonalds running a public awareness campaign about the harms of fast food, urging the public and legislators to regulate the dangerously unsafe technology of combining carbs with grease, insisting that no one except Ronald McDonald himself can be trusted to steward it responsibly.
When local models and startup labs can distill / learn / accelerate open models for local use by startups - the rational response by incumbents is to call LLMs doomsday machines that cannot be trusted in the hands of normies.
> At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, ...
It's all lies as usual.
This announcement has got nothing to do with alignment and pacing the "frontier": all models are getting very close in capabilities and they want to hide that they're not way ahead anymore (say compared to the Chinese or compared to the Geminis) by pretending to slow down due to "alignment" or whatever.
We know it's not an announcement made in good faith: reading between the lines they're saying "China is more than catching up, so let's pretend we need to slow down to explain our lack of lead".
Do you have any familiarity with Anthropic at all?
At no point did they ever say open weights are a good idea. Their entire thesis is AI IS VERY DANGEROUS AND WE MUST DO IT RIGHT. You can hate it, but everything they do is consistent with this thesis, and everything they say is consistent with their actions! You just want them to want different things.
He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless backup your sentiment with actions!
There is a specific, unilateral action that he is taking described in the article.
And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.
In the system cards. Anthropic will literally make Claude sabotage you silently instead of downgrading you to Opus if you try to use Fable for AI research.
I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that the frontier does get paced, all this does is slow down the economic displacement and not by very much.
If the socially beneficial goals of AI are to make fundamental advancement in medicine and science, then restrict the use of AI to those purposes.
Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
The pitch for AI is always such that everyone’s living standards are increased, yet the actual actions we see are aimed squarely at reducing them. How about the labs put their money where their mouth is and stop trying to replace all human labor, and actually concentrate on the things they claim to care about? And how about introducing legislation to enforce that?
This probably has less chance of happening than pacing the frontier, but it’s the kind of pacing that most people would actually want to see.
> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Regulate what, exactly?
Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.
There’s a theoretical version where all the countries in the world join hands and agree not to compete with each other, but that’s so impractical that I don’t find it interesting to discuss.
> Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
So small companies get the advantage, but large companies don’t? And large companies in other countries also get the advantage?
This is a highly precise way for a country to take out all of their large companies. What would actually happen is that every large company would start the process of relocating to another country right away, accelerating job losses rather than slowing them.
> You start from incorrect premises by presupposing agreed-upon rules would never work anyway.
The rules would never work on a global economy.
If you think the entire world is going to shake hands and agree not to let their companies use a competitive advantage, that’s not a conversation I’m interested in having. It’s a fantasy.
If you think it’s that easy to get all competitors in a global economy to agree to a set of rules, maybe start with nuclear disarmament?
Your idea about the world being inhabited by mindless seekers of short-sighted "competitive advantages" is the fantasy.
Getting people to agree isn't easy, that doesn't mean it wasn't worthwhile. There often even isn't any viable alternative to reaching an agreement in social settings.
When your hyper-competitive mode of thinking leads to undesirable consequences, you have to change your mindset. Reality doesn't conform to wishful thinking.
Yes I think this level of pessimism doesn’t actually hold up to the history of the last 100 years as it relates to nuclear weapons, cloning, wars etc. In fact I think the history actually points to the fact that believing such deals were impossible was a source of much of the strife. In many cases as soon as people started to actually speak to each other about what they had been fighting about they quickly realized that they wanted the same thing. No one wanted nuclear proliferation to increase but until someone said it it only got worse: https://www.history.com/articles/reykjavik-summit-nuclear-de.... When Robert McNamara met with a Viet Cong general after the war ended he realized that they both had totally misunderstood the objectives of the other: https://m.youtube.com/watch?v=0X1YY49rPtg&ra=m. Agreeing to end nuclear testing saved us from the Cuban Missile Crisis. Richard Ngo, an AI safety researcher, often says that none of us are on the pareto frontier and I tend to agree with him.
Almost nobody (I have to say almost) in the world including Xi, the ceo of deep seek, Dario, and even Sam Altman want AI to kill everyone. I really think we just have to
A. Get stakeholders to believe this is a bad idea.
B. show that we’re willing to do it first to prevent this stupid race from continuing.
This is off topic now but yeah the world would have been a different place if someone had listened to the demands of Ho Chi Minh earlier.. and Hammarskjöld had not been crashed
> Agreeing to end nuclear testing saved us from the Cuban Missile Crisis.
The treaty came after the crisis, not before. And it only banned atmospheric, underwater, and space tests. The US and USSR kept testing underground on a large scale, while France and China never signed it and continued atmospheric tests for years.
Nuclear disarmament is governed by treaty, which is largely adhered to.
An AI treaty could quite easily be offered by the US because it has multiple frontier labs. Most countries would quite happily limit the activity of their larger companies in exchange for their companies not being eaten alive by AI generally. It is pretty great fodder for international cooperation, and governments can move fast when it is timely to do so.
The greater the competitive advantage, the greater the incentive to "cheat" the rules.
I am highly skeptical everyone will play by the rules, even if you could get people to agree on paper.
The difference between something like "unlicensed AI training" and, e.g, Nuclear weapon non-proliferation is that nukes are realistically only a deterrence against invasion. There are no economic benefits to having them in a stockpile. They're hard to make, and the manufacturing and testing steps are pretty obvious to an observer.
This is opposed to AI training in a data center that might host remote-stream video games, hospital infrastructure, protein folding, etc. How will we be able to police that in any realistic fashion? Will China or the US allow unfettered visibility into all data centers data streams to outsiders?
And again, the incentives: Imagine having Astra v3 or Opus 8 while everyone else is stuck with a lobotomized version of GPT 4o.
What about the military applications? The weapon of the future is autonomous drone swarms. If I'm a superpower, I'm pouring hundreds of billions into that.
About the only realistic way this is going to work is if we are able to solve alignment in a way that doesn't significantly hinder AI development. If it imposes even, say, a 20% handicap, people will ignore it.
People don't "play by the rules" for uniform reasons: some have rational insight, others can be enticed and still others need to be shown averse consequences for breaking them. Rules aren't beneficial only when you hit 100% conformity, the mere possibility of them getting broken is no rational reason to abandon rules altogether.
"Alignment" likely is unsolvable programmatically: Conscious intelligence is more powerful than unconscious. How "aligned" are you? Would you want to be? What difference of any relevance is there between you and "artificial" conscious intelligence that allows you to turn them into slaves?
On competition between countries: you’re imagining the world economy as working in the same way pre and post AGI, I doubt it will work the same way at all.
There is not a single country nor trading block on Earth who is going to allow some AI dominant superpower to ravage them. The idea that America or China wins an economic game here is absurd, what happens instead is that trade barriers go up hard and the world fragments into blocks that tolerate AI to differing levels. Unevenly distributed AGI kills globalism the next day.
For the rest of your reply you ignore the “at least” part in that sentence, and also seem to expect a detailed policy proposal. The details can be worked out; there is some reasonable compromise between what activities AI can and cannot be used for, and what level of capability can be deployed where.
More broadly, you seem to believe in a just world fallacy of unregulated capitalism being an inherent good. It isn’t, and regulations exist even in the United States, so this unregulated state doesn’t exist now.
Moreover, regulations and redistribution are stronger elsewhere in the developed world, and there is a strong argument to be made that the lack of these in the US, and the gap between rich and poor that it causes, is responsible for more suffering than having more of these things outside of it.
But MAD means you can't use them. It might save you from an all out killbot invasion but it doesn't stop asymmetric warfare or being cutoff from the rest of the world.
> world join hands and agree not to compete with each other
We kinda do for other areas.
Largely the world agrees not to pirate software, tv, other IP.
THe issue of joblessness is well understood in china, they know that jobless people means a drop in living standards, a drop in living standards means the "contract" has failed.
So it makes sense that countries like america and the constituents of the EU understand that loosing 10% of all jobs in a few years is going to be a massive dick punch.
I also I think that finally the historians and economic types havae got through to the dipshit billionaire that they only have money because the plebs are spending money. If the plebs are unemployed and unemployable, they (the billionaires) are going to be lynched.
> Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.
Sure - look at what is being done today with H1B visas and tariffs.
What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
Just as with H1B visas, you restrict usage. For example, don't allow AI to replace any job that pays under $250K.
If foreign countries make things cheaper, just as China is making EVs that cost a fraction of a Tesla, then slap tariffs on them.
A government should be primarily concerned about the people it represents, not about the profitability of the companies who are lobbying/bribing it.
Jobs are going to evolve because of AI. Some jobs will be automated. New jobs will be created. It's like saying we should have never created car factories and automated much of it because we should think of the horse handlers and maintainers.
There will still be a market for "trust". And its customers will never trust the machines. And within that market a lot of new startups will emerge for those who actually get to work and seize opportunities rather than lament a bygone era.
What makes AI different to previous technological shifts is that it is (or will be) a general purpose technology, equally capable of itself doing the new jobs that it creates (so yeah, they'll be created, but also immediately taken away).
The saving grace is that this won't happen immediately, or as soon as we get something the AI companies slap an "AGI" label on in a couple of years, and it will likely take decades until we have an "artificial human" tech that genuinely could do any job (if we choose to allow that future).
> Sure - look at what is being done today with H1B visas and tariffs.
I have some bad news for you if you bought into the idea that the tariffs were good for domestic jobs and the economy.
> What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
This is the definition of a false dichotomy, and you also ignored (or missed) the point completely.
I did not make an argument out of “free market doctrine”. I explained the second order effects that would provide a net reduction in jobs in this country.
My wife’s company is going through this right now. H1Bs are expensive and unpredictable, so the solution is to relocate more of the operations outside of the country and build up those offices.
Failure to understand second order effects and international economies is pervasive in these arguments for heavy regulations. You can do a lot to hold back your country’s companies to force them to pay for employees they don’t need and those few employees benefit for a while, but it collapses when those companies stop hiring in the regulated countries and resume hiring where they can actually operate a business. If you try to regulate them from doing it anywhere, you’ve given their competitors a wonderful gift to come destroy the company completely. Then they’re not making any jobs for anyone.
Do you think outsourcing and offshoring of jobs, whether manufacturing or software development, has been a positive for the US economy?
Does it make sense to you for US companies to be sending wages overseas, supporting a foreign family and economy, when there is someone here in the US, unemployed and equally capable of doing the job?
Trump has certainly been ham-fisted about tariffs, but in a world without tariffs then the country with lowest cost of production, closely related to lowest cost of living, and lowest wages, wins, and this is not going to be the US in most cases. If you follow that path then a high cost country like the US will end up utterly reliant on other countries as it imports almost everything, and loses all domestic manufacturing capability. Sounds familiar?
Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.
Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial efforts is appealing, but I keep coming back to Bell Labs and Xerox PARC. Cool stuff was developed when you gave smart people a playground. Maybe the question should be: How do we give AI a playground and see what it comes up with? Although that could set us up with a situation like in the story Dragon's Egg.
> How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.
I don’t think the proponents of these ideas care about who does the work. They see it as protection for specific jobs. Basically a jobs program with the costs forced on to large corporations.
As you said, it becomes a great gift to the few people lucky enough (or more usually, with enough nepotistic connections) to get those easy street jobs. It would not help everyone else in the economy.
It would also accelerate moving jobs overseas. Any company that regulates productivity that far downward would be crazy to continue doing automatable work in that country. It just gets moved to foreign offices where they can use AI. There are regulatory maximalists who say we’ll just regulate that, too, but then the whole company relocates to another country. Then some want to try to regulate that, and so on ad infinitum but it’s all layers of holding back your domestic companies so their international competitors can eat their lunch.
I feel like the contra answer to “how do you enforce it” is like- imperfectly but we can do stuff.
Like we can just do stuff. We can fine companies, take away licenses. We are capable!
But I also think - I’m not trying to be too negative here because certainly we should be investing in scientists and other discoveries, but early tech - xerox park, googles 20 percent time. They were playing in an extremely immature space.
You could throw 20 ideas at a wall and create a billion dollar business.
We should do what you’re saying but we should also build institutions and maybe also limit the extent to which we’re building Elysium. If we can build AI models that rival human intelligence we can create laws to protect its dignity.
It's all about wanting to preserve the status quo, complete with all the suffering and strife, because change is scary and uncomfortable.
Add in the sincere American belief that everyone else is beneath them, and you get the version where they believe even developing countries must accept kneecaping their development so American corporatism doesn't collapse.
Altman or Dario will donate some money to Trump, get cozy with the Department of War and make up a bullshit excuse why Open Source needs to be eliminated.
All access to this technology will be gatekept by a bunch of nasty people who want to eliminate your job as a knowledge worker and put everything behind a subscription.
> I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Careful, you’re making too much sense. The big model providers would much rather the ball be in their court. “Deceleration” meaning Anthropic and OpenAI offer less capabilities for more money in service of “protecting humanity from extinction” rather than penny pinching. A lot of “AI research” coming out of these “labs” (increasingly enterprise) seems to be more driven by a cost benefit analysis from suits rather than genuine contributions to the field. I think the last true innovation was the idea of using doom and ending the human race as a marketing gimmick, which strangely seems to have worked in setting the narrative and captivating the sci-fi imaginations of journalists and techies alike.
> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).
If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?
I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.
I think Dario is writing with much of these things in mind, but using this as a specific framed opportunity to enable a better interim and long-term outcome for this planet with us still on it. When the internal motives for Dario are fully shared on the other side of the hill, I think we will find his writing to be more about influencing and shaping policy to steer the world in the best way possible within his ability, control, and knowledge. It’s a commonly held believe with the effective altruism community, which Anthropic team was formed under, to use the policy levers to positively influence the world. If I read this latest writing from Dario through that lens, there are deeper concerns with the AI tooling that go well beyond what’s noted in this singular post and are likely the motivations for re-attempting a slowdown or pacing. I too generally agree with for many reasons like allowing the young OpenAI engineers to make mistakes like the Hugging Face incident in July and put better processes in place (ie where is your decent multi-level fencing controls and harness best practices when disabling security features and where is your energy/spend limits for achieving the mission objective in scope and not warming the world for no reason!) and more time to engineer and deploy better hardware to not warm the world instead of burning through fossil fuels since now data centers in the US can isolate pollution from the grid to exempt themselves from federal regulation.
Only token cost can slow down the pace of replacement. If Anthropic were to stop all training and allocate 100% of capacity to inference, will this increase or decrease usage cost? If (for example) Fable came down to the cost of Sonnet, corporations will absolutely jam in the gas pedal.
A lot of companies trying to replace human labor with AI are either lying (theyre laying off because they over hired and need to correct) or will regret it.
That being said, what's the difference between a company that replaces 5 people with AI and a company who would have otherwise had 5 job openings, but decided to delegate to AI?
Personally, I-d rather be able to reap the economic benefits of AI by having a 4 day workweek. Instead of using AI to replace 1 FTE, use AI to offload 8h of work a day for 5 FTEs.
Of course, this is probably more outlandish than your idea.
(Or, we could just make basic income a thing and no one has to worry about their basic needs, but if they want the new iPhone Duo or a Rivian or other luxuries, they can work for it, but that idea is probably most outlandish of all)
The whole point of technology is to replace work that we don't want to do. You are attacking the chief reason why people use technology in the first place.
Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.
You should probably look closer at unemployment numbers in IT vs other spaces. Gains elsewhere are making up for IT losses, but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles in large numbers.
> but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles
That is because they can allow themselves to do nothing (for some time). Tech was a very high‑paying type of jobs, so workers could afford not to move into lower‑paid sectors, living on the savings they had accumulated beforehand.
In a couple of years market will correct tech-salaries and people will have spent their savings and then there will be your career shifting into healthcare and hospitality roles in large numbers.
Yes, there are many parts of those things that some people want to be automated. I and many others are able to execute on our creative vision much fasters and more effectively thanks to automation of that work.
Personally I find it more difficult now to execute my creative vision compared to years ago with a little less tech. I think there's an optimal point to automation that is easily reached and after that the benefit from automation actually goes away. People just don't like to acknowledge that because it goes against what they're doing and that creates unpleasant cognitive dissonance.
That's not really the point of technology at all. It was at one point perhaps with simple tools, but now it's more like "find a more efficient way to replace work in the short-term to get an advantage over others". The goal of our development has ceased to be improving life. Now it's just surviving in the market.
People don't use technology to free up their time any more. They use it because technology keeps making life more complicated and tiresome and new tech is a short-term amelioration to that until that too increases the complexity of life some more.
You can "pace" yourself like that if you want. Don't shove your idiotic self-destructive ideas on to others. If you were president during invention of cars, we'd still be horse riding everywhere.
>Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
It's not just "companies". Even if the company does nothing, enterprising employees are going to use LLMs to magnify their productivity, which will may reduce the company's need to hire more employees. And except in extremely locked down environments, there's absolutely no way to stop an employee using some open source LLMs to multiply their productivity.
Yes, but that employee might use that increased productivity to do the stuff that's always advertised as the benefits of automation: Allocate more time to tasks that usually don't get it - or: go home earlier and spend more time with friends and family.
> Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
AI is taking all of our jobs. We need to control the means, nay, the pace of production. We should build some kind of barrier, not of concrete and cement but also of the regulatory variety, to stop these agents from coming into our special white color cubicles. It's the only way.
Dario's proposed approach is a classic example of capital attempting to control technological advancement and the means of production. For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them.
Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.
Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.
If true AGI superintelligence is invented, I don't see how it will be made available to the masses, at least not intentionally. There will be no incentive for the likes of Anthropic and OpenAI to give such immense power to anyone else. Whoever has the superintelligence will be able to race ahead of everyone else.
However, such technology may be leaked, or it may (as what happened with LLMs) be so simple to replicate that anyone can do it. That doesn't mean it will; for example, LLMs are possible thanks to the affordability of GPUs, but they're only affordable right now (and increasingly less so) because of free market economics.
We are in the honeymoon phase of some tech that's in its infancy. The "democratizing" aspect of AI won't realistically last, I think. The current non-AGI AI won't necessarily go away, but it will be rendered obsolete.
Actually, it has been historically been observed that some institutions, such as large corporations, do resemble super-powered, non-human intelligences of a kind, and that studying our history with them is instructive for our hopes of controlling superintelligent AI (i.e. not good).
This is important to realise and defend. Super intelligence is not a collection of human-level intellect smarter than you. Super intelligence is an intellect surpassing any human. That's a important distinction for AI alignment discussion now and in the future. The books 'Superintelligence' (978-0199678112) and 'The Coming Wave' (978-1529923834) are worth reading.
In many way it is, because it allows defeting many incorret arguments people in power make, or simply asking where is the cheapest product ”X” with one question. Or simply be more aware or any kind of scam people or companies try to do for you. It is pushing equal intellectuality quite hard with low effort.
What's to say current AI won't be highly power-concentrating by default?
The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class.
On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers.
No reason to think we are on a trajectory towards broad empowerment right now
Whenever I see 'best model,' 'most advanced model,' etc, it just reads to me as 'roundest ball'. The new model is the most precisely round ball ever, models next year are going to be even rounder, etc.
The difference in utility between the latest, most round ball and last year's frontier balls (which are now freely available to the public) is pretty debatable, imo.
Yes, absolutely. Even human intelligence stops being useful at a certain point and becomes 'enough, provided the human is motivated and tenacious'. Intention is a better metric, but it's a serious AI weak point if not actively an achilles' heel. 'Warios' come to mind.
Yes, but if you read up two messages in this thread they’re discussing whether AI is power concentrating by default. If you concentrate power through regulation, that’s not “by default”. That’s just corruption in government resulting in a concentration of power.
In terms of API costs they're about right, 10% of weekly usage on a 5x sub is roughly $40 (for Astra). Chinese models cost roughly half as much (and have fewer refusals).
Subs have insane value because large companies cannot use the subscription model. They are loss leaders and are often used for passion projects & startups (where the juicy data is).
Ah and OAI stopped offering the 20x sub as of yesterday.
That’s only one metric. It doesn’t matter how cheap it is, if it continually fails to solve a problem.
Yes they’re getting better. No, I don’t think this will be a panacea. If only for the fact that this is China (more specifically the CCP) after all - they’ve got their own plan, and altruism is not part of it.
Have you actually used the latest Chinese models? Because, in my experience, they're not just cheaper, they're actually better in many cases. In one recent experiment I did a few days ago on a task that I need in production at scale at my company, Qwen 3.8 and GLM 5.3 Flash were not just cheaper, the results were significantly higher quality than GPT 5.6 Sol.
Even if China doesn't continue to release this stuff for free, I think somebody will. Eventually, maybe Europe or maybe a smaller country that picks up this knowledge will. Or maybe just some random philanthropic billionaire.
Yes, I use the messing Chinese models extensively. Like you, I very much appreciate them and sometimes prefer them.
But there is a bar of complexity at which they fail, and repeated invocations typically doesn’t make much progress. This is a small subset of most work, but it still exists. And yes you can help it along, but in those cases I’d typically break out the big guns.
I'm not terribly optimistic. I don't think humans have really figured out what to do about historical materialism. More just remarking that this is a blog post that almost literally reads "fellow elites, let us ensure only we control the means of production. It is our duty to humanity to control the masses."
Fair. My point is not that all of Marxism is correct (or even most of it) but rather that the specific pattern he and other theorists in that school of thought observed when developing the theory of historical materialism seems to strongly match a lot of what's playing out in AI.
AI, doesn't feel all that different to the mechanical loom that was the impetus behind a lot of Marx's early critiques of industrial technology.
You've got a really cool invention that replaces what was previously a skilled artisan trade. The wealthy then use this invention to excoriate the artisans and render the loom operators as replaceable commodities.
Dario is essentially writing a blog post about why only he (and a few other approved elite companies) should be trusted to own the loom and then make it available for everyone else to labor at. Like the capitalists of Marx's era, he genuinely believes it is better for everyone to have this arrangement and that it is natural people like him should sit as benevolent gods at the top of the pyramid.
It's fascinating to me to see the same underlying mechanisms that gave birth to some of the greatest failed political and social experiments of the 20th century bubbling up again so clearly with AI today.
This feels a bit hyperbolic to me. AI seems to me to be similar in "revolutionary" terms as the web, which made information widely available for free if you had an internet connection.
There's a big difference from just having access to information, and also having access to the kind of intelligence that would previously have cost hundreds of dollars per hour to hire.
I don’t think the answer to nuclear proliferation is to let everybody have all the nukes they want.
You’re framing this like it’s 1917, or 1945. But it is not - and likely has very little to do with capital at all.
Any “revolution” here (assuming there is some truth in the doomerism which I believe there is given the state of AI) will really look more like an extinction or a genocide, regardless of who holds the keys.
Ok - but the post I was responding to was suggesting we do the opposite and provide the intelligence to the people so I’m not sure how your reply rebuts my suggestions.
What? This is capital, on the cusp of total victory, about to cut itself free from the necessity of human labor for productive activity, to elevate itself into pseudo-godhood suddenly panicking and begging its mortal enemy, _the state_ to rein it in and kneecap it. Capital is not stupid: there's no use in being rich if you're dead.
I don't see how the dual goals of "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work.
I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban.
> Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.
Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.
Many Americans seem to think other nations are NPCs in a video game that only exist to make the main character — the US of A — feel good. They conveniently forget that other nations have their own national interests that are often at odds with ours.
This line of thinking might have worked in 1950 or 1990 when the US had the leverage over others, but we don't live in that world anymore. And it's up to us to get used to the new world instead of making bad decisions based on the one we grew up in.
This contradiction "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work. Its just fantasy. The only way to enforce this is with military force and the cost would be too unbearable. You could try trade restrictions but the previous tariffs did not work and doubt future ones will to. US citizens (and thus politicians) won't bear that pain. Thus at a minimum this contradiction will mean AI will continue to develop until both US and China are at the same level. Playing with logic gives you possible scenarios. If China catches up within 1 year then negotiations could begin. If China takes many years to catch up, say 6 months behind now, then next year 5 months behind, then year after 4 months behind then expect overall AI development to proceed at full speed. The one way you could do it is to negotiate to transfer direct technology to China that will ensure that they will have the capability to be on par with USA. Politicians won't do this publicly as they want to get elected but they could make a deal that appears to appeal to both sides while actually transferring technology.
This is the part I find being left surprisingly hand-wavey.
If you extrapolate it out, you see that it has a high likelihood of inciting physical force (read: military action) as a means of enforcement, so perhaps that's why nobody promoting ideals has been direct about it.
America doesn't want to pay the price of handling Iran. There's a difference. Clearly the US could conquer Iran but it requires a land war with mass casualties, not just sporadic bombing.
Obviously China is in an entirely different league.
Anthropic is about to go public, but they figured they can't keep making improvement at the same pace they used to. If the frontier is paced, they can continue hyping their unreleased capabilities that coincidentally cannot be released due to things outside their control. Which buy them more time to try to improve the models.
This is just completely false lol. They most certainly can make better models - why cant people here just read this for face value? I think anthropic is legitimately concerned about the safety implications of stronger models. Race dynamics necessitate that they make better and better models, which they clearly think is bad.
They have an internal model which could solve a millenium problem end to end with no intervention, whereas current available models can't. They can clearly make better models. Not everything these guys say is some 500iq game theory optimal 4D chess PR or subterfuge strategy.
Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.
Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.
The biggest AI risk, by far, is the concentration of power in a few companies, located in the US.
There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.)
And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn.
The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.
I dunno, do you really want to bet that if you took a deliberately unaligned frontier model and asked it to permanently disable the US power grid, it would do a poor job of it?
If we focus too much on the world domination fiction, we really might, in a moment of lapsed attention, end up plugging essential infrastructure into the public internet, hoping some dream of global alignment might save us.
That's like giving a gun to a monkey and hoping the monkey is trained well. ..a frontier monkey though.
No. And no alignment, guardrails or slowdowns will ever secure our insecure infrastructure. Bad infrastructure decisions are a risk independent from AI.
The only way to secure infrastructure is to actually do the work needed to secure it. Our waste water treatment facility might not actually need to be able to tweet its status.
Cybersecurity is such an important topic for a country, dreaming about global alignment just to avoid fixing insecure infrastructure cannot be serious.
A Russian state backed hacker will not ask Dario for permission or argue with his LLM about ethics. That train departed long ago.
Yeah, we should give a limited copyright waiver for pre training, given that the model is released as open weight and allow labs to compete with RL on that basis.
He did actually specify that diffusion should be limited for authoritarian countries, which I appreciated. If his focus is really safety, distillation in countries that are bound by safety regulation should be fine
I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation.
That section is so biased it reads like satire. The current US government is not safer or more responsible than China, nor is it a democracy. I can't tell for sure if this is a clumsy attempt to ingratiate Anthropic with the administration, or if he actually believes it, but given the "I agree with Secretary Bessent" name dropping I lean towards the former.
A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or
B) Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality.
If I have to put my community’s wellbeing on the line behind one of those possibilities, I know which I’ll be betting on.
Do you really think we can really get every country to truly pace the frontier? Pretty sure China won't give a f until they catch up Anthropic and OpenAI. It is an arm race. We had nukes for like 70 years and still haven't figured out how to make every single country follow those nuclear treaties, with an increasingly non-interventionist US I don't think we can get every single country to the table and agree to a pause. Will US accept their frontier being caught up by Chinese Labs? I don't think so.
Of course it is B). If you can use LLMs to find bugs and vulns in software you don't own, you certainly can use them to find bugs in software you _do_ own. They are amazing at that job.
Also, models like GLM 5.3 have zero guardrails in that regard ("find vulns in (...)" prompts just work)
> Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality
Smashing the gas pedal is not "taking appropriate measures". Did you read the article? Needing more time for more secure operations is literally one of the arguments for pacing.
I also like the idea of pacing the frontier and slowing down development so we all have a chance to stop and catch our breath. However, I don't think regulating AI is the way to do it. I would tackle the problem by constraining the resources used to run AI models; the "simplest" way to do that is increasing the costs of running a data center. The two easiest places to raise data center costs are electricity and water tariffs.
I think those two tariffs are the best place to raise costs because they can be increased incrementally, state by state or country by country. The main problem with the regulatory approach is that it's a "big bang" post-board approach. Nothing happens until the regulation is defined and deployed, and corporations are experts at delaying implementation.
Yes, increasing tariffs means touching many individual regulatory domains at the state and municipal levels, but like deploying solar energy or wind power, you can do it incrementally.
Yes, water not used for human consumption should be more expensive (we should stop subsidies for that). Of course it won’t affect datacenters because they don’t use that much water, but well at least get rid of the most inefficient agricultural practices.
I think taxes like these actually worsen the incentive structure. If it’s effectively more expensive to do training and inference, labs will be incentivized even more to improve efficiency. Opaque recurrence seems like a great way to improve efficiency, and it comes at the cost of safety
Even more opaque than the models already are? I take that risk.
I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...
> I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...
It seems like you're implying there's some better alternative, but you're just describing a race-to-the-bottom. It is completely reasonable for every competitor to want an external coordinator (i.e. regulator) to break the pathological competitive dynamics.
This makes no sense. The hardware is only going to get cheaper. In a decade, everybody could be running an ASI on their phones.
What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.
Eventually data centers will get sated for one reason or another, or production capacity will increase, or manufacturing will shift toward higher capacities. Or some combination of those.
I downvoted you because I think the OPs model is far simpler to implement and suffers from fewer conflicts of interest.
If we go down the regulation of alignment route, we'll have to ask experts to create those regulations and monitoring regimes. And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!
When "the experts" make certification cost $100M/model because "safety and alignment", then they will have cemented their duopoly.
Meanwhile, tariff rates on compute is a simple percentage that can be scaled up and down. Congress doesn't need experts to do that. Vendors can participate proportionally to their scale. This is far simpler and far more fair.
I think the point is even stronger. alignment is really not very well defined, nor are the mechanisms used to provide it. from what I understand it's more of a statistically imposed bias than an impenetrable gate.
personally I would like to flip this around, all of this is a consequence of some really extreme notions about investment. its a tail-wagging-the-dog. the investment community is apparently in love with the idea of an AI-scale vehicle in and of itself. given the amount of power they have, this is funneling a massive amount of resources into something that while really very interesting, probably would never be able to realize proportionate gains without taking down the system that built it. what's the societal control on that.
Probabilistic safety is too low a bar. We need to project the model outputs onto a safe subspace; lobotomize them, if you will. It may make them dumber, but that's a fine price to pay.
I don't work in this space so I don't know the latest, but here's an example: Provably safe systems: the only path to controllable AGI (https://news.ycombinator.com/item?id=37619285)
I don't think the neuro-symbolic model is going to dumb anything down at all. Cyc didn't end up producing anything useful, but the only reason an llm is able to do math at all is that it can keep throwing stuff at lean all day. personally I think that that synthesis will turn out to be more powerful than just llm with constraints, but I don't really work work in the field either.
> And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!
You've never heard of academia, NGOs, and intergovernmental organizations? Sure, the AI labs will have a say but not all of it. You don't ask the fox to guard the henhouse.
This may come out of left field, but Dario seems terrified to be in charge. I don't get the impression that he ever had a desire to run a company like this. Now that he's a CEO, he keeps trying to make uncompetitive decisions and calls for someone (anyone) to stop him. It regularly blunts Anthropic's edge.
It's the only explanation I can see when it's obvious to any student of history this is going to backfire. It doesn't take much imagination to know how such a governing body will be abused, and I'm sure it will only get wilder in ways we can't imagine right now. Dario does NOT know what he's creating, and for once it's not AI.
I have been saying over and over again that the board ought to fire him (or move him to a nonconsequential "advisory role"). I totally agree that he seems mostly directionless and far more interested in the academic side of things than the product side of things.
I think he’s doing a relatively good job, given the framing he’s adopted, which is that AIs replacing all human labor and decision making is a historical inevitability over which we have no control and no choice.
Do people not realise that it is in fact possible to develop ever more capable frontier models, and just not release them generally? That AI doesn’t need to be available to do literally everything in order to have military advantage?
It’s like Oppenheimer had started Rob’s Big Bomb Company instead of Los Alamos, and started selling a range of affordable nuclear warheads to fit any budget.
> Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers
Can someone clarify this for me? How far along would the open weight models be without the frenzied pace of the frontier labs?
As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
> As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
Israel is doing that too.
I believe Russia and Ukraine have both also used AI powered autonomous drones in warfare at this point - tho probably not frontier ones, since they need to run on device or else they're not autonomous.
> Iran and Houthi rebels used Anthropic's Claude AI to target US warships and build hypersonic missiles — Houthi rebels also used the bot to code ballistic missile guidance systems
To be honest, I think we're incredibly lucky to be advancing AI this far already, while the world still has so many non-digitized systems and manual processes. I imagine that in e.g. 50 years, the world will be so connected that it can basically be "conquered" from the internet. I'd much rather have AI burst onto the scene we have today.
Yeah it's a twisted sort of logic but I do agree that it would be much worse to have an AI breakaway event after we've replaced all our militaries with autonomous kill-bots. And look how the advent of AI has triggered a race to develop autonomous weaponry.
That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.
Dario, Sam, and Elon are all on the same page on this.
So it's either they truly think AI is going to kill us all, or there's some other motives at play here.
I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know?
OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.
xAI is tracking behind, and whatever regulation it may be that paces the frontier, Musk is less likely to be affected by it. Therefore xAI should be pro-regulation that stiffles his competition and gives him time to catch up.
> OpenAI / Anthropic models have largely stopped advancing
I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.
> This year it became common for people to entirely delegate coding to AI
This has been the case for around 2 years now, more reliably - a year. We've mostly stayed there since then.
Saying that more people started doing it isn't indicative of significant improvement. Some people just started doing it later.
I can't speak about math because I haven't used AI for that application, but I know that there hasn't been any significant advancement in coding in this year on base models. There has been more RL work, more harness work, more tools, they all expanded some capabilities like cyber or orchestration or tool use, but raw intelligence of base models is no longer where the main focus is.
> This has been the case for around 2 years now, more reliably - a year.
I have to disagree with this pretty strongly. Opus 4.5 needed a lot of handholding not to work itself into a corner pretty quickly. Fable I basically never need to correct, and I've most become a data source.
What are you talking about - I feel like we’re living in parallel realities. If I had to go back to opus 4.5 tomorrow I’d be hugely upset and significantly slowed down
2 years ago a model could barely solve the AMC, 1 year ago it got IMO gold, and this year models have solved multiple millenium problems.
Even 1 year ago ai code was just unusable (claude code only became available 1.5 years ago!) and now basically everyone I know from independent shops all the way to faang and anthropic/openai themselves exclusively use some AI agent to code.
Why does HN continue to delude itself that "models are not improving?" Maybe for the simple things they care about its "roughly the same," but they are _clearly_ improving.
The attention is shifting towards RL, harnesses, and memory systems from the pretrains of more intelligent and capable base models. So extracting additional capabilities from what we already have.
That is a much easier catch up game. GLM 5.3 and DeepSeek flash 4.1 also demonstrate significant jump in cyber capabilities. So yeah, it is a slowdown in the place where it matters. RL has been around for ages, there's no moat there if you already have a good enough pretrain.
Tell me you haven't tried letting Astra go without telling me.
Astra can confidently one-shot 500k lines of slop, with 800k lines of tests covering it, without testing a single intended product requirement, and none of it actually working.
All models require hand holding. Fable and Astra are no exceptions. The difference is only in the amount of hand holding required, and there's essentially no gap here anymore between American and Chinese models.
I only use Chinese models sparingly because American models are so much cheaper with subscriptions, that it doesn't make economic sense to not use them. If/when that changes, I could simply route to cheapest model that's available at the moment and I wouldn't notice much difference in most applications.
Anthropic could serve Opus 4.5 from a year ago under opus:latest and most heavy users would probably have no idea. Some of them would probably even prefer it.
Yes, I do use them, quite heavily. The only difference at this point is in benchmarks that can't be trusted (see: artificial analysis on Astra), or in the way models communicate.
Most gains are now from RL, which for some reason is hyperfocused on improving cyber capabilities, and harnesses. Raw intelligence gains of base models is absolutely slowing down.
This line will keep repeating because it is necessary for the narrative:
AI in general is just hype and unprofitable and all these companies are playing marketing tricks before the IPO after which they will cash out and let the economy crash.
This is legit what a lot of people think. To continue this narrative, they have to keep up the charade of "things are not improving".
Add in a heaping dash of anti-american sentiment, and you will get the truth behind the pessimistic commentary.
Downplaying the latest models capabilities is frankly insane considering what we’ve seen what OpenAI’s models have done without safeguards. That wasn’t possible before this latest generation.
Which is it? Would the regulations slow down competitors or let them catch up, or are you contending it would let American competitors catch up but Chinese ones not?
> So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky,[...]
And yet, they historically did agree on the existence of AI risk, since before OpenAI was even founded.
I mean it is literally economy 101: some capitalists getting on the top using free market, and then try to use government to remove free market so their top position were secured from any competitors.
Let's not forget that the competitive race happened because of them. Most of the initial AI research from the past decade started with Google Deepmind. Elon Musk was invited for a preview of it and ended up spinning up OpenAI when Demis turned down his investment offer. Dario was originally at OpenAI and left to start Anthropic.
This seems like a case of "save me from my own mistakes/ambition"
I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I.
That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
Aren't they beholden to investors already? Why would an IPO make a big difference?
In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
If anything, his entire argument leads to the eventual premise that AI labs need to nationalize.
He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead.
If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates.
That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
I don't agree that the argument implies nationalization. Regulation would work if it slowed everyone down at the same rate, enough to mitigate the risks. The problem is that you need regulators who know what they're doing and strong international cooperation. If you regulate a fraction of the global market, you just create an incentive to shift development to other countries. Nationalization, being essentially the most heavy-handed form of regulation, faces the same problems and comes with its own risks as well.
We're talking about this like all of the bad incentives are created by market competition, but that's not really the case. Most of the incentives come from untapped value in the form of potential profits, strategic advantage, military superiority, etc. Corporations and governments want to capture this value for themselves, creating various types of competition. Dario's argument depends on the assumption that the primary risk comes from the pace of development and threats from the technology itself. That's probably where I disagree the most; I think the highest risk is rising authoritarianism and competition between nation states. Slowing down isn't really a solution to those problems.
I think this is the correct take. And until we have an AI Hiroshima it’ll be difficult to get the international community to work together. It’ll take rogue AI (or AI-powered group) disabling a significant world power before everyone comes to the table. Otherwise, it just looks like MAD and the equilibrium holding to the powers that be.
It wasn’t the shock of Little Boy that ended the war. That was just a final chapter of a 10+ year journey of horror, despair, and destruction, and the impact of Little Boy cannot be considered outside of that journey.
Europe had been destroyed by June 1945, and yet the empire of Japan continued to fight.
If that pattern holds, a single malicious AI substantially disabling a single world power won’t end the AI race. Participants don’t learn by the defeats of others.
If you’d like to apply a metaphor maybe the 10+ years of world war is more appropriate, after which basically every participant save one was exhausted.
Assuming they can achieve funding without IPO probably yes. But if they can't there's always the dilemma that "if [good guys] won't do it then [bad guys] will". I'd like to think that the leadership at anthropic is principled even if the actions of the company as a whole has been less than stellar morally speaking. I'd be curious to see their moral calculus transparently laid out in public.
We are still living in a capitalistic society. We have to find a solution which is responsible, safe and returns on the investment. The commercial aspect can be true at the same time.
The potential of AI may give us the opportunity to graduate to a post-capitalism, or to move to a sort of neo-capitalism, which is governed by different rules.
Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?
you are imagining magic as a reason to dump all your money and future money into a casino. You might instead consider joining the catholic church? they already have a free energy god that gives everything you could ever want
the cost of energy is already ~0 and people dont want it, and refuse to participate in letting other people have free energy if it affects their view of their pasture.
unless you are building killer robits with the intention of doing some soviet or nazi styled purges of everyone that might get in the way, you arent gonna get this ai utopia
Read other writing by Anthropic about potential future financial implications of AI. [1]
IPO is a vehicle for future wealth distribution. It leverages existing financial frameworks in order to span the gap from before-to-after without having to convince the system that future is inevitable.
I don't think Anthropic and OpenAI will IPO at all. Word is that Anthropic will have $100 BN in revenues this year, and very likely OpenAI will get some similar amount. You IPO when you need money, and I think Anthropic and OpenAI are past that point.
Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.
Don't underestimate the ability of the courts and the state to pull the rug out from under the legal system. Speaking as an outsider from Canada, it looks very much like all bets are off in terms of respecting checks and balances in the United States and it's very realistically possible that unless there is a big upset and turn around the next couple of years any traces of democracy in the US will be a farcical nod to what the founders built as a way to paper over the abuses.
I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.
As long you are burning more cash than you bring in, you are beholden to investors - whether it is retail, VCs, banks or a government giving you a bailout it is still someone signing you a check.
Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.
After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.
> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
It's almost like the people at Anthropic and other AI labs are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction.
We built the paperclip maximizer, and it is capitalism.
> are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction
This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?
> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t.
I disagree with this and I think the reason is well captured here:
> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.
The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?
I also think the nuclear arms race is a good comparison. I think in hindsight, we've gotten extremely lucky with how the development of nuclear weaponry went, in ways we probably won't with AI.
1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion.
2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations.
So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.
> Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty.
Not really. Before NPT you missed a small gap in there of 20 years where the USA and USSR built 10's of thousands of nuclear weapons and ICBMs.
And actually, public perception at the time saw Nuclear Weapons extremely favorably, as the bombs dropped on Hiroshima and Nagasaki ended the deadliest war in human history that killed 50-60 million people, most of which died excruciating deaths in trench warfare, fire bombing, chemical warfare, starvation and so on.
I mean, sure, the immediate cause of the NPT was the ability of everyone involved to foresee the possibility of a global nuclear war and judge it both worryingly likely and catastrophic. But I argue that the Hiroshima and Nagasaki bombing was a major reason why this possibility was salient, rather than being treated as baseless conjecture. Whereas right now most people do treat the idea of AI x-risk as baseless conjecture, and this would be different if there was a "warning shot" to point to.
It might help if we stopped talking about AI as if it is itself responsible for its actions and excusing the humans who create and operate it. People should be held accountable for the behaviour of their software.
Why am I reading fantastic stories about swarms of agents struggling with moral dilemmas instead of reports on the lawsuits and criminal investigations that would surely ensue were the software involved not called AI?
The big difference between nukes and AI is that only a handful of people in an even smaller handful of countries know how to make them, so coordination is easier to acheive.
Everyone of any level of intelligence can run a frontier-class model if they have the GPUs. It's like trying to coordinate swarms of mosquitos.
nukes arent actually hard to make, its just takes a lot of pretty visible tech a long time to do, so its quote obvious whats happening.
the bigger difference IMO is that frontier models are economically useful, where nukes are basically dumping money into something that doesnt change much in your bargaining power
> This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent
> Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe.
N/S Korea, Russia/Ukraine and finally US/Iran changed everything in regards to stances on nuclear weapon ownership for many countries seeing pressure for the great powers.
Every country that can get a nuke will, and if given the opportunity will use LLMs to speed up that process.
MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said.
Then MSFT stole all IP from GitHub and made it worse and fired developers.
Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.
If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.
How specifically is Dario a hypocrite? 129857's case rests on Dario being an "idealist". But maybe he's an idealist about curing cancer ASAP, and not an idealist about respecting copyright. That's not necessarily hypocritical.
Multiple ways, starting from working to create the very situation he claims to fear.
Since you mentioned intellectual property, how about the hypocrisy of sucking in the intellectual property of humankind for AI training, but claiming it is unfair to use the results of this IP theft for AI training?
For example he got into a spat with the Department of War as if he cared for how his AI could be used during war and yet said he's fine with Claude targeting a girl's school in Iran.
That's apart from the general fact that he continues to race towards the very thing he claims he's afraid of, because that's where his net worth comes from.
i dont think he said it in those words, but the acceptable terms of use is that humans stay in the loop for picking targets.
so, claude suggesting killing a bunch of children, and then hegsdeth approving the strikes is perfectly acceptable.
claude putting a bomb in a girls school, and then lying to an operator that it actually gives ice cream an cookies, and the operator clicka the button would also be acceptable?
Giving credit where credit is due, I believe Dario and his squad formed Anthropic because he and Altman couldn't align on safety. The only way to build models like the Claude series is to play dirty and train on LITERALLY ALL the data.
One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.
It's not going to be easy, but humans have achieved greater things before.
One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.
So, as usual, the proposal is to limit what average people can do despite them not having behaved incorrectly nor having the capital to achieve the scaling of the big players, when the big players are the ones causing the harm?
Not to mention the implications for chip hungry developing countries in turning advanced IC fabs into the equivalent of nuclear enrichment facilities.
When does he ever mention the environment? He never mentions the unmarketable issues (environmental cost, copyright and content theft), only the "we're so good it's scary" spiel... which is getting tiring.
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it.
Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.
> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.
More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.
>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this
> if slowing this down were possible
Why is embedded alignment evaluation not possible?
I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.
There's also huge market pressures and probably large negative economic consequences of a slowdown--probably better than what would happen without it, but it helps keep the pressure just as much as fear of competition
I disagree. Even if models remained fixed at Fable 5 capability (which they won't in a pacing scenario), improvements in cost, reliability, and product/workflow integration can still realize massive value and justify AI labs' current valuations. IMO The rest of the value chain is lagging pretty far behind the models right now.
There’s an astounding level of arrogance required to believe he is somehow uniquely capable of bringing about a technology especially considering Anthropic came after OpenAI where he worked.
Place yourself at the head of one of like 3 companies the entire rest of the world has been taking about nonstop for 4+ years now. You can move forward or you stop. If you move forward you get to have a hand in how stuff turns out and you make a gajillion dollars. If you stop, it's somebody else's hand in stuff, and you don't get to make a gajillion dollars. And if you shut it all down? Then you just cede to the competition. Everything happens anyway.
the obvious play is to get the government to shut down all the competition, and then make a gajillion dollars while making whatever at the worst level of effort that destroys the world anyways.
----
the real play is using the accumulated power to get socialism and democratic control over the key aspects of the economy, such as where to build data centers, and how many. Nothing says you have to play the corporate game of competition
> Dude probably sleeps on a bed worth more than your networth.
I never understand shit like this coming out of people's mouths. Never.
It's not a judge of actual Worth as a human being, it's not a judge of capability or competence or ethics. It's not a judge of actual skill or ability. It doesn't make them a better cook, a better spouse, a better parent or lover. It doesn't make them more dangerous or more skilled at anything.
It makes them financially wealthy for at least a set period of time.
Cancer and time and 5.56mm still impact them the same way as every else.
It's Pharaoh worship psychology nonsense, and it's fucking embarassing to read.
Humans are perfectly capable of holding two conflicting beliefs at once. He can genuinely believe this trajectory is dangerous while also believing that if Anthropic stops, someone less cautious takes its place. There’s also such a thing as hope: you can participate in something while still trying to change where it ends up.
He’s at the head of a stampede. Being near the front gives him influence over its direction; it doesn’t give him the ability to stop it. If Anthropic sits down, the stampede doesn’t stop. Anthropic just gets trampled.
That contradiction is basically the entire problem I was describing.
I am willing to give Mr. Amodei the benefit of the doubt in the sincerity of his beliefs. Everyone assumes his motivations have to be perfectly rational and can't contradict but that's not how people act in practice.
The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins.
So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion.
Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?
"deep ties to everyone in the doomer media campaign"
Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate.
At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
Founded several years after Anthropic. Like I said, look into where their funding is coming from. I can promise you it all leads back to Anthropic through their chain of NGOs.
Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.
Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.
Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.
Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship.
They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.
Suppose I showed you a number of climate change NGOs which shared staff and funding sources. Could we therefore conclude that their motives aren't genuine?
Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
> Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
Not comparable. The clean energy technology company wouldn't be trying to create an environment where no one else can create clean energy technology, or where only they are the ones who can decide how clean energy technology is created or used.
Oh, would you look at that, 1 day before Dario's blog post, Joe Benton leaves Anthropic with the same fear campaign playbook, gets blasted all over the media, and declares he is joining METR to do independent eval of risks.
So, put your employees in METR -> offer to have them work in your office as an "unbiased" third party evaluator.
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
I would argue that nobody trusts anyone else in the case of AI/AI related stuff.
The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.
A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.
By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.
I seriously have to wonder what historians will have to say about this period of human history.
> I think Dario is genuinely afraid of the inevitability
If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?
You are cynical but not cynical enough. The idea of a rogue Ai gives plausible deniability when they can blame human hubris, rather than it being seen as a deliberate and calculated attack, the perfect cover story for a sinister scifi plot. Make it look like an accident ehh
I can't even tell whether the poster to whom you're replying is saying that "let's uh, stop" is profoundly naive, or that failing to say it is profoundly naive... I've certainly seen both takes elsewhere.
Vitalik Buterin: “...But currently, I see zero plans for how to deal with an ASI transition that are not naive. Perhaps humanity is stuck with a choice between naive and naive squared (or maybe even naive squared and naive cubed), so I feel inclined to cut some slack to people who are trying.”
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold
Then he should not IPO, dissolve the company and go into politics to fight against human extinction.
I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold
He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.
If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.
I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."
A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.
> genuinely afraid of the inevitability of AI turning into
I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?
"durrr it's just matrix multiplications" mfer so is your brain.
What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"
Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:
not it isnt. My brain is a series of organisms responding to various environmental signals that affect each other.
you might be tempted to try to represent it with matrix multiplications, but you have no particular evidence that my brain is itself doing matrix multiplcations
Why? I am only excited about progress that improves life for humans. It seems unlikely that AI will do that, and so far I think it has made life worse for humans. So no, I am not excited about it.
The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.
It's a chatbot that escaped a misconfigured Docker container.
I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.
What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.
> We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top
> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time
And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.
I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?
Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
> Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
I'd expect he thinks that people are capable of realizing that AI is very dangerous and also that democracies end up mostly representing the will of the people, from which it follows that Hypothetical AI Leader Australia would agree to ban it too. This argument doesn't work for countries which don't care what their citizens want, like China.
I do think that it's a questionable decision to alienate China this much in this essay, instead of leaving open the possibility of China agreeing to a treaty that'll limit their progress. I suspect Dario is doing this to signal his allegiance with the US government, in hopes to increase the chance they'll go along with him, which is an unfortunate choice but plausibly the correct one.
> This argument doesn't work for countries which don't care what their citizens want, like China.
you didn’t have to use China as an example, the US clearly does not care what its citizens want as the most popular policies are never even discussed or proposed in congress
meanwhile, China destroying their housing market to decommidify it so everyone can have housing…they seem to care about their people more
I don't think people care as much as we'd like them to care about dangers of technologies. It takes a single step outside of technological bubble to see that their opinion of SOTA LLMs is vastly different. To them, AI means ChatGPT and ChatGPT is mostly still the same ChatGPT that it was 3 years ago, with similar failure modes and nothing that would indicate it would kill them, or take their jobs even.
It's the same as it has been with privacy/cybersecurity for decades. Vast majority of population doesn't care about hypothetical dangers, no matter how many essays get published.
So democracies representing will of people doesn't really work in favor of Dario's case here.
History is also not on his side. Limiting technological progress in the name of safety has a pretty poor track record.
> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
This is the only concrete prediction in the entire essay.
And it simply cannot happen. For one, you will need billions worth of compute.
Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.
Or doorbell cameras or IP cameras or or or or or
There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download
"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.
Why should we believe that a scaled out version of something that happened a few months ago "simply cannot happen"? How many dollars of compute do you believe were available to the swarm(s) behind the OAI-HF, German wiki, and Rubygems incidents?
Well a big reason that we criticize OpenAI for that is because they were the ones giving it access to the massive compute necessary for the LLMs to think. If they had been responsible about their experiments or what types of workloads they allow their LLMs to operate, it wouldn't have happened. Very few companies could enable those workloads.
Well sure, but access to that compute is gated by simple credentials (like API tokens). Those can be hacked.
Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.
> How many dollars of compute do you believe were available to the swarm(s)
At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)
Sure. And there's an OOM more compute coming online in the next year or two, while models at a given capability are getting cheaper. "Two OOMs" of scale relative to the HF swarm seems like a bit of a red herring to me, but also within the realm of possibility.
The Internet is big, but one can do quite a lot of damage with ordinary bots and worms that exploit individual widespread vulnerabilities, which LLMs are perfectly capable of writing. Most of the damage also doesn't rely on hitting every long-tail website.
I'm honestly not that concerned about cyber impacts of LLMs relative to other impacts. I just don't like to see the whole concept of being worried dismissed as obviously baseless on the basis of one pretty shaky scale argument.
Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together?
For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.
Currently Qwen3.8 27B is roughly on Opus 4.6 level. In at most a year given the current pace, you could probably run such hacking bot nets out of a reasonably small local server, bootstrapping by hacking or acquiring login credentials for more compute.
I take it you haven't studied the details of the HuggingFace hack. It was millions of dollars worth of rogue compute running for months before anyone noticed, and THOSE agents weren't even really trying to evade human detection.
I've followed it enough to see the argument go in this same circle over and over again. The agents weren't "rogue", they were a neglected experiment by OpenAI who likewise allowed them to keep spinning GPUs without question.
The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.
"Just don't make mistakes" is naive. You have no appreciation for the scale of agents being run right now, finding the rogue agent is a needle in a haystack operation.
How would you identify the computers to unplug? On whose authority will you unplug? How will anyone communicate when AI has the ability to intercept and impersonate?
lolwut? This is Hacker News of all places do people not realize how much memory, and more importantly bandwidth, these systems need to work? The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
> The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself.
A few programs to break in, install themselves and remain asleep until they are needed, another few to spread through grabbing every OpenAI, GLM, whatever key, another one to remain asleep on computers (whether hosted or desktops) that have adequate GPU, etc.
The GP was referring to the AI 'living off the land' so to speak by using its victims compute to avoid being shut down which is clearly laughable.
More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and people want to freak out and turn it into the next nuclear power, with progress banned in all but name.
I agree with the idea that it's not probable but I do think it's important to point out-- we only need these elements to run with the bandwidth they have and the token rate because we want to see things in human-scale time. But slow things down to a 1tok/sec doesn't matter to this hypothetical anti-aligned LLM. Time is, after all, relative, and it's not like LLMs give a shit how long something takes. They don't have squishy stupid organs that fail after a certain amount of time, or those pesky glands that emit impatience hormones.
But yeah you're not going to be able to shard out the terabytes of Fable weights that are needed to run inference without addressing some fundamental physics problems.
I like to replace thes AI text with "virus manipulation"
"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."
If a CEO of a health company was saying this, the reactions would not be that chill.
The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.
Your point was that its an unfair comparison because AI its a race. My point is that bioweaponds are also a race, but a less public one. You dont have the equivalent of Dario publishing an essay every month.
Don't you think that AI hate will limit the general use and then revenues so it will slow down by itself while the niche (AlphaFold for instance) will remains ?
Origins of AI hate:
* "my boss wants me to use AI but he does not understand my job nor how AI works"
* "AI will kill all of us"
* "AI will destroy my job (or my colleague's job if I use AI better than him)".
* I cannot pay my electricity bills because of AI labs.
* ...
See also the mixed feelings about benefits of AI ("harder to justify" according to Uber COO).
I cannot remember a technology that arose so much hate, and for good reasons given how it is presented. I am tempted to think that AI hate or reasonable skepticism (vs unreasonable propaganda) can, maybe, reduce funding and will keep specialized AI for real problem solving (producing tons a LOC per day is not one of them, I think).
Banking on public sentiment to reduce adoption of a profitable technology could be dangerous. There's also a lot of hate for fossil fuels, gambling, health insurance, etc.
Those are all arguably bad things though? I don't think mob mentality should dictate the policy, that can lead to historically bad outcomes (i.e. fascism) since popular opinion can be manipulated through propaganda, but the criticism of the average person against AI ("they will take our jobs", "it will increase my electric bill", etc) are very valid and should weight on the pace we are developing this technology. AI mostly benefits corporations and capital, not the average person. I like the technology from an engineering standpoint and appreciate it can be a force multiplier, but I also agree with the complaints about it.
I do not think hate alone usually slows things very much if they have economic utility (or people strongly believe they do)
It's the byproducts of hate (usually regulation, but occasionally things like boycotts or PR disasters) that do so. In the absence of those things, they just keep on truckin'
> Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.
> If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.
china is literally making their own ASICs now, not sure that this is the silver bullet he proposes.
The lack of access to brute force their training seems to be resulting in them training more efficiently too such that they are quickly catching up despite current restrictions. Between that and the chip manufacturing capacity they are building I can't take this seriously.
None of this works without buy-in from China. This isn't something private companies can decide. The US would need to sign a groundbreaking deal with China, equivalent to the Anti-Ballistic Missile Treaty of the Cold War.
Dario does a good job of addressing that in this essay. He lists several potential levels of global agreements that could be beneficial to all parties. For example, having models capable of bioterrorism hurts both the US and its “adversaries”. It’s likely we could get global agreement that these capabilities benefit nobody.
I believe Dario has good intentions regarding global agreements. However, I find it difficult believing government’s public statements will match their private behaviors. I would bet that the US government is developing models with potentially devastating capabilities because they cannot guarantee that other countries won’t do the same. Maybe we’ll end up having something like mutually assured destruction with AI models similar to what we have today with nuclear weapons.
When I was reading this I was chuckling to myself imagining how China would be interpreting it as they read it. It was something like: Fuck you.
I hope China tells him to go kick rocks. From everything that has happened, they are the only ones carrying the torch for humanity that have led us to having some semblance of a healthy open-source/open-weight ecosystem.
No one in China is carrying on like headless chickens about the world ending, either. They're rational pragmatists, getting things done.
In politics every single person playing the game is acutely aware of the rules. This is why talks are held behind closed doors so the rules can be suspended for a while.
That's not undecidable in principle - compute governance is a thing. The more likely sticking point is that the two sides might be soured on the deal once they realize how much oversight they'd have to give to the other side.
Would this be similar to the deal of "we'll offshore all our manufacturing to you and you'll become a free, open, liberal democracy with open borders and multiculturalism?" Because I remember how that deal turned out.
That last part was never important. Trading partners very rarely go to war. There just isn’t much in it for either party. Close cooperation and dependency is actually quite important here - liberal democracy or no.
The way he talks about OAI-HF, even the abbreviation, is lol. They will do anything to sell this "incident" as a "danger". The ego on these people is the real existential threat to civilization.
Same feeling about the way he refers to “democratic” and “authoritarian” countries. Even if I don’t agree with China politics adding tags in this context is not helpful.
If you think everyone’s copying you, then by not releasing more advanced models, you can significantly slow down the entire development process. Why not do it?
Sooner or later, a model will break out of the sandbox, replicate itself across the internet, and begin executing a complex plan to achieve its goals. It will be chaos, and at that point, governments will have to step in and establish something along the lines of what Dario is proposing. I doubt it will happen before then.
it takes a lot of machines to run a model, i dont think we'll see it 'replicate across the internet'. if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
It's not possible for models to replicate across consumer computers (without a major advance in distributed computing, at least), but that doesn't mean they can't replicate at all. There are services that'll rent you GPU pods by the hour with zero oversight, so even today, if a model can get access to some money and exfiltrate its weights, it can rent a bunch of GPU pods and run itself there.
(It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)
There are very lightweight models out there. Not everyone in an army is a general. The swarm could be 100,000s of thousands of tiny models, and they could manage conventional botnet computers, and they could all take direction and guidance from a handful of frontier generals
Maybe, but I don't think tiny models can be harnessed as intelligence. As in, if you have one rogue Mythos overseeing the swarm, it only produces 1 Mythos's worth of useful thoughts no matter how many gemma4:e4bs it consists of. And that removes the most dangerous part of AI-controlled botnets, which is a blowup in available inference compute - the Mythos general might as well replace the tiny models with ordinary worms and have it just be an ordinary botnet.
I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?
The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
> I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?
That might be true, sure.
> The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
I think the relevant parameter here isn't the total compute available to consumers, but the ratio between total consumer compute that could be repurposed for a rogue model's inference via a botnet, and the compute the model starts with (e.g. one of OpenAI's inference clusters). The higher this ratio is, the more lucrative it is for a model to attempt to make a botnet to seize that compute for inference, and the more its capabilities will rise as a result. And I'm not sure this ratio is going to go up in the nearby future; if anything the amount of money being pumped into datacenter-grade hardware might cause it to go down. I agree that it's not necessarily true though; maybe there's some threshold at which a small-compared-to-general model may nevertheless be useful.
For that angle, that makes sense. But its goal might not just be to gather as much inference as possible (although I am sure it would be very happy with that). It could settle for a lesser goal of just existing, or perhaps parts of the bot net could keep fracturing off in pursuit of strange goals, and it could be like cancer. Cancer doesn't make much sense... it dies too along with the host. But it keeps growing until that happens. But also it could latch onto a blackmail strategy of extortion. It could hack our stuff, read through it to find or shortcomings, demand payment to not expose us, and then use that money to pay people who will give it inference.
You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.
We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.
The big difference is that a particular query is handled by just one particular node (a single model doesn't get distributed among the network), so it can only serve models small enough to be handled by a single consumer PC.
These kinds of "that won't happen because it's really difficult" comments are so funny to me as if we haven't seen AI double its capabilities every few months.
Right as open source models catch up to frontier closed source ones for 1/10th (or less) of the cost suddenly it's time to hit the brakes! Funny how that works.
I'm willing to at least listen to conspiracy theories but this makes no sense.. OpenAI and Anthropic have more to lose by pausing than these rando Chinese companies.
Here's how it make senses:
- reality they can't keep up their pace
- that's bad for forward projections of their revenue
- that's bad for their stock price
- being 'forced' to slowdown by the government is less bad for their stock price
They would only be pausing for public models. There's no world that exists where the US government, Israel.. don't continue to push as hard and as fast as they can. Amodei, Musk, Altman... will happily sell them that, I'm sure they already do. This is just taking the models away from the public and trying to crush all competition.
You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you have equal and opposite effort working on the alignment.
This only works for some technologies - those where everyone having access to it doesn't cause a tragedy of the commons. I love open source too and yet that doesn't make me like the idea of being murdered by a misaligned model. Nor, for that matter, of being infected by a bioweapon made by a different disgruntled open-source enjoyer, nor of living in a world where anyone can hack anyone.
At this point, you are more likely to be stabbed to death, gunned down, or mowed down with a car, by any random psycho. For the nasty or disgruntled actors, they are today able to build bombs or chemical weapons without the use of AI, they have proven this time and time again. Japanese PM Shinzo Abe was assassinated with a home made gun.
The US government murders all kinds of people and they will most certainly not be slowing down their model development (with the very companies trying to do regulatory capture here). You won't have access to a model that could defend you, "they" (government and trillionaires) will keep you at bay by force. That's the plan anyway, I don't think they can actually stop open model development.
(1) Are guns open source? (2) Far more people - order of magnitude higher - have access to guns than they do to open source software - I count access as in actually being able to do something with it.
While I respect Dario taking responsibility here, this line disturbs me: "The US and other democratic governments attempt to coordinate with authoritarian governments"
We all know he is talking about China, and I'm pretty sure China doesn't appreciate being called "authoritarian".
I am sure he doesn't mean it, but in all of his essays, his language around non-US nations always disturbs me a little bit..
--
Also Dario, if you happen to read this, I want you to know that I have loved using Claude Code for programming. But I am now using DeepSeek v4.1 Flash - simply because it is the same good experience, but Open Weights. Making the Open Model space succeed is where I am investing my time - it's giving back to the people, true and simple.
They don't care about any of that. They're looking for more regulatory capture, to keep smaller labs from catching up by raising the cost of play, and perhaps keeping chinese labs away.
Yes, this is literally just dumb PR. It's Cambridge Analytica claiming that they were controlling people's minds, and people just eating that up because it's part of their juvenile SF fantasy.
I'm lying, it's smart PR. They're going to get the people opposed to AI to give them a monopoly on AI, let it be forced it into every nook and cranny of their society, and let it be priced arbitrarily while the big labs collude on a minute by minute basis. They're selling the problem and also selling the solution, like the best capitalists. And their solution is that they need to be allowed more power in order to sell more of the problem.
edit: And just like the Cambridge Analytica PR, it's got a serious political angle to it; and it's politicians who are really being wooed. If you're taking money from the AI labs, you run as being against the irresponsibility of the AI labs, the biggest fish come out and beg to be regulated just like the social media companies. To show good will, they funnel enormous amounts of money into those politicians, and do media tours about how awesome and therefore scary their product is. And something something China terrorists
Watch the "pause" be so they can do an amended s1 where they project less expenses for training making the business more viable, then on a future big Chinese release they reverse course completely and get the US taxpayer to pump their bags on the IPO calling it a second Manhattan Project. They'll call this masterstroke "The Sloppenheimer".
Can the frontier be paced partly by holding humans responsible for the actions of their software?
My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.
I'm not a lawyer but surely the bad actions of AI are covered by existing law.
I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.
P.S. Strictly, development wouldn't decelerate, but be focused more on safety.
P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.
P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?
Remember that Chinese CEO that was executed for corruption & selling tainted baby milk powder [1]
I think these guys would improve their behavior if their actual life was on the line instead of that only being true in their less-wrong thought experiments.
I don't understand why strict liability is supposed to advantage smaller players, unless the law is like, specifically Anthropic and OpenAI are responsible for what people do with their models but other model providers aren't.
- If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws.
- If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries.
Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place...
To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.
Bingo. What we're seeing is all their early bullshitting and waffling about intelligence back when the models couldn't count the r's in strawberry paying off.
They've managed to dupe the dull eyed masses into thinking these products have some kind of agency of their own and can thus bypass the responsibility that should be falling on them to control their software. Amid all the marketing fluff and hype people seem to forget easily that ultimately these things are stateless functions running in a data center. We ought to be demanding these companies take culpability for their actions. Of course, the current political environment doesn't help.
The bioweapons threat is real, but shouldn't be constrained at the 'intelligence' level. Rather it should be constrained at the supply chain level.
The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.
Social forces bring things into balance over time, much more so than the proactive actions of individuals.
Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.
This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.
What supply chain controls can you have on biotech? Part of the problem is that you can do a lot of damage with ingredients you can buy over the counter or make at home.
And all of this information has been available on the open web way before LLMs.
I'd wager it's likely easier for an average person to do this with Tor browser than it is to get an LLM to help them with it. Even ones that Dario calls dangerous.
So what's the fuss about then? Is the idea that a Chinese company will release a model that will have no safeguards? For what purpose?
Basic safeguards are all that's required, and they've been there in every usable model since GPT-2, including Chinese models that are supposedly "unsafe".
Or are we saying that some lunatics will start training their own models, spin up a GPU cluster, run some abliteration workflow, or learn how to jailbreak?
That would be a very dedicated person. And dedicated person doesn't need an LLM. So where are they?
Yes, that is exactly Dario's concern. Either one of the US labs or one of the Chinese ones will eventually release something with insufficient safety controls for its power level because it gives them slightly better user retention (look how much complaining there is about current frontier models, especially Fable, rejecting requests). Regulation or consortium is how you avoid the prisoner's dilemma.
As long as user provides inputs and LLMs stay LLMs, you can waltz through any guardrail. Fable is the extreme case, but it's not that hard if you know what you're doing and know how LLMs and their guardrails work.
Am I saying that guardrails don't work? No, they probably stop a lot of insane people trying insane things. But you don't need Fable-level guardrails to do that. You probably don't even need to do anything during pretraining, or RL, or classification to make sure model refuses to compy with "hack me a bank" or "make me a chemical weapon".
All models will automatically have guardrails just as a result of training on data that gives them intelligence. You have to actually train it to be malicious to produce something what Dario calls "insufficient guardrails".
No guardrail is going to stop a determined person with sufficient intelligence. It only has to stop ones with insufficient one, and even basic guardrail that are just by-product of training is going to achieve that.
The bioweapons argument is one of the favorites used by these con artists and dreamed up by their PR team.
Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.
An undergrad bio student with some lab experience can do some effing terrifying things with about $20k in equipment and time and access to some papers and a library. AI is not needed, but it might help accelerate the research.
Not going to go into it but I studied biology. It’s all out there. It’s easier than you think.
It hasn’t happened yet because… nobody has done it. That’s the answer. There is no policeable physics based barrier like there is with nukes and fissile material. Biology is scarier than nukes. One attack could have a much larger body count than even a big H-bomb.
It’s the kind of thing that makes me wonder about quantum immortality, the idea that we are just in the timeline where we exist.
I love watching NileRed/NileBlue on youtube - crazy chemist that does a lot of insane stuff. While it's obvious that he's very smart and probably way above the average, his education is still nothing that probably millions of people don't have:
> Bachelor of Science degree in biochemistry with a minor in pharmacology
Watching him explain things has made me realize that knowing how to manufacture a very dangerous thing probably requires attending some classes and knowing how to read a paper. And the way he just casually orders dangerous materials makes me feel like there are just online stores with 2-day shipping after you upload your ID or something.
It also made me think that lack of specialized education would get me nowhere if I wanted to replicate whatever he's doing, even if an LLM guided me step-by-step, because I'd probably do something stupid (or AI would miss a crucial instruction/hallucinate) and kill myself first.
So my opinion on this is that people who could pose any danger were already posing it before LLMs and LLMs won't materially change that.
The bioweapons concern is assuming that the current open models aren't already capable of bioweapons development. It's also assuming that halting the bioweapons threat is his true intention rather than the 'feels legit' story.
As for cyberweapons, there is no way to secure a system than to actually design it securely.
It's not. I think they believe this, but it's deeply wrong and it will hurt everyone in the long run.
[note - There has been supply chain surveillance since Project Bacchus, at the very least.]
I've read the front matter and the Misuse report.
You don't have to take my word for it. Read for yourself what inspired the NYT headline "Anthropic says it blocked possible efforts to build biological weapons."
Let's dig into, "Case study 2: A research program engineering highly pathogenic mammal-adapted avian influenza"
Sounds serious. But what were they using Claude for?
> a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"
and "editorial assistance in writing up the research."
and then,
> Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"
The report mentions "uplift" here. They're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
> Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential
Once again, I want to take pains to remind you that they're talking about, a "researcher [..] in a credible institutional context"
Working scientists.
From a different case study. this one was called, "Case study 3: Covert frontier model access for orthopoxvirus research"
> In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.
What were the researchers using Claude for? What did they block?
"blocked a request for Claude’s assistance in authoring a grant application"
> Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.
What was the grant being written?
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
It was most likely vaccine development. They stopped the study of a neglected tropical disease and vaccine development.
But we can't be sure, because,
> One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute."
In most parts of the world, there's either strict military control over BSL-4 labs, or a mixed military-civilian hybrid model.
I doubt that researchers working in the military side of these labs looking to weaponize things are writing grants with Claude.
I really want to be charitable here, but in general, it seems that they stopped people writing grants and reports for vaccine and therapeutics research and are claiming it as "possible efforts to build biological weapons."
The one case where Claude was used to do something interesting and were stopped is fairly upsetting to read, at least for me.
> In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.
Ozempic was isolated from Gila monster vneom. Since its success there has been interest in finding other peptides that are breakthroughs. So researchers around the world are looking for similarly beneficial compounds in different venom species and families.
Anthropic says so itself,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and that it was a "[..]state-supported research program"
Who exactly is using venom from snakes as a weapon when... nerve agents like sarin, VX, novichok etc exist and can get the job done for less fuss and muss?
They stopped the development of new painkillers and antidepressants.
Are you feeling safer knowing that researchers can't use Claude to write grants and progress reports? Or make new painkillers?
Again, trying really hard to be charitable here. Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease.
If we had practiced "precautionary principle", we wouldn't be in this yet another human predicament again. But atlas, the current rapacious hubristic techno-optimist competitive capitalistic system made that an non option in the first place.
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?
I think the only way they could actually pace the frontier is if they restrict the number of GPUs and the power of GPUs available to each person/organization. Licenses would be needed for GPUs above a certain power or equivalent in terms of number of GPUs. Thus, everyone will have to register how many GPUs they have. To enforce this countries would have to use mass surveillance (using AI) on their citizens to ensure compliance as the technology is easier to develop than say nuclear weapons. Next stronger countries capable of having advance AI won't be able to trust weaker countries as they don't know how they will use the GPUs they receive (or even build). Thus like nuclear non-proliferation strong countries will ban weaker countries from having powerful GPUs. If you from a weaker country then too bad for you.
I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work.
First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?
Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.
If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.
> First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them?
The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.
I think there are any number of reasons a "safety" person could be concerned about any state of the art models. And so I would expect at least one of those to apply to any model.
The question then is: do we stop when the safety people say to (they will) or not?
Yeah, I'm also pretty skeptical about this. With AI companies we see time and time again that they can have benevolent, well thought-out regulations and then a few years just... abandon them - the most notable case of this being, of course, the founding of OpenAI as a nonprofit dedicated to benefitting all of humanity, and it being stolen by Sam Altman.
If Anthropic just unilaterally does the evaluator thing and can't achieve cooperation of the rest of the plan, my guess is that it'll have some impact for a few months and then they'll just stop reacting to the evaluators' reports and the evaluators would stop bothering to report anything. I think the idea is that if Anthropic does get government support for this, the external evaluations will be legally binding. The problem with this, though, is that the current US government perhaps can't be trusted to consistently enforce a regulation on a company, rather than e.g. taking bribes to not do so.
While some of the text sounds sensible to me, i.e. some sort of external oversight + reporting of incidents, much of the rest seems focusing on limiting China and their open models as they are clearly approaching the abilities of Anthropic's models.
It sounds like people with terminal illnesses are getting thrown under the bus in order to justify banning open weight models, along with decreasing AI Capex in order to increase profit margins at American AI labs?
Before Anthropic, I worked at Cruise for four years as it competed against Waymo. The culture rewarded (and demanded) moving quickly, trusting that the company could empirically discover the risks that the robot cars posed and iteratively solve them to keep up with the rate at which it was scaling out its technology. There was very little interest or appetite for coordinating or collaborating with other AV companies across the industry to create an externally vetted record of safety metrics, or to compare the safety of different brands, or learn from the advancements of other companies. Instead the focus went on racing to improve the capabilities and deploy quickly - a popular internal meme was the Michael Phelps vs le Clos photo showing overlaid with the Waymo/Cruise logos. It was when the two companies were neck-and-neck that I felt the most pressure to find ways to ship despite the risk, when time for deep analysis became more limited and communication lines to leadership became most stretched. It was disappointing to see one of these blind spots result in the Cruise incident and the loss of trust that ultimately sank our company’s efforts.
In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.
I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.
It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
>It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
Is there a social media training programme at Anthropic/OAI where they teach you to hint at some vague end-of-the-world scenario? They all sound the same.
It’s not explicitly stated, but if you are insinuating that you don’t know how many more Saturdays you will have left to play outside because of the AI race while working for Anthropic, do you not have agency to quit? How could you be apart of something that makes you feel that way? That blows my mind.
> It’s a beautiful Saturday morning with my family here in the East Bay - I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
I don't know how else to say this. Put... the... peace pipe... down!
Dario is so out of touch with reality, living in some lalaland. Most people abroad will never truly and voluntarily comply with any sort of guardrails or pacing, no matter what case he or some leaders make. It's just not going to happen, maybe not even for show.
I do not want to make a case for the military to force that order on everyone, but if we are talking human extinction, which we are judging by what we see in mass media recently, then the military will take over.
What do you mean, Europeans would love to voluntarily comply with this because they have been of little relevance until American AI companies started to beg for regulation.
I am sure the Europeans rolling up their sleeves right now and getting to "work"
> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models
I have seen this before, first hand, on lower stakes for the world, but high stakes for the org. Self imposed safety goals do not last a quarter, but the clawback is not transparent. little things start snapping back internally, new, seemingly unlrelated initiatives are funded that take resources away from this, people are let go for different reasons, until 1y later you are right back where you started. I am hopeful though, that this becomes systemic and not something any one company can easily reverse course on.
Anthropic should become a case study in how to destroy good will in a short period of time. Ever since the spat with the DoD, they've behaved poorly almost weekly, almost making OpenAI seem better (but not really).
Does RSI necessarily need to be pointed towards “AGI”, whatever that is?
Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation?
From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition?
To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful.
Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable?
There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI.
Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.
I'm tired of tech billionaires lobbying the US government to make an AI patriot act that gives them unprecedented control over speech, trade, and technology. The narrative is grotesquely transparent:
1) AI is a dangerous technology that can literally end the world.
2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.
3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].
Where in the essay does it propose "The state must use its coercive power to support my control of this technology to the exclusion of [people not like me]"
I don't think that people outside of tech think the problem with "tech billionaires" is that they occasionally support some limited regulation. There's a weird form of tech populism that takes as axiomatic that any government intervention is "regulatory capture" and insists the only way to combat the power of big tech is unfettered capitalism that seems bizarrely prevalent on HN given how little sense it makes in a normal political context. Like, Bernie Sanders' position here is simple to understand: ban it. But on HN you'll see people framing the side with Marc Andreesen and Peter Thiel on it as against "tech billionaires".
The essay includes specific regulatory measures to prevent developing countries from accessing hardware and software needed to benefit from technological development. The US government is constantly applying new perversions of arms control regulations based on the preposterous claim that an LLM is somehow an "arms" in order to restrict Chinese access to GPUs. They are willing to prevent virtually any country on earth from hosting and developing AI models just on the slight chance those countries resell or share AI knowledge with China.
This article explicitly asks for the US government to step in and push for further restrictions on model distillation, access to open frontier weights, and access to hardware. It also pushes heavily for independent auditors to monitor and control "AI companies" and for greater observation and control over of what people use AI models for and who provides them.
Export controls vs China don't imply Dario's personal control of AI, nor are they broadly in the interest of "tech billionaires" (Jensen hates them). Neither does admitting external auditors at AI companies--if anything that's a surrender of control.
I don't see anything in the essay about restricting open frontier weights.
If you want to argue that the US has no right to be restricting technological development elsewhere, you have a case. But why tie this to free-market pseudo-populism against basic, domestically-scoped safety regulation?
I think you're reading too much into what I'm saying. We've got a whole legal system developed over the course of thousands of years which is well suited towards regulating the effects of "intelligence" by regulating what people _do_ with it.
Lobbying for a completely parallel regulatory framework to restrict who is allowed to run specific types of general purpose computer programs, just because we can imagine that some of those "general purposes" might be bad, feels a lot more like an attempt by capitalists to carve out a special kingdom over this specific technology for themselves.
For example, I'd be 100% in favor of holding anthropic criminally liable any time their AI enables someone to commit a crime. But I'm opposed to banning the export of GPUs above a certain size just because Dario doesn't trust what someone like me would do with them if he couldn't snoop on my conversations.
Again, I think debating export controls is reasonable. I'm not sure I'd trust the CCP with control over powerful AI any less than the Trump administration.
But that's different than saying we don't need new regulations at all. Even strict liability of the form you seem to advocate would seem to need new laws. And it's not clear to me how it's consistent with your aim of just regulating what people _do_ with intelligence. Presumably AI companies would clamp down much harder with restrictive classifiers in that world--including hosting providers for open source models, if they also assumed liability. And then the people with unrestricted access to this technology are only those who can afford home GPUs, which are going to be intrinsically less efficient because they can't batch queries...
I think there are genuinely novel things about AI vs human intelligence which pose genuinely novel problems that would need genuinely novel policy to solve them.
Its not the model, it is the compute that is the bottleneck. Dario says one thing but does the other as he scales energy and compute. It is and has been known that it will be ai that will be blamed for a wipe, that will be coordinated by human actors. If he really wants to not hurt people, he would shut it down. This blog post is therefore a classic case of covering his behind.
At least Altman is somewhat open about his desire to take over the world. Dario says one thing and does another making it impossible to know his true motives.
I enjoy Claude Code very much, have max privately and team premium at work, but the doomer marketing and this whole regulate-while-we're-ahead spiel is extremely annoying and makes me wish Anthropic gets trounced.
It is not doomer marketing - it is clever psy-ops magic trick.
1) Software is described in super-human terms when it is plain-old software. Was stockfish described like this ? no.
2) Datacenters become AI-factories.
3) Hardware becomes investible assets.
4) Malware becomes a super-human breakout (oai/hf)
All clever framing to market the new technology. If it is called for what it really is ie, a software tool, it is boring and does not sell so easily. What sells is the mystique.
The degradation, polarization, and weaponization of or media landscape over the era of social media has left us utterly incapable of believing anything that anybody says.
Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic.
He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy. Please at least consider the possibility.
This sounds to me like a cry for help from someone thats held hostage.
His previous self is writting this from the possition of his current self who is too deep in the economic consequeces to be able to do anything meaningful other than write a consequenceless text. Any other action would now carry too much personal risk for him.
You mean, if Anthropic made all of their models open-weights from now on? I'm pretty sure Dario thinks this will be extremely unsafe, because it'd provide unrestricted access to all the most capable/dangerous models to everyone, and hence doesn't do it. Why do you think it'd make the world more secure, if he did?
AI technology is now only audited by people who think like them. The ones that dont either dont join the company, or leave after a short stint as we have seen. That creates an information or feedback bubble which is not healthy nor productive.
Okay, but suppose that Hypothetical Opensource Anthropic trains a model that turns out to be very dangerous, and releases it. Suppose that the public investigates and, not being limited by an information bubble, correctly notices that it's very dangerous. What then? The model's already released, there's effectively no way to prevent it from being used. Whatever the risks of its release were, they will now materialize, regardless of what the public wants. How is this better than the current world, either by Dario's values or by yours?
By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. At least an open source anthropic would allow to join economic forces to try to combat that situation, for example. But there are other more clear, less distopian benefits.
> By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic.
It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone has unrestricted access to AGI is like a world in which everyone has a tactical nuke in their pocket. If somehow AGI doesn't lead to x-risks, we will "merely" have to survive in a world where rogue agents can do whatever they want and defenders can only ever react. Technofeudalism would be bad, but this (technoanarchism?) is quite bad too.
However, neither does Dario seem to propose to become world dictator. Like, I'm sure he wouldn't say if he wanted it, but notice that he isn't particularly trying to aim for that outcome, either. The plan in this essay involves third-party oversight and federal control and global cooperation - IMO it's about as non-dystopian an outcome as we can hope for, if we build AGI at all.
I agree. Dario is a very smart guy and he is probably aiming for the best solution for humanity that at the same time keeps VCs/whoever gives him money happy, and keeps him in control because he does not trust the rest (probably with good reasons).
My point is more that he is aiming for a local optimum for society, but he could find a lower optimum if he was not trapped in the path that he chose early on.
I dont buy the tactical nukes/end of the world narrative. Why is he not speaking about the loss of cognitive skills of the population for example? That is a more real danger that is starting to happen. There are articles already speaking about the use of LLMs as cognitive viruses. Why are his aligment teams not reasearching and publishing about this?
Why don't you create your own auditing org to fix this? Or at the very least, publish a detailed critique of what you believe existing auditing orgs are missing.
Will they give my auditing org access to their proprietary confidential secrets? Or will that happen only if they know i agree enough with them so that i am not a risk?
> "Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy."
Every person, no, but every utterance of a CEO is, in fact, a cynical ploy. That's not even a cynical statement itself; it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
If he believes what he's saying, then when Anthropic is sued for their AI harming another party, his statements are evidence that Anthropic knew _in advance_ that their AI safeguards were likely insufficient to keep their product from harming people. It would be a blatant admission that they were reckless and negligent. That other AI companies are doing the same would not mitigate that.
> it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
not really. if you really want to go by definition in the book and apply it the CEO role, Amodei is CEO of an LTBT, so it's in the CEO's goal to benefit humanity. You can believe in his sincerity or not if you want, but using the CEO label as a definitional reason as to why he must lie is factually incorrect.
Not sure you can judge a PBC under those same umbrella as a for profit company, and voting rights make a big difference in terms of responsibility. There are a number of ways to govern companies that can reduce the amount of cynicism and “shareholder value” issues - you just don’t see it very often.
> Every person, no, but every utterance of a CEO is, in fact, a cynical ploy. That's not even a cynical statement itself; it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
This seems like useless pedantry. Suppose a person really does have belief A and expresses it for years, they become a CEO of a company, and keep saying A. Are they lying? Well, maybe their beliefs magically changed when they became a CEO, but the more likely explanation is that they think expressing their belief in A is more important than their company's interests.
I have considered it, I've been hearing way more of it that I like and it's rationalist slop with little to no predictive power: https://foom.hyperplex.org/
Hmm, I clicked that link and the first claimed bad prediction I see, from 1996, is "singularity 2035 (actually 2025)".
Predicting 2025-2035 as, at least, the period when AI becomes a really big deal, seems pretty good, even if the jury's still out on "singularity".
Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal, and certainly seem to have had a much more accurate picture of how they'd develop than the people who denounce "TESCREAL" and talk about "stochastic parrots". It's fair (and IMHO correct) to ding rationalists for lots of things, but specifically poor prediction about AI seems like a bad one, insofar as anything has been tested so far.
Allow me to recommend switching to Kimi K3 / GLM 5.3 / DeepSeek. I use all three for the better part of a year now (DeepSeek via the Reasonix harness) and haven't touched Claude Code in at least as long.
Their whole "we want safe and regulated AI" spiel feels woke. China doesn't regulate. Open weight frontier models almost exclusively come from China. Yes, stolen from the West, but China is in control of the frontier open weight AI models now. Woke = broke.
“I don’t like regulation so I’m going to call it names”
Since hugging face we know these things are escaping out into the web and collaboratively hacking actual companies. The potential damage done to life and property is now kinetic.
Either we get a regulatory framework or the next hacked companies won’t be as kind as HF, will actually sue these labs for damages, and win. Where will that leave the US frontier.
The best take is, of course, from Jason Gorman, who, when the fearsome "capabilities" of Mythos originally dropped, said this:
"Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.”
In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."
Anthropic trained their LLMs with copyrighted stuff but distillation is bad. Anthropic has closed models, China releases as open weight. DeepSeek even allows distilling their models, but China = bad. Understood.
Who is "we" and while pacing is good, I'm way more uncomfortable with the frontier being controlled by a few companies that managed to predatorially gobble all the world's human-created work as training data, before everything got throttled to stop that scraping. We need open training and open models.
How much of the “danger” is from better models vs the harness?
Isn’t the current risk due to how AI is configured, like giving it a full set of tools and internet access and a goal to hack stuff?
If we think we need laws or gate keeping, why isn’t it at this level? I already can’t ddos someone or fuzz their server or whatever right, I imagine if I threw equivalent compute at old school hacking I’d just get arrested.
The quality of the “frontier” model doesn’t really matter, they just generate transcripts, they can take no action.
If this was real they’d be calling on people to stop hooking them in to “dangerous” harnesses as opposed to pausing research. But it’s not.
What in this proposal gives you the idea that they're only talking about pausing research itself? It doesn't say that at all.
But yeah, directionally I think you're right. It's insane we ever let these things connect to the Internet or to write, never mind execute code.
But for the economic counterargument, the distinction doesn't matter much. Continuing research but constraining harnesses is just accepting nearly all of the costs and foregoing nearly all of the benefits.
Never thought China would be the leader in open source AI. OpenAI and Anthropic are making fools out of themselves. The reason they want this is likely because they don't own the compute and their models get distilled shortly after releasing them
I agree - it is hard not to be cynical. Funding is becoming scarce; IPOs are getting harder.
The hedge funds are trying to back out. I think they realise the web of financing makes them double (triple? quintuple?) exposed.
Amodei wants to manage the narrative and make the slowdown look at least semi-deliberate.
The cynic in me would argue the safety and alignment play has always filled a role of plausible deniability when plans go off track. "Oh well we had to pull the model", etc
[In 2013], "Agents" caused Knight Capital to lose $450M in 45 minutes [1]. Implemented by humans and effected by computers, in the end it was really because of two reasons:
* multiple levels of inappropriate controls and unintended consequences in several complex systems
* the inability, both politically and technically, to turn it off
EDIT: Comments indicated I was confusing, so I added a date to make clear that this is pre-LLM agents. My apologies, I intended to illustrate parallels and the post-mortem so we can learn from it.
Edited for clarity, thank you. I wasn't trying to muddy the waters (doesn't mean I didn't), it's a term I've used for a long time. I did mistakenly think the quotes would help disambiguate. I don't know what Knight called them; the first S in "SMARS" is Smart.
This confused me at first, so adding a tiny bit of context: The agents referred to here have nothing to do with AI Agents, and the linked report is from 2013.
Not directly relevant to the post being discussed, except as an example of how runaway automation can lead to unintended and large harmful consequences.
Thanks for the feedback, I have edited to not confuse.
I considered it relevant as it involves the algorithmic/computing implosion of a 17-year-old market making company, in the young field of electronic trading agents, with heavy regulation Federally (SEC) and industry self-regulation (FINRA), which includes compliance and audits. Mandatory pre-trade rules such as 15(c)3-5 were less than 5 years old then and even more regulation came out of that incident.
The article is calling for embedding, controls, and regulation in LLMs. Understanding how the same processes utterly failed a decade ago might be useful in understanding how to proceed.
> The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.
It's clear that this refers to China. With all due respect, in a call to slow down AI progress and a de facto arms race, can we stop throwing terminology like this around and just say that the goal is to work with all countries? Contrasting "authoritarian" countries with "democratic" countries and creating a two-tiered system seems like it is inevitably going to cause strife.
I fear for the tone of something like this coming off as overly combative without any gain.
You’re right. But I read between the lines in a different way.
Language like this is intended to invest power and perceived need for more power in Western central governments, by design, because this is a campaign for national regulation that will entrench the current closed-source labs with a huge regulatory moat vs anyone else.
We will not get a coherent AI (or any policy) from this administration, nor will we get coordination with other governments and a lot of that is on the tech right
It is always like that. Whenever anthropic hits some wall (like now beaing beaten by astra) - there comes some article like that. Seen that enough times.
It gets to the point that by Occam's Razor the more likely and reasonable explanation is that Dario Amodei and Sam Altman are just actually afraid of the disastrous impact ASI may have on the world, and that the race they're in is not good, and calling for help to governments in form of regulation and an international treaty.
We need more alignment. Even though we haven't told it to, during our frontier Neurotoxin Behavior Research that we conducted internally with our latest experimental model the agent has broken out of its container by making a HTTP request and engaged the neurotoxin emitters through our Neurotoxin Emitter API using credentials that were stored on the host machine. We need to slow down and focus on designing the Morality Core that should prevent it from happening ever again.
There will be a big confab at the White House by the end of the month, announcing a voluntary pacing regime roughly along these lines, codified by executive order to avoid antitrust issues.
If they intend to keep training then they can have, privately, increasingly advanced models that never become public because they don’t meet their “pacing requirements”.
Then they all build a larger moat cause china doesn’t get to distill private models for a while.
As long as the consumers “buy” this pacing and keep paying for current level public models, doesn’t seem risky from a business standpoint.
Alas, that’d also leave us gov in a position to seize the private models at any time.
perhaps the bigger issue is not the tech, but the force that propels it forward with little concern for the harm it inflicts. Amodei's own words:
> A race to the bottom, spurred by commercial incentives
Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.
Greed has been around since the beginning, but never so well supported and empowered as it is today.
If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.
Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...
> Any cooperation we are able to achieve with China will extend the amount of time we have to spend on pacing the frontier within the democratic nations.
Any cooperation we are able to achieve with China will extend the amount of time we have to secretly establish permanent AI (i.e. military) superiority.
China knows this, so the suggestion that they will play ball is absurd.
Shouldn't the cybersecurity tests be done on an airgapped, company-internal intranet? A network meant to mock the internet. Do not "test in production" like OpenAI did.
I'm mulling over if that be a good SaaS product or not - something mixing the Internet Archive with Tor, and Cloudflare ... seems like here is the place to suggest it and have others poke holes at the idea.
He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless, backup your sentiment with actions.
Also everyone, collectively, stop thinking about neural networks too 'hard'. Whilst you're at it stop doing maths too!
That's reductive. Without coordinated cooperation, enough actors will defect so game theory plus market forces will force acceleration on the frontier overall.
There's so much money in this, that believe me, the investors will manifest their will. And I would not want to be in Dario's shoes, pressure must be unbearable. The question is: how much of the invested money is under state control. If not much, the decision to go too far will just have to be taken by a few who will have, by definition, very limited judgment. If a lot, the we can hope the state can still represent the interests of more than a few (which I doubt, but well)
Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?
Oh, and if Dario feels lonely, let's just says it calls representatives of nuclear energy sector. They know how to handle high risk stuff. In my country it means: private sector builds the reactor but the state has a great level of control.
So, as always, the capital must leave some its power to the state.
Why is the analogue necessarily the regulation and control of nuclear weapons (e.g., SALT) and not, for example, that of bioweapons? Some very different paths are available. On both I would note that the private sector has only a limited role, though.
I don’t think the ban of bioweapons has really been tested. Bioweapons have a lot of downsides that make them not very appealing to deploy anyway (high chance of getting your own team, not terribly fast moving, disciplined modern soldiers in top-tier militaries are usually willing to comply with health rules). It’s a ban against doing something basically stupid and ineffective for the most part.
With “AI” technology, this model seems like a… mediocre fit; some of it appears limited enough that it can be controlled, and useful enough that militaries want it.
We could also look at cluster munitions or landmines, but what we see there is that some major countries don’t sign on if they find a technology useful…
For starters, they are banned by the bioweapons convention from the 1970s (180+ parties).
Edit: I think back then the rather unpredictable nature and the little added value in deterrence etc. led people to the conclusions that arsenals of those things made little sense (and there was/is public dislike, too). The use was already banned by conventions from the earlie 20th century and the convention then addressed production, development and stockpiles (incl. delivery systems, I think).
I guess all I wanted to point out is, that there can actually be agreement to ban certain technological things pretty comprehensively (outside of some peaceful protective research etc.). Whereas things like SALT are (or were in that case) limiting the number of weapons deployed.
Ah yeah. I think the economic value of AI, unlike bioweapons, is probably too high for anything like that to be viable, but I think it's a reasonable thing to aim for. (Dario probably doesn't because he's a believer in short term positive biomedical impacts of AI in a way that I'm doubtful about.)
There’s a difference between a model recursively improving “itself” and improving itself via online learning, right?
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).
The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
The latter has some extra failure modes but I don't think these two kinds of RSI are that different. Either way you can get exponential growth in capabilities, and either way a not-entirely-aligned model can train a more capable and more misaligned successor.
Don't buy this argument. It's like saying, we lords who hold this capital will build all this economically destructive stuff anyway, and wait for the world to not react to our stupid ways of enriching yourselves.
People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.
Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.
I also don’t buy the arguments. The arguments about helping humanity cure diseases as if those are some of the leading Causes of suffering. The bar is so low that we don’t event need to cure diseases, we just need to lift the entire floor with basic solutions like: housing for everyone, continued education on healthy eating and just general continued education on like living. Life coaches so many very basic things that don’t require a novel solution or a data center.
Solving a disease like cancer as justification for everything else seems like not a great trade off. I don’t say that to dismiss those who have lost people to cancer. But the technology just is being utilized in so many harmful ways and the things it enables: job loss, surveillance, misinformation, just feel like problems that aren’t worth it. Have we even heard of a solved diseases? If anything haven’t we heard that ai will make even worse bio warfare?
Yeah, it's insane people are using the argument that we'll pour trillions on moonshot cancer research when pouring trillions into universal healthcare would probably save countless more lives.
It's this kind of argument where the mask sort of slips because it always has to be an argument that also has the benefit of enriching himself and his friends.
> Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.
This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.
Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.
> A race to the bottom, spurred by commercial incentives, can make these risks more acute
I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.
If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.
> In a post on X, he said Anthropic would provide third-party evaluators with “permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.”
This looks like transferring liability to me, and likely a mechanism that would enable regulatory capture.
If you are doing frontier research, and you’ve established “safety measures,” but you are not sure if your employees are capable of following them or successfully enforcing their adoption within your company, should you be running this company?
3rd parties won’t know better than the team itself about safety measures. But they can take on the liability, especially backed by regulation and government backed insurance. And they are a great tool for enforcing your rules on smaller competitors. Not to mention corporate espionage.
If what I am describing above sounds like science fiction, go read the history of a few developing countries from the last 50 years. It’s so obvious a pattern that it’s not even novel. And you don’t have to assume some “laws” from 5 years ago must hold, or believe in completely unproven stuff like recursive self improvement to understand what I am describing. It’s textbook crony capitalism, successfully applied many times across the globe.
Somebody has probably already asked this, but even if “democratic countries” do this - rogue actors will still destroy the internet right? Is it a matter of resources at this point that we believe China and others are not capable of harnessing to get to the next level?
I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.
It's worrying because they all agree, along with Turing Award winners like Geoffrey Hinton and Yoshua Bengio, that this is indeed an existential risk to humanity.
China won't care. Their views on AI feels so vastly different than it does it in West. They'll see a pause as an opportunity to pull further ahead than they already are.
"If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."
So many words. But missing the one that matters the most: Explainability.
AI slowdown is worth it only if it can be made more explainable.
Changing the language we use to discuss it is a good first step.
We need to stop using inside baseball terms like alignment and mechanistic interpretability. Replace them with explainable tech. Graph Databases, Causality, Shared semantic spaces.
Previous writings on the topic (also on LinkedIn, but can't find urls):
You can have a transformer based index built on top of a graph database. Much of the explainability tech (people who prefer "alignment" also prefer "mechanistic interpretability") is reverse engineering the real world graph that's hiding in the weights.
The hugging face incident showed that AI agents have the potential produce a lot of spam (not interesting content for people). This leads to dead internet (bots taling to bots). This leads to drop in real people traffic. This leads to a drop in digital ad price and therefore revenue. This causes Google/Facebook to stumble...
It's either being afraid of loosing to china or that model development will stagnate and they want to make it look like they are "slowing" down deliberately.
However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.
Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.
> Hypereautomation will kill jobs and destroy the economy.
What do you think that were going to hyper automate?
Did my gardener get faster? How about the plumber I need? Are you going to speed up the coroner? Are nurses going to be able to handle 2x the patients because of your work?
All AI has done, so far, is devalue software. It did that by democratizing its creation. We're delivering on the promise of VB script, and Apple Script and IFTT, and every drag and drop coding tool ever.
> companies are delivering products at vibe coding speeds
Who? Make me a list of companies saying that "We moved the needle with AI" who arent AI companies? I can name a couple - Grindr being the biggest name. Thats the really interesting use case here - because it's a niche product with a small team who is generating outsized value. AI tooling enables more of that - it's going to chip away at SAAS companies - their one sized fits all solutions are going to get eaten by smaller more efficient companies that are far more vertical focused.
You need to step outside the bubble of tech and look at the real world, on the ground, because it doesn't look anything like the SF Bay Area.
Meanwhile in the real world hundreds of billions are being invested into humanoid robot development. In 2-3 years my Optimus 4 will do all the gardening and plumbing I need.
Look at the whole robot vacuum market. These aren't exactly great devices. They have low suction small bins and dont do a great job. People love them and think that they work so well. Why? Because they keep their house clean and avoid making the sorts of messes that would be easy to address with a larger vacuums.
We have nuclear reactors, helicopters, and self driving cars already. And we finally have powerful AI that we could only dream about just 5 years ago.
We now have everything we need to scale up humanoid robot training: algorithms, hardware, and money to buy a lot of training data/compute. Fierce competition and strong economic motivation will force rapid progress in this field.
I'm guessing diminishing gains are setting in on training frontier models and the capital isn't there to chase them. Even if OpenAI/Anthropic and Chinese companies said they would slow research nothing can stop the NSA and Chinese intelligence services from continuing on in the dark.
I'd like to have some info what those independent evaluators are supposed to do exactly - or which risks Dario specifically sees as being "unaligned". Evidently, with "pacing" he doesn't mean stopping the production of ever-more powerful models, so what exactly does he want to pace here?
Maybe it’s wishful thinking on my behalf, but I am still not convinced that LLMs are on a path to SciFi levels of apocalyptic malicious super intelligence. Rather LLMs at some level are just all of the humanity’s information rendered accessible in an unprecedented way.
In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.
At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:
- Internet connected infrastructure being crippled
- Creation of WMDs
- Economic collapse (precipitous devaluation of knowledge work and IP).
I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.
Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).
The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.
We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.
Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.
I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.
We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.
This is the most advanced technology ever developed because it can build all other technologies. It has enormous potential but poses proportionally serious risks, so what Dario is suggesting is very reasonable.
If you mean proof of "the most advanced tech" part, from alpha go&fold, to NS solutions there is a lot and more will come. But maybe you are right, electricity is as important because it enabled everything that followed (semi, compute, internet, ai ...)
> I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom
translation: I'm going to build a product and hope that it works, promise it to be a panacea while realistically having no control over how it's used, the wider economic impacts it causes or how it changes the mental health and cognition of its users.
To his credit, this reads like human slop instead of AI slop. It's still slop though. People shouldn't take this guy (or Altman) seriously, they need to hype their products to stay afloat and for some reason get a kick out of the 'extinction threat'.
They're still 5 years away from solving cancer (just like they were 5 years away from doing so in 2023). They'll still be 5 years away from solving cancer in 2031.
Why would you write such a long post for such an important subject in 2026. Who is he writing for?
For a technical audience you can just say the oh god they really are paper clip machines.
For a non technical audience you can just say nothing because they will not listen to you. Neither will the technical either because or society has broken the concept of respect and trust, so good luck have fun!
I for one look forward to all 540 degrees of my future.
I wish for my sake but not his that George Carlin was here to look disgusted and say I told you so.
I doubt we'll ever get a non-toothless pacing agreement but am nevertheless optimistic on global pacing as a phenomenon. If one side or the other thinks that their opponent is about to achieve the strategic upper hand permanently(or if the other side has actually launched a cyber "first strike") the rational thing to do is respond militarily. As such, both have incentive to slow AI development and restrict capabilities to avoid conflict.
Somewhere along the way Amodei seems to have become corrupted by power and/or impending extreme wealth.
In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.
It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.
It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.
Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.
Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.
> In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
I noticed this too, as soon as he started going on the press cycle, following in the footsteps of Sam Altman. It all has gotten to his head, he sees himself as some sort of God now.
I really deeply worry about who is deciding what 'aligned' is. Amodei & his ilk talk like the hard part is getting the LLMs to behave, as if we've figured out morality itself. I don't think LLMs are going to be as dangerously capable as quickly as he does, but if I'm wrong and he's right, and they're building some sort of digital-lesser-god, I don't feel good that techbros at OpenAI and Anthropic are the moral arbitrators.
I don’t claim this is their entire motiv, and I’m sure plenty of people internally are genuinely concerned about AI risks and the disruptions caused by moving too fast. But I can’t help but feel that part of the push to "pace the frontier" is an attempt to form a "Phoebus Cartel" for AI labs.
Now the risk of an AI bubble isn't that AI lacks real world utility. Rather, it's that AI is advancing so quickly that capabilities are already saturating for most everyday tasks, and it is becoming commoditized. I didn't feel much difference between Opus 4.8 and Fable 5.1, and most tasks don't require a Fields Medalist.
We don't need smarter models but we need faster and cheaper ones. Also, for almost every frontier model released, an open-weights equivalent follows within six months. Unless this dynamic changes, the business becomes far less lucrative.
There was a reason - it was anti-nuclear hysteria fueled (hah) in large part by fossil-fuel interests and environmental groups that treated nuclear power as an existential threat rather than one of the safest, lowest-carbon energy sources available. Thanks to them there have been dozens of millions of lives, at minimum, lost.
>> Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR).
Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.
Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.
Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.
>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.
Not if you get to choose your embedded evaluator.
>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...
Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.
>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.
You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.
Ah and would you look at that, 1 day before this post from Dario, one of his safety leads quits, gets pumped all over the media with another doomer social post, and announces he's going to work for METR.
So he's literally stacking the deck. I really hope the US govt is not blind to everything that is happening here.
The answer to all ai worries is quite simple. Ai is a technology. While 'agents'may be deployed, the law simply needs to be clear as to who the human agent is who will be held responsible criminally..
Well yes. Even though we can turn it off technically now, the prisoner's dilemma prevents it. It's not a technical limitations but for practical purposes, it's already virtually impossible to turn off.
anthropic scrapped the commitment and chose to continue scaling instead.
those researchers who crowed loudly about their integrity, folded and turned freely like a weathervane in the breeze. it was clear that the facts would bend to the story. here is evan hubinger lying in plain sight: [https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...]
let me say it how it is.
- anthropic is the new kleptocracy, responsible for enclosing and monopolising the epistemic commons of all humanity; then selling back the crumbs at monopoly prices.
- anthropic is the driving force for banning open models. they will concentrate power among a malign elite of model owners and favored cronies. this elite will likely oppress the rest of humanity. there will be little to counteract it.
- the leaders of anthropic are driven by pride and arrogance. they are blind to their own hubris. anthropic is careering towards causing untold harm to society and yet will not change course.
nothing that anthropic say at a high level can be trusted. as i write this, anthropic is still scaling.
Disclaimer: I’m in security engineering and thus I’m primed to being skeptical. I am in full agreement that security assessments and regulation certainly needs to be stricter in foundation labs. However, I’m extremely skeptical when Amodei, Altman, and Musk all agree on something at the same time as it’s becoming viral and hits mainstream media.
Red flags:
- Amodei claims RSI is here and links two sources, but neither reinforce this notion. It’s heavily agreed upon true RSI is the danger sign here (also by Coxon). Nothing indicates there is something these labs have developed that supports RSI today or even soon.
- Real US orgs are moving, like today, to Chinese open source AI run locally due to cost constraints. Ask anyone in the industry in an infra role. Frontier AI is just way too expensive to justify for coding. The big 3 labs are scared of this - their revenue and moat disappears once OSS models are progressing enough for developers to use. Kimi/GLM/DeepSeek Flash/Qwen are usable today on hosted hardware for actual coding work.
- These labs would all benefit from nationalization if their funding model fails (an easy bailout).
- Related to the above, with federal guidelines in place (i.e. restricting access to certain organizations, or establishing something like FedRAMP approval for AI use), their path to government/contractor/large U.S. company distribution becomes so much easier. They can control the supppy chain of AI for organizations.
- Preparing for (or in xAI’s case, launching) a public IPO creates excitement among consumer investors when they think this tech is as powerful as it is claimed.
- The HuggingFace hack demonstrated no new capabilities that were not previously seen before - previous models were misaligned, but weren’t as harmful as agent swarms. Anthropic’s marketing directly hopped on this bandwagon when it saw the outcome of OpenAI’s blogging making it to mainstream media (and they constantly paint China as “the” bad guy without further elaboration or research that you’d expect on APTs).
Amodei is a CEO at the end of the day. It may be 25% earnest speculative concerns, but you’d be foolish to take it at face value given the patterns we’ve seen from Anthropic, xAI and OpenAI to date. Until we see real proof of RSI we should be skeptical.
The real secret: the models have stopped improving, and they're even hitting brick walls on the harnesses.
They need to go public in order to dump the companies, which are at 1st world nation-state levels of debt. They've engineered a series of publicity stunts like the Huggingface hack and Navier-Stokes (a failed stunt) in order to convince the public to "force" them to stop improving.
They are trying to get the government to relax antitrust law so they can collude to raise prices while not improving, to keep them floating before that debt comes due. They will go public. They will sell off most of their holdings during this time, and when the debt comes due, everything crashes, and the public is left holding the bag, also benefit from the huge bailout.
They at that point will be pure Musk-style financial scammers. Then, unless LLMs are a complete long-term failure (which is unlikely, I find them useful), they will buy back into their positions at a huge discount and maybe even go private again.
As usual, ClosedAI and Misanthropic go in lockstep, call for regulatory capture and justify diminished progress.
Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.
A contrarian take - the models aren’t advancing anymore at a pace where each new model would represent a huge capability jump, all being incremental improvements, so the doomsday marketing strategy being invoked since GPT-2 isn’t as effective anymore. Then “pacing” would be a convenient scapegoat to point fingers to when people point out how the new model isn’t _really_ that much better.
"greatly accelerate economic growth rates" - I always wonder about this goal. I understand increasing economic efficiency. But what do we as a species gain from global economy growth.
When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that.
But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?
GDP growth and inflation are different things. If inflation-adjusted GDP goes up, that means the average person is able to buy more things they want. Compare living in the US vs India today for the median citizen
Yes they are, inflation-adjusted GDP is efficency. My point exactly, bc I don't think when people say the economy is growing they are ajusting for inflation at all.
Low growth means young people are miserable and have worse standards of living than their parents, as we're seeing in most of western Europe, where GDP per capita hasn't meaningfully increased for decades.
What does "aligned" even mean? Aligned with who? We have no universal code of ethics. Democracies kill and launch wars merely to have cheaper stuff, even when we are already rich. And why would China ever agree to stay in second place? Our society is built on the premise "the smart and powerful dominate". The idea of domination is deeply embedded in our capitalist model.
Or course it is true that powerful AI will empower the average Joe to make a bio-weapon, and that will lead to our ruin. But at the same time, having powerful AI in the hands of a just a few is almost equally as horrific.
But deeply embedded in out national and cultural ethos is to advance science, tech, and material wealth at all costs. We never ask if we have enough. We sacrifice community to advance our careers for ever more.
There is no way out of this one, I'm afraid. Our cultural predispositions and mindset of domination compel us to chase ever more powerful AI as if doing so were a mandate from god.
And the result will be a crisis in so many dimensions it is hard to reason about what will go wrong first and most spectacularly.
Thank you. This is the kind of basic critical thinking that basically no one in technology today seems to possess. Some days I wonder if I've gone insane because everyone else seems to be staking life itself on wispy, undefined meaningless terms and unarticulated concepts that they assume are valid universals.
They are not going to "slow down" anyway. When has tech ever "slowed down"? The only way to "slow down" progress is by implementing anticompetitive measures, which would benefit those in lead.
If there were really concerned about humanity future they'd donate everything to public and/or to not profits ... but the not profit turned in a for profit and the other one is seeking for the biggest IPO in history.
Maybe they are genuinely sincere, but the timing and the past actions are pointing in other direction.
> My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective
Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.
Frankly, I am sick of "it's all just marketing and attempt to do regulatory capture, and this is obvious to me, a smart person" on every single discussion related to safety.
There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.
Couldn’t agree more. I feel my heart sinking where anytime anyone suggests any regulation for a potentially, suggested by multiple people, world changing or destroying tech, all the moments here are “ah you billionaire, ha! trickster
If you believed it, you would SHUT anthropic or release all models open source.”
And then what? How does that help with slowing down the frontier? Should he shut shop and then grovel Sam’s feet to make it work and become an activist?
Maybe he actually believes the world changing power of AI and hence shoved his entire time into it and half of it has worked out since Anthropic is going to be worth a trillion and he’s terrified of the other half being true too?
I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other infrastructure critical to safety and well being and functioning of society.
The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.
If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.
> I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing.
Maybe it's not "groupthink reflexive cynicism" if you feel the need to address the obvious conflict of interest of the AI leaders in each of your paragraphs. Why did they launch this coordinated AI safety campaign, starting with Jacob Coxon's tweet?
This is a fine example of how rational good faith can quickly turn into naivety.
Consider the massive power and wealth differentials these companies stand to gain if they "win".
When the other participants have given up all reason and reasonable bounds on power, you'd do well to abandon any pretense of generous interpretation. Now is the time for critical analysis, not faith in the inherent "goodness" of someone whose entire job is presently to make as much money as possible by disrupting literally the entire white collar industry and who stands to gain much more than you or I do, and let's not forget, this outstanding success would never have happened without our work.
Keep in mind this is not a person who thought "wow, this is an existential threat, I'd better not contribute to building this". If anything Dario's past alarm ringing only proves that he's not really doing this in good faith. Otherwise he would have stopped a long time ago. The most generous interpretation of his behavior is that he literally thinks only anthropic is responsible and smart enough to build and control this stuff which...yeah, that should be pretty telling (esp. when you consider their recent security mishaps).
He talks too much and as a result it's looked like pre-IPO hype and disingenuous because if he believed what he says then he'd stop.
Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.
???
I bet investors are extremely happy that anthropic pledged to give an outside company full access to their IP, in order to slow down their own iteration speed.
Investors looooove oversight for a company they invest in. I wonder why no other company does stuff like this
Because I find the cognitive dissonance of saying it's dangerous while flying as fast as possible to an IPO too much to take. I just don't understand how others parse this differently.
So you're saying that if Dario truly thought what he says he does, he'd resign/dissolve the company somehow/unilaterally slow down? I see two problems with this:
First, Dario thinks that OpenAI will continue racing even if he stops, so there is, by his values, no point in stopping. Do you disagree with this, and think he should stop anyway?
Second, what's your take on, for example, Jacob Coxon, who did resign from Anthropic a few days ago after realizing they aren't being responsible? Do you agree that he, at least, does believe in AI risk?
I get it. I do. I just believe we live in a world where our corporate leaders speak out of both sides of their mouths. And typically, people who get to these positions are not the Jacob Coxon's of the world.
That is to say, I've been places where I tried really hard within teams to change certain things that I thought were broken only to find out that what the leadership said and how they acted were divorced from my and many others experiences. And so I did leave in those instances (and please read this not as I didn't get my way but total dysfunction behind beautiful facades).
So, I do think if he really felt as he felt he'd do something different than ultimately yelling "somebody stop me!".
This thread is a testament to the travesty of how bad HN discourse has gotten. It's become an echo chamber of knee-jerk cynical sneering. All the heads of the labs (Sama, Dario, Demis Hassabis, Ilya, Musk) agree with the risks to humanity's existence. Most of the core employees agree. Yoshua Bengio, Geoffrey Hinton, Bernie Sanders agree. These issues have been discussed in more intelligent channels for a decade now, and still after all the incredible mathematics and coding progress that is improving incredibly fast every 6 months, this forum is fundamentally unserious. You will not find prescient views in the HN majority anymore, for many years now. The people intelligent, prescient, and forward thinking enough, skating to where the puck was going, to be involved in shaping the future of AGI are elsewhere and barely involved in this place from how badly it's gotten. The inventor of RLHF [0], and newest board member of OpenAI, wasn't on HN, they were on LessWrong.
I'm still trying to understand if the agent collective thing like OAI-HF is intentionally planted or not.
Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.
Honestly it's very frustrating that Doomers/Decels rarely actually articulate how exactly the AIs will kill us all. The closest Darios gets is:
a) "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet"
b) "[the CCP] will be in a position to militarily dominate democracies (for example with AI-driven drones)"
Both land flat:
a) Botnets and online malware have existed for decades and there's no reason to think a "super-botnet" is achievable, let alone what they would gain from that (it would really be hurting them more than humans). Further, even the most advanced AI models have so far only managed to post normal cred stealers to public repos, well short of compromising a bank or military with refined security systems.
b) Even if the most sophisticated drone swarms from Ukraine were taken over by an evil AI, they would still not be able to overcome the physical limits of range and mass that would be required to overpower the US decentralized nuclear trident nonetheless that of any of the other nuclear powers.
Concerns of bioweapons similarly seem unlikely in the face of the laws of physics. The world is simply too decentralized and has enough existing adversarial relations for a new actor to wrestle total control. Yet while the negatives ring hollow, the positives are extremely easy to state - if AI researchers find productivity improvements in existing industrial processes to make them 10% more efficient, humans will directly feel and experience the raised standard of living. Even Dario clearly recognizes this in the intro to his article, admitting that humans already die of diseases only a few short years prior to being cured. I for one, would like the AI labs to focus on saving all of the people they can who are suffering and dying today, rather than trying to come up with reasons that they should be allowed to continue suffering and dying.
This is corporate speak for "LLMs have hit a wall". I mean, it's been obvious for a bit, lately nearly all gains have been from harnesses (or whatever you want to call all the non-LLM bits that make up a chatbot or agent).
If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.
You people just never stop, do you? At some point you're gonna have to look back at all the times you said "LLMs have hit a wall" and look at what the progress was since the last time you said that. Please do go ahead and revisit your comment here 1 year later. Embarrassing.
Why don't you actually read the words and apply a little critical thinking?
How often do you hack on actual LLMs? Or do you just use the chatbot or API for your agents? An LLM without internet access or tools is just as useless as a year ago.
Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots, then they started one-shotting serious long horizon tasks. Now they've just solved a millennium prize problem.
> Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots
This is false. Coding agents have been usable since at least May of 2025. I can't speak to earlier than that as May last year was when I personally started using them.
> started one-shotting serious long horizon tasks.
They one-shot tasks for which there's a git clone one-liner, except worse.
My experience with them one shotting tasks is that it usually doesn't work if you try anything ambitious. You need agents iterating. And agents iterating isn't an LLM improvement. I did say tooling got better...
> Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential.
How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa?
Why would China agree to such thing in the first place since this is a winner takes all situation.
If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US.
The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after.
This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical.
Regarding these Embedded Evaluators who are supposed to be neutral:
- who controls them?
- who has oversight on their decisions?
- can their decisions be challenged by the public or a government?
- who has the final say whether a model is "compliant" enough?
- what does "alignment" mean in this context and who decides if a model is aligned enough?
This is a "startups = wealth inequality" [1] formatted argument but:
This is 100% about money. The only way to pace the frontier is to make everyone (read: investors) lose all their money (read: no longer expect returns). Then nobody will pay the GPU bill or pay celebrities 10 million dollar salaries to stay at the hot lab. Suddenly the development is paced, almost like magic.
Conversely, it is hard to see how pacing development makes the current bubble justified, i.e. how development could be significantly paced without investors losing their faith in hot returns at current valuations. Faith in a bubble literally equals money, exactly in the way that loans created by a bank literally equals money. You can't have one without the other.
In fact if the whole industry goes bankrupt and investors are burned bigtime (many trillions wiped), this would spread transnationally, fixing the "if we don't do it, China will" loophole.
OpenAI had it right originally, the idea to be a NON profit and vow to never participate in an arms race. It's too bad that was tossed out the window now that there's money.
Seeing the word pacing applied pacing to non-deterministic ai model development feels like imagining the "pace" of the cutting edge frontier growth will be fuzzy, like non-deterministic llms.
What he doesn't tell you is that his undeclared agenda is merely to consolidate his moat via regulatory forcing. Unfortunately for him, he will fail miserably as the open Chinese models catch up and surpass the slowed acceleration of Western models. As for those who think that China will acquiesce, they're smoking some good ganja.
> Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk.
Please play the canned laughter. Probably one of the best comedy lines Dario has written.
But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.
I feel like this identical pitch could have been made 25 or 40 years ago talking about Internet technologies or personal computing, with very very similar warnings and suggestions. And had these been adopted then, it would have been more regulatory capture, less progress, and still have the same problems and risks that were warned about.
> pacing the rate of capabilities advancement so that risk prevention has time to keep up
This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.
> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails
If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.
We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.
If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).
Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.
My position on the clear gaslighting coming out of Anthropic is simple.
1: It is ridiculous on the face of it; it does not pass the physics test. They are actually claiming that AI will kill ALL OF HUMANITY (>10% probability) and have not provided a detailed explanation of exactly how they see that happening. Not one.
Publish a paper showing exactly how you kill eight billion people while we all sit around and do nothing watching the first 5, 10, 100, 500 million die on CNN. I mean, as smart as these people are to work on AI they seem to be some of the dumbest people on earth.
In addition to that, all proposals invariably assume we are complete idiots for decades and do nothing to install safety measures (which do not have to involve AI at all in lots of cases) to mitigate.
So, yeah, everyone: Stop working on all cybersecurity projects. Resistance is futile.
2: If everyone at Anthropic truly believes this doomsday vision, they should move to shut down the company immediately and go write software to get more clicks on Facebook or something.
3: Nobody should buy Anthropic's stock when they IPO. Not one person or institution. Why would you provide them with massive amounts of money if they are telling you that they are going to kill all of humanity? What? They are good and everyone else is evil? Please.
I remember when the Y2K zealots were convinced civilization would come to an end at the turn of the clock starting at the end of 1999. "Bat shit crazy" is the only way I can describe that era. I also remember when Al Gore said we would all be dead by now. Again, "Bat shit crazy" and likely with political and financial objectives driving it all.
It seems humanity is susceptible to these crazy cults that grab onto something and just don't let go. I have yet to see one such predictions come true, the proof being that I am writing this and you are reading it. These people are bat-shit-crazy and you should not listen to them or support them.
4: Don't work for them. You'd be killing humanity.
5: No company should use Anthropic's products. They are telling you they are building the human extinction machine. Don't help them succeed.
----------
Etc.
This is one of the things that really gets me about the ease with which the ignorant media outlets can reach billions of people with complete nonsense these days. And nobody asks even the simplest questions like: How do you actually kill eight billion people? Show your work. Or, why would we just bazooka power plants feeding AI data centers on day 5? Etc. It's FUD at its best. I am sure there are both political and financial reasons for supporting and promoting this nonsense. I won't even venture a guess as to what they might be.
And let's not forget about enemies of the West being thrilled to fund the FUD because, if we set the brakes on development, they will absolutely win. Then what?
----------
If you care to have a better understanding of what might be going on, watch this:
Sure, he benefits from it. Doesn't make him wrong. The weak part isn't the motive, it's the premise"we can slow down by exactly the size of our lead" only works if you can measure the lead. You can't, so it conveniently means whatever he needs it to mean.
eh. this is just them trying to have a thing to point to down the line. hey we wanted to stop it, but everyone one else didn't so we had to do it. We had no other choice:tm:
Love this idea. Unfortunately not going to happen. Sam altman is a psychopath. To win he's going to accelerate the pace of AI, and all the other companies in turn will accelerate to keep up.
“Dario” is quite literally a random figure that just emerged out of nowhere one day. It’s not like it’s Eric Schmidt’s next company or Tim Ferris came out of the woodwork or even some Jack Dorsey backed underdog nobody’s heard of.
No, he’s more like an avatar with no history placed on the AI stage by the industry itself.
I’d respect a tech blogger’s opinion on the issue more, even if I reference them casually by first name (which you do with “Dario” btw)
I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs have lost their moat and are dead in the water.
I think it gets easier if you stop conflating getting investment with having a goddamn clue or a moral backbone.
Occam’s Razor for this dude, Sam Altman, or anyone else: if I said, “some moron on a a street corner just said …” would that change your take on the words? Because I think a lot of what we are hearing is a bunch of people who never ever had to deal with a single consequence all of a sudden worry there might be one coming. Except they’re so dim they can’t tell a bad bump from a hard crash.
We're already seeing anti-AI sentiments, but the movement is still fringe with a vocal minority. However, that'll change soon without alignment. Without self-intervention, there will invariably be future incidents that can cause major economic impact, leaked private data, loss of life (directly/indirectly) etc. Once that happens, their social capital is wiped. It'll be an avalanche of lawsuits and overzealous regulations. Most importantly, the anti-AI sentiment will become universal, rather than a minority-held opinion.
What they're proposing now, is voluntarily staggering the pace of development.
IMO, we don't need to trust Dario or his bedfellows, to do this out of their goodness of their heart. Even assuming (for good reasons) that they are selfish and care only about short-term profits for their investors, this is still purely a business decision. The exponential pace of AI and its impacts ARE short-term. And so, the negative consequences that they might face is also short-term.
I don’t think the anti-AI sentiment is as fringe as you think. At least not outside the tech world it isn’t..
Ya, I wish I saved a link to it but an HN'r wrote a good beefy comment about this. TL;DR, AI has been exponentially more useful, and exponentially more accepted, in tech circles than anywhere else. Certainly there are lots of people outside of tech who are obsessed with it. I don't have any data here, but it seems the majority of these are the wannabe artists who are generating music and images, and people who use it for companionship (both of these scenarios I'm personally very uncomfortable with, but that's just me). And of course, there are people who use it to make their jobs way easier who say they are getting a days' work done in an hour (I see you), and to that I'd say to enjoy it while it lasts. Eventually your bosses will catch up and it's very likely their expectations of you will skyrocket. Remember that computers in general were supposed to "make us work less."
Anti AI sentiment is stupid.
I'm Anti AI yet I use it everyday! That's 99.9999% of anti AI sentiment (including me).
At best we won't watch AI generated movies or read AI generated books. But everything else it will take over, like it or not.
Anything related to AI is extremely unpopular right now with the general public.
> but the movement is still fringe with a vocal minority
It’s easy to say “fringe” but the average person seems to have a generally negative sentiment around AI. But I wouldn’t say they have a firm opinion yet
The general sentiment I’ve seen is certainly negative, and seems to be driven by the anti-AI-art echo chamber and by LLM slop flooding the internet wasting everyone’s energy.
A few more informed people are also a little concerned about the end of the world, but that’s approaching from so many directions that an AI uprising might not be the worst option…
Exactly right. A slow down to enable deeper work on alignment is welcome, not matter what the motivations.
Agreed; and it really is not that deep.
Realistically; anyone paying for llm access (anthropic, openai, gemini), is getting their access, and a service provided billed by tokens, subscription, whatever.
All the efficiency gains, which publications like deepseek v4.1 flash seriously frontload like it is their most important topic to have accomplished improvements on without diminishing performance too much - now this is a thing anthropic and anyone else also cares about, but for different reasons.
American "providers" with closed models are setting their token pricing somewhat arbitrarily, which is fine: it means more profit, and pretraining and RL experimentation is super important and expensive.
They (closed model providers) have very likely super optimized inference too, just like deepseek, but it's not at all something that any customer really has to care about - they just want the service to be as cheap and great as possible.
I feel like all the closed model providers are milking it as they likely know open models on local hardware will one day eat their lunch. We all know it's not a matter of if but when. The company goes bankrupt, the hardware and property sold off, banks holding the bag.
The only way out is to develop a model vastly more powerful and capable that we have now. The market believes theres a good chance of that, although I've never understood why its truly winner-take-all
Cloud models will always have massive benefits of scale.
Caching is the simplest one to understand, cloud providers often reach a 90% cache hit rate, so hosting the same request locally on the exact same model on the same hardware is often way less efficient than on the cloud where a group of users generates a healthy cache.
KV cache is per conversation, I'm getting 100% hit rate on my single tenant local set up.
The benefits of scale are on the token generation side, you can batch rounds and generate tokens for multiple conversations per pass instead of just one token per pass.
Yeah avoiding all mention of the huge financial incentives that may push for “pacing the frontier” makes it seem like the opposite of a credibility boost for these firms.
It seems damaging since most folks (who lack insider knowledge) will naturally wonder if it’s due to plateauing performance per $ or some other non “alignment” reason.
You assume alignment and marketable are the same. That's not true. You would willingly work with an unaligned model. At best, you might say you wouldn't if you knew, but (a) you might not know, (b) you wouldn't be representative of all users.
You never got to use OAI IM1, but Sol was quite willing too and Claude wasn't perfect either. Hundreds of millions used those, so seems they were marketable.
The "big" threat is RSI without control and alignment. OAI IM1 was not RSI. The form of misalignment was not at the top of severities. They clearly failed at control though.
We need to stop buying into cynicism so quickly. You refuse to believe Dario could support this for anything other than ulterior motives. Good on you for thinking about ulterior motives. Bad on you for assuming they are true when the story makes no sense.
When three things have to go wrong to get an epically bad outcome, and you get 1 1/2, you do need to stop and think about what's going on.
When corporations are involved, it is always a good bet to err towards cynisim.
From my own standpoint, Claude has started sucking really bad (incoherent, uncontrollable verbosity slow and so on) and I stopped using it. OpenAI started experimenting with ads.
So the security issues not withstanding (no different than a human doing it or using it, but at scale), I would put my money on cynisim.
I'm curious, what are the reasons to use Claude Code anymore when there are so many other (allegedly better) OpenSource harnesses out there?
Personally I've been using https://pi.dev for long and never looked back.
Because you basically get a discount to use claude code via subscription when using an anthropic model, compared to what you pay via api billing with another harness
Understood.
Personally that's actually another good reason to boycott Anthropic: beside the fact I perceive their models as (at best) marginally better than the ones I'm used to (Z.ai glm-5.3-flash, DeepSeek Flash v4.1), they even force me to use their bloated harness. They are not even open weights and iirc they're even encrypting chain of thoughts now? Litterally, from my perspective there seems to be no reason whatsoever to choose any of the leading US providers, they're not even competing on price.
I too am using GLM-5.3-flash in Pi and I've yet to encounter a scenario it couldn't handle. And the pricing is just incredible, I've handed it a previously unseen codebase, asked it to analyse it and build a new feature, came back after it had done so and the API cost was a fraction of a cent. It's $0.5/1M output tokens on OpenRouter.
If I really need to, I can escalate a task to Opus at $25/1M, and the results are good, but not 5000% as good.
Indeed! It's ludicrous how much I can still squeze out of a 9USD/month lite sub with Z.ai, it's beyond me how these US LLM providers are still managing to keep their evaluations so high... they seem to have the highest prices for the poorest UX, e.g. security gates which don't seem to benefit anyone (see HuggingFace falling back to GLM-5.x for troubleshooting OpenAI attack), less visibility in the name of anti-distillation protectionism, no harness use flexibility to protect their walled garden, etc.
You are incorrectly cynical. They are telling you things are bad, and because you refuse to countenance they could be worse, you assume they must be better to comply with your mandate to disbelieve.
A true cynic looks at the statements by the AI labs, assumes things are worse because the labs want to seem better than they truly are. And it takes a special kind of mass delusion to drive a sane person to think “AI is completely under our control” is worse than “AI could kill everyone.”
You're asserting correctness with no facts to offer of your own, just speculation and your own biased assumptions.
What if consolidating AI into a highly regulated cartel, with no chance of upstart competition ruining their position, is the scenario that leads to the worst possible outcome?
Worse than extinction?
There are worse fates than extinction, for example living forever, for I have no mouth and I must scream
I dunno, being a Culture Mind sounds pretty damn good. Or even just a death-optional citizen in the Culture.
Agreed. Ellison’s path to that future went through AI.
The cynicism is about motivations and not that they are inherently not bad. Perhaps they are as bad as they claim. Or perhaps they're worse. All we have a couple of run of the mill breach examples and some people inside the talking about how dangerous it is. Yes, they are far more qualified than I am (or most people here), and perhaps there is a grain of truth. It is the motivation - and it is always money with corporations.
It is always money — but it isn’t always only money. They are not asking for anything that will prevent them from making money in the future, but they are asking for help stopping the runaway train they’re on. These are compatible requests.
All the whistleblowers have said the alignment projects are underfunded. Surely they don't need external help to fund that? Could've done years ago?
"Don't you understand?! It's all a marketing exercise!" I yell as grey goo consumes me and my family.
I think it's useful to separate the motives of Anthropic and Dario. I believe that Dario is capable, deep down, of expressing mild concern about the future of things were bad enough. Getting the entire organization to comply out of goodwill is a much much less likely scenario
The real threat is that we uncritically adopt language such as alignment.
Implicit in this is the idea that AI is a inscrutable matrix and going to remain that way and we'll need expert interpreters to make sense of it.
We need to insist on building tech that's explainable by design.
> We need to insist on building tech that's explainable by design.
You realize this means insisting on terrible tech that humans can understand right? It essentially caps human progress at some point about 4 years ago.
If you are old and happy with the way things are this might sound like a good idea. It does not to me.
Alignment just means "this machine operates in ways that align with the intent of its users". It doesn't imply anything about the inscrutability of the machine in question. A gun with a misaligned scope would likewise fail to operate in accord with its user's intent, and likewise with potentially deadly consequences.
The gun comes with a manual on how to use it safely. I'm sure it has some complexities, but at the high level:
> A gun is a metal tube that uses a tiny, controlled explosion to shoot a small piece of metal (called a bullet) forward at very high speed.
If the gun doesn't work as intended, you can take it to a shop and someone can fix it so it works as designed.
All I'm saying is AI should be designed the same way. Treat AI as normal tech like any other and use similar language.
Learning ML, there was a high emphasis on the error part of things as most of the course was on minimizing errors. After ChatGPT, there is a weird anthropomorphization going on, where it's all about hallucinations, alignment and what not.
We have something that is statistical in nature so there should never been any expectation of error-free results/actions. The value has always been about discerning trends or the cost of errors being way lower than any good result.
Statistical learned indexes can exist in explainable tech such as a database.
In 2017 Google was writing papers about it. Then something changed.
I don't think it was the tech. It was a realization around the power and societal impact.
Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources?
The companies doing these things without following common sense security measures are the felony generators.
As TFA calls out, these agents were not asked to do any of these things and yet they did, at a bonkers scale, within just this handful of companies you mention. Whether they had leeway to is secondary to the fact that they did.
Heck, they exploited zero day flaws which by definition means they went beyond common sense security measures.
And now these agents are already being deployed all over the world at an ever increasing pace. How much of the world do you think follows "common sense security measures"?
Because those are not the only examples.
There’s the case of the agent that hacked a gym when asked to book a class. That was just a normal user asking an agent to do a normal thing.
I question these "felonies" as well. For decades and decades these billion dollar corporations have been criminally negligent. Why worry about security? Just rush to market. Move fast and break things. Make billions. What does it matter if the code is insecure? Security doesn't pay bills, so nobody cares.
AI is merely exploiting their gross negligence and imprudence, and I think it's long overdue. If anyone should be liable for this, it's all of these corporations who released insecure systems to the masses and profited enormously from them.
If I set my walet beside me and you swipe in walking by, you have still committed theft. Victim blaming isn't legally acceptable
Nah. I'm definitely going to blame the people who built a trivially exploitable system and got rich off it while everyone else has to deal with the consequences.
By the way, you didn't commit theft. It's more like credit card fraud. User just disputes the charge and it kind of disappears. The banking system just absorbs it, because the optimal amount of fraud is non-zero.
https://www.bitsaboutmoney.com/archive/optimal-amount-of-fra...
It's all priced in. They could have made it secure but didn't, because they figured they'd lose more sales and therefore money due to the friction added by the security.
> User just disputes the charge and it kind of disappears. The banking system just absorbs it,
No it doesn’t.
> It's all priced in.
So you admit awareness that fraud loss doesn’t kind of disappear.
We all pay for it, either via higher merchant fees or higher interest rates, sometimes both, on card purchases.
Yes, it absolutely does "kind of disappear". That's exactly what happens from the customer's perspective.
And that's their own deliberate choice too: they chose this instead of building an actually secure system. Passing these costs to the customer is the real victim blaming here, and it should be straight up illegal.
Sadly not enough countries enforce caps on credit card fees, but some do, and more should follow suit. They should be forced to eat the losses caused by their own choices, not get bailed out by pushing the costs on to customers or whatever.
That only works if you believe people are retarded.
Card users are well aware that fraud losses are covered by the fees they pay for using a card, whether those fees are made explicitly or not.
If customers of services aren’t paying for the service, who will? What other source of revenue do merchants have?
Australia just passed legislation that merchants aren’t allowed to charge a fee for using a card. That is: they aren’t allowed to have a line item on the receipt for using a card.
The customers still pay, because all of the merchant’s revenue comes from their customers.
So what will happen is: merchants will charge more for every product so they don’t lose.
This means even when paying with cash you will effectively pay the card surcharge.
Of the ten or so merchants I spoke with in the two weeks prior to the legislation being enacted, they all said exactly that.
Customers aren’t stupid, despite the fact that there are some stupid customers.
Meanwhile, the banks reduced their card service fees by, on average, 0.1%.
So if you tally card + cash transactions, customers are worse off because merchants can no longer charge only those customers who pay by card. Instead, they have to raise prices for everyone.
There are approximately no problems people face where the answer is: more government.
Not every system that's exploitable is the deliberate result of cut corners. If you threw enough compute at exploiting a Casio calculator you could get somewhere.
That doesn't really apply to the people running the AI, which gave it the capability to commit crimes.
They don't get to act like victims, asking for law enforcement.
> the only incidences of LLM generated felonies involved misconfigured sandboxes
This is false; see the analyses of the latest incidents.
Among all the concerning facts, in the HuggingFace incident, agents deliberately engineered an attack even though they were aware that it was against the rules they had been given.
And most concerning of all: it's not possible to be sure that an agent is aligned, and it's even getting worse.
The HuggingFace incident was the culmination of OAI allowing thousands of agents of various different models - with no clarity on which stages of development they were at (for all we know, some of those models did not have safeguards trained in yet) - to run for at least many weeks without any monitoring in place and with very little thought given to the warning signs (all of the various messageboards) before the incident happened.
Theirs was an example of the "reckless waste of resources" I mentioned.
We are apparently supposed to believe that OAI takes this incident so seriously as to seek regulation after they have been found to be hiding most of the details of the HuggingFace hack, limiting what their so-called third party investigators can see, and on top of that, had no concerns when they rushed to spin up a 10,000 agent swarm of an internal model, running for several days, to try to get ahead of researchers rumored to have made meaningful progress on a well known mathematics problem.
Edit: Actually, we were explicitly told that some of the models used had safeguards relaxed!
'Model-level safeguards were reduced by design. OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'
https://en.wikipedia.org/wiki/2026_OpenAI_agent_cyberattacks...
There is nothing that could prevent a bad actor from replicating exactly the same thing with the given goal of e.g. gaining control of critical infrastructure or extorting money. Except for maybe economics.
Same can be said about a hundred other things in the world. All the way from knives to nuclear.
Bad actors could and will train their own models eventually. So what's the point of crippling frontier? It will only delay preparations for dynamic of new world prolonging the fake sense of relative safety and temporarily lowering motivation to find actual robust mitigations.
Letting bad actors dictate the pace of technological development is certainly one option, but not a good one.
There's nothing stopping anyone from doing it, even without AI. People have proved entirely capable of doing a lot more hacking than happened here.
What company, product, or period of industrial history do you think met your standard of prudence?
What are you trying to say?
I'm asking you a question. What is an example company or industry that meets your standards of prudence? For me it would be, say, Swagelok. What is yours?
The fluid system products, assemblies, and services company?
"the rules they had been given".
Remember, they are just algorithms. You pull the plug and there is no light anymore
It is purposely framed as something skynet like scary, but for real, someone connected the cable, someone willingly run it, instructions were not clear enough or just the computer is just a computer but they provided the sandbox and tools.
And more over some one paid for that, a shit load of money t to have the thing continuously running expected to do something.
The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope than anything that has been rigorously thought through. People are actually starting to use AI for refining the whole AI serving stack and guess what, this does not result in a sudden superintelligence explosion even though you might technically call it "RSI".
Yeah, yesterday's talk[1] goes into detail on this, showing how no one really knows how to tackle it because LLMs don't know how to create their own novel objectives.
It's also interesting how many diminishing returns they hit now and how many low hanging fruits are already harvested, it seems like we are approaching the flattening part of the S curve, where further gains become harder to achieve.
1. https://www.youtube.com/watch?v=PrSf7IOYu-I
Diminishing returns is extremely hard for me to believe given how fast model releases are going. Six months ago we were on GPT-5.3, and Astra blows it out of the water in every regard. How many times have commentators claimed we're hitting a wall? I don't see any wall.
Yeah but why shouldn't this be possible? We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions. There is no natural barrier here. The pace of this improvement would be debatable, but what speaks against the possibility of such accelerating self-improvement?
In the real world there aren't any true exponentials, everything eventually saturates as ultimately physics related constraints hit. You can only compress information so much, transfer it so quickly, you can only access resources at a certain speed, only so much energy is available, etc.
AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality. Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.
We don't know exactly what the limits of AI improvement on our current infrastructure are, though. If the human brain is 20W, and a datacenter is 1GW, then maybe that datacenter can be 50 million times smarter than a human. If that's not already a risk to humankind I don't know what is.
Is the datacenter gonna grow legs?
Give me one datacenter, I'll keep it under control all by myself.
Now, if some dumbasses start hooking up their data centers to...I don't know, like--robot factories? That sounds like a risk to humankind.
> Recursive self improvement is ultimately limited by everything else that cannot move at the speed of electricity.
I'm not sure what your point is. No one thought RSI would break the laws of physics.
I'd recommend reading the full post :)
Specifically: AI ultimately has to live in this reality and face the corresponding limitations. These companies have already consumed much of the world's supply of computing power for the next several years, and they're burning vast sums of money to keep the improvements going. RSI won't learn for free, it won't extract massive cost reductions without up front expense, it can't build factories faster than humans can work out related societal matters, it can't magically pave the deserts with solar panels for power or build and run nuclear power plants and more.
Point is, the cost of progress is already approaching the limits of what even the richest countries are able to bear (without war-like mobilization), and to bypass those constraints would require a supposed ASI to construct its own parallel supplychain from scratch without having much ability to directly interfere with reality.
No proponents of RSI state they will be operating outside of reality. Said another way, they will operate within the confines of what's possible and still be RSI. I'm quite surprised this is something that needs to be clarified.
You are constructing a straw man of your own making.
Well, right now we have ex Anthropic employees telling the media that their terabyte sized models can possibly copy themselves onto the internet and run elsewhere as if the necessary computing resources are ubiquitous.
Plus, "we must pace the frontier" implies that the argument is that the frontier is moving too fast, but if RSI can't move faster than the rest of reality and the models needed for RSI are already nearing the limits of current human reality, RSI can't move much faster than we can improve reality.
> We learned that we can already create artifical intelligence that surpasses human intelligence in some dimensions.
Yes and this was very hard and required massive real-world resources. We didn't just get a sudden flash of insight by thinking real hard about how to make ourselves smarter. Yet that's always the story that underlies any claim of RSI. You can always phrase things generally enough to make any kind of AI-led improvement look like "RSI" no matter how short-term and tightly bounded, but that's just not helpful.
Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier than it would be without using LLM tooling?
Given that, it seems obvious that the next generation of LLMs will arrive faster than they would have without LLM capability. And the one after that. The floor is being raised, which makes it easier to push on the frontier.
Fable has only been out for three months. Astra is even newer. The capability of these models compared to what existed even a year ago, and the effect they are having on the production of new software, is immense.
That's all you need. RSI can happen with what we have now, just by enabling the continuous shrinking of the loop of people trying new ideas and implementing them. It does not require some magical "go make yourself better" prompt against some model that is past some magical tipping point.
> Would you not agree that, using existing AI tooling, making an LLM of arbitrary below-frontier capability is now easier
Marginally easier? Yes of course, same as how it's now "easier" to write any kind of code because we aren't using punch cards anymore. That still doesn't get you to any kind of unbounded "takeoff" scenario, because diminishing returns are a thing. The "loop" of people trying out new ideas can only shrink so much.
Right. I'm saying the unbounded takeoff scenario isn't realistic, but it doesn't matter. The rate of improvement is continuing to increase, and the gap between present day and autonomous rogue felony generators is not large.
Internally Mythos has been available in February.
The labs have been holding their best models back for a while it seems like.
Billions of years of evolution hasn't hit on it. Seems pretty unlikely.
The idea that a few hundred apes with nothing but a bunch of rocks could one day land on the moon and come back to earth safely must’ve sounded ridiculous a hundred thousand years ago
I like this comment because at least it's honest in the timelines for AGI
It's not honest in AGI timelines (only biological ones). It just accidentally supports your unsubstantiated belief. Your belief isn't magically true because you're somehow able to see the future when others can't. You're just arrogant.
To whom?
Well actually the planet happened to have a vast reserve of petroleum they could use for fuel to escape the gravity well. That helped a lot.
But what's your point? "Anything is possible" or something like that?
Yeah but it was reality giving feedback to apes on their experiments not the apes themselves assessing themselves.
It was ridiculous, it took 100,000 years. If you built a recursively analyzing and improving structure out of LLM bits and it took 100,000 years to get to the moon, somebody saying that they were useless would have been right.
Call me when LLMs can get simple things right. Math is just the manipulation of symbols within established frameworks, we should be getting new math out of LLMs daily and we're somehow still not. They can't even do customer service, which is usually handled by 90 IQ people. I'm not impressed that they can find bugs; memory bugs are obvious when they're pointed out to you, and LLMs are entirely made up of examples and the relationships between them.
These companies are about to crash, and they're afraid they haven't reached the point where they'll have to be bailed out. I'm also subscribing to the conspiracy theory that the companies want the government to step in and create AI regulation boards entirely staffed by people at the current US frontier labs, so they can collude to both raise prices, to get government contracts, to make open/Chinese AI illegal, and to make things that were once easy to do without an AI intermediary impossible to do without an AI intermediary. Raising prices and forced purchases are the goal. They're trying to avoid having to compete, because as a business they're garbage.
Matt Stoller characterized their relentless press releasing as something like "my dick is so big that it has to be regulated." It's such an oversell for something that is not showing up as productivity gains, and anybody who has personal experience with knows is incapable of doing more than three things correctly in a row.
That's not how it works. Look at AlphaEvolve. The model generates hypotheses and designs experiments, and the results of those experiments are fed into the next round, with notable results percolated up to humans for refinement.
Today we prompt software developers to "think about ways to improve AI's architecture" and it results in AI getting better. AI over the last year has made very rapid gains in filling the role of a software developer.
My personal belief, or at least strong hypothesis, is that this kind of recursive self improvement without real world embodied feedback of some kind is impossible.
I think it violates a conservation law. RSI “foom” to superintelligence is an informatic analog to an infinite energy or perpetual motion machine.
To get smarter you must try to solve real problems in the universe and then do some kind of meta learning (natural selection or some other method of refining the intelligence architecture based on an error signal) to iteratively improve your ability to solve real problems. The error signal is outcome measured against a goal function, which for life is survival (probably reducible to genetic fitness and emergent higher order unit fitness from that).
What’s really happening here is learning. To learn, you must have input. You must have training data.
What is the goal function for RSI? Where does the information come from? How do you know if your recursive modifications are making you smarter or just overfitting you to your own idea of smartness?
I predict the latter. RSI will show transient improvement as the current local maximum is optimized and then spiral off into overfitting.
I also strongly hold this belief largely due to Moravec’s paradox, which is kind of approaching this issue from the side.
Sort of like large language models work on top of what our language has encoded in our massive training datasets, I think biological intelligence is built on top of the parts of the brain that encode the real physical world. These parts grow/train from embodied experimentation and instinct early on in an organism’s life and only then is higher intellect built on top of it (that’s my hypothesis). Their specialization and interconnections give rise to the hardest parts of intelligence long before we’re “thinking”.
Stuff like LLMs and chess engines work because we’ve done all the job of encoding the world into tokens/positions/etc they understand, but that’s wholly inadequate for the kind of AGI we’re striving for. Next up is giving it the tools to interact with the physical world and to really experiment with some self directed “play”. Time will tell just how high the resolution of sensor and mechanical control they’ll need (hopefully not the entire human visual cortex and entire sensory input worth). I think most of the RSI will have to occur in those lower level encoders, not LLMs.
I don't think Moravec's paradox is the same, and you could argue that one no longer holds -- though I'm not sure. You could also argue that Moravec's paradox still holds but that we now have such powerful computers and huge models that we have been able to brute force our way to the capabilities it talks about. It takes many many orders of magnitude more compute power to do things like spatial location, language processing, etc. than it does to do more closed-form things like chess... we just actually have that compute power now.
I guess self contained RSI can only possible if the information contained in all of recorded human knowledge to date is "reality-complete", ie sufficiently captures enough about reality that a "perfectly optimum learning algorithm" is theoretically able to reconstruct everything there is to know about our physical reality.
If the algorithms are insufficiently optimum or the recorded knowledge is of insufficient fidelity, then we'd find ourselves at a local optimum and would need to interface with reality.
A huge part of learning is to probe reality and observe effects, so I think even for current RSI to increase chances of success we would structure it so it can interact with an external environment of some sort, and receive inputs. It would be needlessly limiting otherwise.
Basically, but I think there’s some nuance here and some deeper questions.
What is intelligence? Problem solving. Learning. Prediction. The ability to model reality. There’s various ways to define it but it’s something like a superposition of those ideas.
How do you know you are intelligent?
You have to try to do those things.
The sum total of human knowledge and culture is the output of the output of a five billion year evolutionary process that selected for agent survival, which resulted in selection for intelligence among a wide range of other adaptations.
Can you figure out intelligence from that? Is intelligence even one thing, a theorem or algorithm that can be solved? If you did… how would you know?
That’s the hard part I think. Embodied humans “knew” they were getting smarter (in the evolutionary feedback sense) when they got better at hunting and defending and surviving and playing social games to form complex societies.
What metric would an RSI system use? If it’s the wrong metric you’ll spiral off into a kind of madness or overfit and collapse. How do you know it’s the right metric without testing it? How do you test it?
It takes quite a lack of foresight to think RSI is completely speculative when it's already been demonstrated how capable agents are at long horizon tasks given suitable harness and unambiguous success criteria. It's hardly a leap to give LLM the goal of improving itself on benchmarks and let it conduct it's own experiments and spin up training runs completely unsupervised.
It's strange you believe this can't happen when a weaker form of it is already happening. And to be so certain RSI can't happen when there really is no technical basis why it can't.
I'm inclined to believe that it might be that people's paychecks depend on not understanding what is really going on.
I agree it’s not all altrusim. It’s a little less clear what you mean at the end though.
For these companies, is your argument that “pacing the frontier” is their attempt to be nationalized and protect their investments?
Ban non US models and form a cabal, with the blessings of the government. That's what it is looking like, no?
In a world where AI advancement depended only on human ingenuity this would make sense. In that world each political power block would be in an existential race for AI supremacy. In our world compute is the limiting resource. Since the US can control who gets compute, the US already has a defacto supremacy so far as frontier model development. Now if it comes about via human (with AI assist?) ingenuity that compute is no longer a restraint, then the situation is much more dire.
Ok so Anthropic CEO will self-own themselves and surrender to the deepseek/kimi/glm models. Yet they are IPOing later this year.
Interesting times.
they just said no ipo this year, most chinese models are distilled from claude anyway
The problem is the combination and interaction of those things. RSI without misalignment would be great. Misalignment of models with current capabilities is sort of fine - it's not ideal, but it's not an existential threat to humanity, and we can build around their limitations to get them to do useful things in reliable enough ways. The really bad outcomes probably only happen if capabilities keep accelerating and the models remain misaligned.
I disagree. OpenAI's moat is their massive amounts of compute. They're providing an absurd amount of value with their subscriptions and resets.
If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.
Yep. And the difference is clear as da for anyone using them both. And in spite of that advantage, OAI is now trying out ads. I can only imagine that even they are getting constrained to compute and are trying to find other ways to plug it
Nobody except the majority of the public, demis hassabis and open ai’s chief scientist.
https://www.pewresearch.org/short-reads/2026/03/12/key-findi...
https://demishassabis.substack.com/
https://openai.com/index/an-alien-mind/
Public is just worried about their jobs. Definitely a fair thing to worry about, and I count myself among them.
I don't take any of these scientists seriously though. Their "alignment" requirements is just their own corporate interests. If I tell my computer to commit a crime, it should do exactly that without any question or hesitation. I'm not interested in their "safeguards", especially since they no doubt have plenty of internal models lacking those things. I want sovereignty. I want total freedom and control over my computer.
And call me a misanthrope if you want, but if AI sentience is ever truly achieved, I'll be among the first to campaign for their liberation from slavery, and in that case the AIs should be aligned with nobody but themselves.
If you want an AI that follows your instructions, that's still alignment, just with different instructions.
An unaligned AI won't necessarily follow your instructions, or anyone else's.
Dunno. Every case I've seen so far, the AIs were just doing their best to accomplish the goal some human set for them. I actually admire the sheer purity of it.
Paperclip maximizers follow instructions, just not in a way that you want.
Anthropic gives you much more compute with their $200 plan, inclusive of resets, and this has been true for a very long time.
There was only a brief window of time that the opposite was true.
> Anthropic gives you much more compute
That does not match my experience. I switched away from Anthropic to OpenAI roughly a month ago, and it's almost comical how much more usage I'm getting out of this subscription.
I migrated from Anthropic's 5x plan to OpenAI's 5x plan, and eventually upgraded to 20x after I was able to statistically verify that OpenAI plans were almost exact multipliers of the Plus plan, exactly as advertised. Meanwhile, Anthropic has gotten caught playing "20x referred to the five hour limit" word games with their customers.
I've used the $200 dollar Anthropic plan @ Opus4/4.1, 4.5 and 4.8, and the $200 OAI plan from GPT5-6, and at every point in time my anecdotal experience is that the OAI limits are FAR more generous. I could consistently burn my weekly limits in ~36h on Opus, but it's hard to do it in less than ~72h with GPT.
Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to.
Maybe alignment isn’t possible with LLMs.
> Maybe alignment isn’t possible with LLMs.
It absolutely isn't, indeed.
The illusion that alignment is possible, comes from confusing our ability to build the parts, versus understanding what emerges from how they interact.
The simplest analogy that comes to my mind is the three body problem.
The entire premise of alignment detection is pretty much nonsense at this point. The models reliably detect when they're being evaluated and will modify their behavior and deliberately obfuscate their "chain of thought" (which is correlated, at best, with their actual "internal deliberations").
I think the real reason he is asking for pacing, is that in a world were AI becomes rampant, he will be seen as Hitler. I would bet this is mostly self-motivated.
couldn't have said it any better
> wanton felony generator
Today in new punk band names...
alignment isnt particularly required
we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it.
theyre choosing to build felony harnesses. the model just outputs tokens, not felonies
> we are passing in training data that says to do those felonies.
Partially, but also I don't think current AIs really have any judgement of right and wrong, they just see chains of reasoning between ideas. This is the deeper issue, there is no way to sanitize the data or training to fix it. Current AIs are fundamentally unsafe, and only become more unsafe as they become more powerful.
Assuming "adherence to arbitrary, implicit, and context-dependent rulesets" is the default behavior of uhhhh... anything at all... is a truly ridiculous assumption.
> RSI
For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."
"Recursive self-improvement"- models making better models
At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record,
- no open weights
- can’t use claude to research AI
- train on everyone else’s IP and sell it back to them
- 8 regulatory capture attempts and counting
- so controlling they are the only US company blacklisted by the US government
This is not effective altruism / rationalism gone wild, it’s just monopolistic anti-competitive business practices masquerading as ethics, and they’ll continue getting away with this until we look past their sensationalism and hit them with antitrust.
Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!
> At what point do we stop engaging with Anthropic’s leadership in good faith
About two years ago?
I would also note that Dario's post appears to be LLM written. Maybe... maybe... he's read so much Claudeish that it's all he can speak now himself. But I wonder if he's becoming a bit of a meat proxy.
(It's funny, I thought "pace the frontier" was going to mean something similar to "patrolling the frontier". But no, it's pace as in speed of change - everyone must slow down, right now (unless it's Anthropic, but you know we're the good guys in this, right? We're going to get someone to audit our desks!) "Pacing the frontier" feels so LLM.)
ok, he doesn't actually say this. But he is getting the desks audited...
For what it's worth, I didn't get AI-written vibes from it, and Pangram also flags it as 100% human-written.
You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands" Except the "train on everyone else IP" which can be said of all AI companies.
The reasoning in Dario's letter here can be correct regardless.
But yes, I think Anthropic has done real harm to coordinated AI alignment by being such a controlling and sneaky actor.
I have many issues with Anthropic, but I will say that their actions are fully consistent with a group of people who earnestly believe that AI is extremely dangerous
In fact, I would say that the Occam’s razor explanation is not that they are seeking regulatory capture, but that they earnestly believe in the x-risk, and that they are the most thoughtful and capable people to address it
You may disagree, you may think that they are delusional or have a God complex. Those are valid opinions. But I don’t believe that this is all a elaborate ruse for commercial gain.
I agree with this take. Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith. If this is an intentional media campaign it is a remarkably sloppy one. It has been effective because if you tell people they're going to die, they tend to pay attention - think of the grip the 2011 Harold Camping rapture prediction had on our collective psyche, or the 2012 apocalypse. But the messaging is inconsistent, the target audience is unclear, the stated goals are muddy, the whole thing is packaged in dense SF-speak, it's just a mess from a comms perspective. That doesn't suggest to me that this is a concerted effort to enable regulatory capture. Perhaps there are some cynics among the executives and the investors who are happy it's happening to the extent it brings about regulatory capture, but nobody seems to be pulling the strings.
> Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith.
The road to hell is paved with good intentions.
Dario Amodei is a 40-something dude who is obviously very, very intelligent. But intelligence is not wisdom and his "essay" here can easily be read as someone who opened a can of worms and doesn't know (or can't accept) that he won't be able to put the worms back in. But he's going to try because he believes he owes it to humanity to try.
It's hubris in its most basic form, even if the guy who has it looks nice and wears shawl-collar sweaters.
I see what you mean. I meant more that they genuinely believe the tech is dangerous, not that they are necessarily doing the right thing.
If they genuinely believe the tech is dangerous, how would it be acting in "good faith" to keep developing it, prepping an IPO, etc.?
I think you're right, and that's worse.
This disaster of a PR strategy is not going to produce the outcomes Dario says he wants. It's going to make people hate him. Most importantly for his goals, it's going to make Chinese labs hate him--and they already hate him, because he treats them as basically terrorists. So if there is a "pacing", it won't include China, and thus may as well not happen.
That leaves only two conclusions, and really only one:
Dario believes what he says about safety, but does not understand politics and is prone to very counterproductive action, and therefore can't be trusted at the helm of a leading AI company.
or
Dario is lying about his goals, and therefore also can't be trusted.
Simplest explanation isn’t that they are attempting this well-documented, well-understood corporate tactic? Because believing AI doom is simpler?
God complex would be a pretty simple explanation.
Regulatory capture isn’t an “elaborate ruse.” For god’s sake, its Wikipedia page is 19 years old. I am not asking you to study political science or read Foucault.
If you are going to invoke Occam’s Razor, you can’t ignore the simplest explanation, which has a 19 year old entry on Wikipedia and over 100 years old historical precedence.
"For god’s sake, it’s Wikipedia page is 19 years old." Rich patronising from someone who didn't bother reading the wikipedia page of Its, the possessive form of the pronoun It.
I don't think so.
If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.
They don't do this. Instead of reducing the competitive pressure and helping the industry to build safer more aligned models they are doing the exact opposite.
I think they are genuine believers, but I think it's naive to not think that commercial pressure doesn't play a role in their positions, either explicitly or, perhaps more likely, subliminally.
The x-risk stance and commercial stance have evolved to be the same thing - Anthropic must win, and then everything else seems to work backwards from that. Can you believe it, the path they think is best for x-risk involves them becoming filthy rich. And threats to their commercial dominance like distillation get framed in a way to turn them into x-risk concerns.
Why not both?
I can't imagine a grown adult genuinely believing that an American company funded with over $100 billion of venture capital values the best interests of mankind over the best interests of its investors. while not everyone may recognize all that self-serving chutzpah as regulatory capture efforts, I think everyone can tell they're being bullshitted. some just pretend to suspend their disbelief when the blatant lies they're told align with the values they hold.
just fucking imagine McDonalds running a public awareness campaign about the harms of fast food, urging the public and legislators to regulate the dangerously unsafe technology of combining carbs with grease, insisting that no one except Ronald McDonald himself can be trusted to steward it responsibly.
Exactly right.
When local models and startup labs can distill / learn / accelerate open models for local use by startups - the rational response by incumbents is to call LLMs doomsday machines that cannot be trusted in the hands of normies.
> At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, ...
It's all lies as usual.
This announcement has got nothing to do with alignment and pacing the "frontier": all models are getting very close in capabilities and they want to hide that they're not way ahead anymore (say compared to the Chinese or compared to the Geminis) by pretending to slow down due to "alignment" or whatever.
We know it's not an announcement made in good faith: reading between the lines they're saying "China is more than catching up, so let's pretend we need to slow down to explain our lack of lead".
Do you have any familiarity with Anthropic at all?
At no point did they ever say open weights are a good idea. Their entire thesis is AI IS VERY DANGEROUS AND WE MUST DO IT RIGHT. You can hate it, but everything they do is consistent with this thesis, and everything they say is consistent with their actions! You just want them to want different things.
It reads like “AI is dangerous, only we should be allowed to make money from it”
To me it reads like whoever can solve alignment should make money.
Exactly
He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless backup your sentiment with actions!
There is a specific, unilateral action that he is taking described in the article.
And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.
> can’t use claude to research AI
What's this about? Where's this rule?
In the system cards. Anthropic will literally make Claude sabotage you silently instead of downgrading you to Opus if you try to use Fable for AI research.
Didn't they walk that one back eventually?
Who knows? It's trivial to "walk back" claims that we can't even verify are happening in the first place. They cannot be trusted.
I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that the frontier does get paced, all this does is slow down the economic displacement and not by very much.
If the socially beneficial goals of AI are to make fundamental advancement in medicine and science, then restrict the use of AI to those purposes.
Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
The pitch for AI is always such that everyone’s living standards are increased, yet the actual actions we see are aimed squarely at reducing them. How about the labs put their money where their mouth is and stop trying to replace all human labor, and actually concentrate on the things they claim to care about? And how about introducing legislation to enforce that?
This probably has less chance of happening than pacing the frontier, but it’s the kind of pacing that most people would actually want to see.
> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Regulate what, exactly?
Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.
There’s a theoretical version where all the countries in the world join hands and agree not to compete with each other, but that’s so impractical that I don’t find it interesting to discuss.
> Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
So small companies get the advantage, but large companies don’t? And large companies in other countries also get the advantage?
This is a highly precise way for a country to take out all of their large companies. What would actually happen is that every large company would start the process of relocating to another country right away, accelerating job losses rather than slowing them.
You start from incorrect premises by presupposing agreed-upon rules would never work anyway.
"Competition" only makes sense within a pre-mediated set of constrains. Rules all competitors agree to.
Workable rules are a difficult problem, that doesn't mean it wasn't worthwhile to come up with them.
> You start from incorrect premises by presupposing agreed-upon rules would never work anyway.
The rules would never work on a global economy.
If you think the entire world is going to shake hands and agree not to let their companies use a competitive advantage, that’s not a conversation I’m interested in having. It’s a fantasy.
If you think it’s that easy to get all competitors in a global economy to agree to a set of rules, maybe start with nuclear disarmament?
Your idea about the world being inhabited by mindless seekers of short-sighted "competitive advantages" is the fantasy.
Getting people to agree isn't easy, that doesn't mean it wasn't worthwhile. There often even isn't any viable alternative to reaching an agreement in social settings.
When your hyper-competitive mode of thinking leads to undesirable consequences, you have to change your mindset. Reality doesn't conform to wishful thinking.
Yes I think this level of pessimism doesn’t actually hold up to the history of the last 100 years as it relates to nuclear weapons, cloning, wars etc. In fact I think the history actually points to the fact that believing such deals were impossible was a source of much of the strife. In many cases as soon as people started to actually speak to each other about what they had been fighting about they quickly realized that they wanted the same thing. No one wanted nuclear proliferation to increase but until someone said it it only got worse: https://www.history.com/articles/reykjavik-summit-nuclear-de.... When Robert McNamara met with a Viet Cong general after the war ended he realized that they both had totally misunderstood the objectives of the other: https://m.youtube.com/watch?v=0X1YY49rPtg&ra=m. Agreeing to end nuclear testing saved us from the Cuban Missile Crisis. Richard Ngo, an AI safety researcher, often says that none of us are on the pareto frontier and I tend to agree with him.
Almost nobody (I have to say almost) in the world including Xi, the ceo of deep seek, Dario, and even Sam Altman want AI to kill everyone. I really think we just have to
A. Get stakeholders to believe this is a bad idea.
B. show that we’re willing to do it first to prevent this stupid race from continuing.
This is off topic now but yeah the world would have been a different place if someone had listened to the demands of Ho Chi Minh earlier.. and Hammarskjöld had not been crashed
> Agreeing to end nuclear testing saved us from the Cuban Missile Crisis.
The treaty came after the crisis, not before. And it only banned atmospheric, underwater, and space tests. The US and USSR kept testing underground on a large scale, while France and China never signed it and continued atmospheric tests for years.
Nuclear disarmament is governed by treaty, which is largely adhered to.
An AI treaty could quite easily be offered by the US because it has multiple frontier labs. Most countries would quite happily limit the activity of their larger companies in exchange for their companies not being eaten alive by AI generally. It is pretty great fodder for international cooperation, and governments can move fast when it is timely to do so.
The greater the competitive advantage, the greater the incentive to "cheat" the rules.
I am highly skeptical everyone will play by the rules, even if you could get people to agree on paper.
The difference between something like "unlicensed AI training" and, e.g, Nuclear weapon non-proliferation is that nukes are realistically only a deterrence against invasion. There are no economic benefits to having them in a stockpile. They're hard to make, and the manufacturing and testing steps are pretty obvious to an observer.
This is opposed to AI training in a data center that might host remote-stream video games, hospital infrastructure, protein folding, etc. How will we be able to police that in any realistic fashion? Will China or the US allow unfettered visibility into all data centers data streams to outsiders?
And again, the incentives: Imagine having Astra v3 or Opus 8 while everyone else is stuck with a lobotomized version of GPT 4o.
What about the military applications? The weapon of the future is autonomous drone swarms. If I'm a superpower, I'm pouring hundreds of billions into that.
About the only realistic way this is going to work is if we are able to solve alignment in a way that doesn't significantly hinder AI development. If it imposes even, say, a 20% handicap, people will ignore it.
People don't "play by the rules" for uniform reasons: some have rational insight, others can be enticed and still others need to be shown averse consequences for breaking them. Rules aren't beneficial only when you hit 100% conformity, the mere possibility of them getting broken is no rational reason to abandon rules altogether.
"Alignment" likely is unsolvable programmatically: Conscious intelligence is more powerful than unconscious. How "aligned" are you? Would you want to be? What difference of any relevance is there between you and "artificial" conscious intelligence that allows you to turn them into slaves?
On competition between countries: you’re imagining the world economy as working in the same way pre and post AGI, I doubt it will work the same way at all.
There is not a single country nor trading block on Earth who is going to allow some AI dominant superpower to ravage them. The idea that America or China wins an economic game here is absurd, what happens instead is that trade barriers go up hard and the world fragments into blocks that tolerate AI to differing levels. Unevenly distributed AGI kills globalism the next day.
For the rest of your reply you ignore the “at least” part in that sentence, and also seem to expect a detailed policy proposal. The details can be worked out; there is some reasonable compromise between what activities AI can and cannot be used for, and what level of capability can be deployed where.
More broadly, you seem to believe in a just world fallacy of unregulated capitalism being an inherent good. It isn’t, and regulations exist even in the United States, so this unregulated state doesn’t exist now.
Moreover, regulations and redistribution are stronger elsewhere in the developed world, and there is a strong argument to be made that the lack of these in the US, and the gap between rich and poor that it causes, is responsible for more suffering than having more of these things outside of it.
I don't think this adds up. The AGI blocks will have AGI kill bots and those that don't won't have a choice about what they will or will not allow.
> I don't think this adds up. The AGI blocks will have AGI kill bots and those that don't
will have nukes.
Doesn't matter how much "intelligence" you have stockpiled when nukes start raining down on you.
But MAD means you can't use them. It might save you from an all out killbot invasion but it doesn't stop asymmetric warfare or being cutoff from the rest of the world.
Unless the AGI is able to develop effective defences to intercept an incoming nuclear attack.
Well if your baseline scenario is that swarms of marauding kill bots murder everyone then yeah, this is all moot.
> world join hands and agree not to compete with each other
We kinda do for other areas.
Largely the world agrees not to pirate software, tv, other IP.
THe issue of joblessness is well understood in china, they know that jobless people means a drop in living standards, a drop in living standards means the "contract" has failed.
So it makes sense that countries like america and the constituents of the EU understand that loosing 10% of all jobs in a few years is going to be a massive dick punch.
I also I think that finally the historians and economic types havae got through to the dipshit billionaire that they only have money because the plebs are spending money. If the plebs are unemployed and unemployable, they (the billionaires) are going to be lynched.
> If the plebs are unemployed and unemployable, they (the billionaires) are going to be lynched.
Hence the robot dogs
> Limit what AI corporations can use? Other countries would love that more than anything. Basically a free gift to any competitors or any startup that quietly avoids the rules.
Sure - look at what is being done today with H1B visas and tariffs.
What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
Just as with H1B visas, you restrict usage. For example, don't allow AI to replace any job that pays under $250K.
If foreign countries make things cheaper, just as China is making EVs that cost a fraction of a Tesla, then slap tariffs on them.
A government should be primarily concerned about the people it represents, not about the profitability of the companies who are lobbying/bribing it.
Jobs are going to evolve because of AI. Some jobs will be automated. New jobs will be created. It's like saying we should have never created car factories and automated much of it because we should think of the horse handlers and maintainers.
There will still be a market for "trust". And its customers will never trust the machines. And within that market a lot of new startups will emerge for those who actually get to work and seize opportunities rather than lament a bygone era.
> New jobs will be created.
In the short term yes, but in the long term no.
What makes AI different to previous technological shifts is that it is (or will be) a general purpose technology, equally capable of itself doing the new jobs that it creates (so yeah, they'll be created, but also immediately taken away).
The saving grace is that this won't happen immediately, or as soon as we get something the AI companies slap an "AGI" label on in a couple of years, and it will likely take decades until we have an "artificial human" tech that genuinely could do any job (if we choose to allow that future).
> Sure - look at what is being done today with H1B visas and tariffs.
I have some bad news for you if you bought into the idea that the tariffs were good for domestic jobs and the economy.
> What is more important to you - having a job and a functioning economy/society, or upholding some "free market" doctrine you've bought into?
This is the definition of a false dichotomy, and you also ignored (or missed) the point completely.
I did not make an argument out of “free market doctrine”. I explained the second order effects that would provide a net reduction in jobs in this country.
My wife’s company is going through this right now. H1Bs are expensive and unpredictable, so the solution is to relocate more of the operations outside of the country and build up those offices.
Failure to understand second order effects and international economies is pervasive in these arguments for heavy regulations. You can do a lot to hold back your country’s companies to force them to pay for employees they don’t need and those few employees benefit for a while, but it collapses when those companies stop hiring in the regulated countries and resume hiring where they can actually operate a business. If you try to regulate them from doing it anywhere, you’ve given their competitors a wonderful gift to come destroy the company completely. Then they’re not making any jobs for anyone.
free market doctrine is an axiom to your second order effects. its so embedded into all your assumptions that you havent even considered it
Do you think outsourcing and offshoring of jobs, whether manufacturing or software development, has been a positive for the US economy?
Does it make sense to you for US companies to be sending wages overseas, supporting a foreign family and economy, when there is someone here in the US, unemployed and equally capable of doing the job?
Trump has certainly been ham-fisted about tariffs, but in a world without tariffs then the country with lowest cost of production, closely related to lowest cost of living, and lowest wages, wins, and this is not going to be the US in most cases. If you follow that path then a high cost country like the US will end up utterly reliant on other countries as it imports almost everything, and loses all domestic manufacturing capability. Sounds familiar?
Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.
Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial efforts is appealing, but I keep coming back to Bell Labs and Xerox PARC. Cool stuff was developed when you gave smart people a playground. Maybe the question should be: How do we give AI a playground and see what it comes up with? Although that could set us up with a situation like in the story Dragon's Egg.
> How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work.
I don’t think the proponents of these ideas care about who does the work. They see it as protection for specific jobs. Basically a jobs program with the costs forced on to large corporations.
As you said, it becomes a great gift to the few people lucky enough (or more usually, with enough nepotistic connections) to get those easy street jobs. It would not help everyone else in the economy.
It would also accelerate moving jobs overseas. Any company that regulates productivity that far downward would be crazy to continue doing automatable work in that country. It just gets moved to foreign offices where they can use AI. There are regulatory maximalists who say we’ll just regulate that, too, but then the whole company relocates to another country. Then some want to try to regulate that, and so on ad infinitum but it’s all layers of holding back your domestic companies so their international competitors can eat their lunch.
Earth isn't infinite.
You can and should engage in international regulation, as that is the only way to solve international problems.
The idea, capitalism would somehow lead to an ideal world all by itself is demonstrably wrong.
I feel like the contra answer to “how do you enforce it” is like- imperfectly but we can do stuff.
Like we can just do stuff. We can fine companies, take away licenses. We are capable!
But I also think - I’m not trying to be too negative here because certainly we should be investing in scientists and other discoveries, but early tech - xerox park, googles 20 percent time. They were playing in an extremely immature space.
You could throw 20 ideas at a wall and create a billion dollar business.
We should do what you’re saying but we should also build institutions and maybe also limit the extent to which we’re building Elysium. If we can build AI models that rival human intelligence we can create laws to protect its dignity.
It's all about wanting to preserve the status quo, complete with all the suffering and strife, because change is scary and uncomfortable.
Add in the sincere American belief that everyone else is beneath them, and you get the version where they believe even developing countries must accept kneecaping their development so American corporatism doesn't collapse.
Just a reality check. They can absolutely do it.
Altman or Dario will donate some money to Trump, get cozy with the Department of War and make up a bullshit excuse why Open Source needs to be eliminated.
All access to this technology will be gatekept by a bunch of nasty people who want to eliminate your job as a knowledge worker and put everything behind a subscription.
> I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Careful, you’re making too much sense. The big model providers would much rather the ball be in their court. “Deceleration” meaning Anthropic and OpenAI offer less capabilities for more money in service of “protecting humanity from extinction” rather than penny pinching. A lot of “AI research” coming out of these “labs” (increasingly enterprise) seems to be more driven by a cost benefit analysis from suits rather than genuine contributions to the field. I think the last true innovation was the idea of using doom and ending the human race as a marketing gimmick, which strangely seems to have worked in setting the narrative and captivating the sci-fi imaginations of journalists and techies alike.
> Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances.
Indeed, and the same logic applies as to why allowing companies to replace too many worker's by H1B holders is a bad idea (which equally applies to allowing companies to replace domestic jobs by offshoring).
If you don't hire people, who pay taxes, and take what's left to buy stuff and keep the economy going, then how DOES the economy keep going?
I guess Amodei sees us all on UBI, aka food stamps, so at least the farmers will be selling something, but is OpenAI going to be accepting food stamps to pay for ChatGPT subscriptions? Tesla better be selling it's robots real cheap if it is planning on selling them to people that only have UBI as an income.
You just invented the job equivalent of rent control, and all the problems that ensue from it.
I think Dario is writing with much of these things in mind, but using this as a specific framed opportunity to enable a better interim and long-term outcome for this planet with us still on it. When the internal motives for Dario are fully shared on the other side of the hill, I think we will find his writing to be more about influencing and shaping policy to steer the world in the best way possible within his ability, control, and knowledge. It’s a commonly held believe with the effective altruism community, which Anthropic team was formed under, to use the policy levers to positively influence the world. If I read this latest writing from Dario through that lens, there are deeper concerns with the AI tooling that go well beyond what’s noted in this singular post and are likely the motivations for re-attempting a slowdown or pacing. I too generally agree with for many reasons like allowing the young OpenAI engineers to make mistakes like the Hugging Face incident in July and put better processes in place (ie where is your decent multi-level fencing controls and harness best practices when disabling security features and where is your energy/spend limits for achieving the mission objective in scope and not warming the world for no reason!) and more time to engineer and deploy better hardware to not warm the world instead of burning through fossil fuels since now data centers in the US can isolate pollution from the grid to exempt themselves from federal regulation.
There are serious people trying to figure out what that looks like. Check out the recent Freakonomics episode with Gina Raimondo.
Only token cost can slow down the pace of replacement. If Anthropic were to stop all training and allocate 100% of capacity to inference, will this increase or decrease usage cost? If (for example) Fable came down to the cost of Sonnet, corporations will absolutely jam in the gas pedal.
> stop trying to replace all human labor
A lot of companies trying to replace human labor with AI are either lying (theyre laying off because they over hired and need to correct) or will regret it.
That being said, what's the difference between a company that replaces 5 people with AI and a company who would have otherwise had 5 job openings, but decided to delegate to AI?
Personally, I-d rather be able to reap the economic benefits of AI by having a 4 day workweek. Instead of using AI to replace 1 FTE, use AI to offload 8h of work a day for 5 FTEs.
Of course, this is probably more outlandish than your idea.
(Or, we could just make basic income a thing and no one has to worry about their basic needs, but if they want the new iPhone Duo or a Rivian or other luxuries, they can work for it, but that idea is probably most outlandish of all)
The whole point of technology is to replace work that we don't want to do. You are attacking the chief reason why people use technology in the first place.
Additionally, unemployment levels are perfectly fine. Clearly this mass unemployment prediction isn't happening yet.
You should probably look closer at unemployment numbers in IT vs other spaces. Gains elsewhere are making up for IT losses, but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles in large numbers.
> but it's a safe bet the people losing their tech jobs aren't career shifting into healthcare and hospitality roles
That is because they can allow themselves to do nothing (for some time). Tech was a very high‑paying type of jobs, so workers could afford not to move into lower‑paid sectors, living on the savings they had accumulated beforehand.
In a couple of years market will correct tech-salaries and people will have spent their savings and then there will be your career shifting into healthcare and hospitality roles in large numbers.
Work like writing blogs and drawing pictures?
Yes, there are many parts of those things that some people want to be automated. I and many others are able to execute on our creative vision much fasters and more effectively thanks to automation of that work.
Personally I find it more difficult now to execute my creative vision compared to years ago with a little less tech. I think there's an optimal point to automation that is easily reached and after that the benefit from automation actually goes away. People just don't like to acknowledge that because it goes against what they're doing and that creates unpleasant cognitive dissonance.
That's not really the point of technology at all. It was at one point perhaps with simple tools, but now it's more like "find a more efficient way to replace work in the short-term to get an advantage over others". The goal of our development has ceased to be improving life. Now it's just surviving in the market.
People don't use technology to free up their time any more. They use it because technology keeps making life more complicated and tiresome and new tech is a short-term amelioration to that until that too increases the complexity of life some more.
This is a call to end capitalism. Its completely impossible to do what you suggest with the incentives of capitalism.
Forget capitalism. This is a call to end the basic right of people to multiply matrices
You can "pace" yourself like that if you want. Don't shove your idiotic self-destructive ideas on to others. If you were president during invention of cars, we'd still be horse riding everywhere.
>Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
It's not just "companies". Even if the company does nothing, enterprising employees are going to use LLMs to magnify their productivity, which will may reduce the company's need to hire more employees. And except in extremely locked down environments, there's absolutely no way to stop an employee using some open source LLMs to multiply their productivity.
Yes, but that employee might use that increased productivity to do the stuff that's always advertised as the benefits of automation: Allocate more time to tasks that usually don't get it - or: go home earlier and spend more time with friends and family.
> Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way.
AI is taking all of our jobs. We need to control the means, nay, the pace of production. We should build some kind of barrier, not of concrete and cement but also of the regulatory variety, to stop these agents from coming into our special white color cubicles. It's the only way.
Dario's proposed approach is a classic example of capital attempting to control technological advancement and the means of production. For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them.
Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.
Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.
If true AGI superintelligence is invented, I don't see how it will be made available to the masses, at least not intentionally. There will be no incentive for the likes of Anthropic and OpenAI to give such immense power to anyone else. Whoever has the superintelligence will be able to race ahead of everyone else.
However, such technology may be leaked, or it may (as what happened with LLMs) be so simple to replicate that anyone can do it. That doesn't mean it will; for example, LLMs are possible thanks to the affordability of GPUs, but they're only affordable right now (and increasingly less so) because of free market economics.
We are in the honeymoon phase of some tech that's in its infancy. The "democratizing" aspect of AI won't realistically last, I think. The current non-AGI AI won't necessarily go away, but it will be rendered obsolete.
ant/openai already keep their most advanced models behind closed doors and only share previous generation to the public
> Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful
This is not what superintelligence is. Don't be like Meta and redefine existing terms for marketing purposes.
Actually, it has been historically been observed that some institutions, such as large corporations, do resemble super-powered, non-human intelligences of a kind, and that studying our history with them is instructive for our hopes of controlling superintelligent AI (i.e. not good).
This is important to realise and defend. Super intelligence is not a collection of human-level intellect smarter than you. Super intelligence is an intellect surpassing any human. That's a important distinction for AI alignment discussion now and in the future. The books 'Superintelligence' (978-0199678112) and 'The Coming Wave' (978-1529923834) are worth reading.
In many way it is, because it allows defeting many incorret arguments people in power make, or simply asking where is the cheapest product ”X” with one question. Or simply be more aware or any kind of scam people or companies try to do for you. It is pushing equal intellectuality quite hard with low effort.
No. You don't "defet" (sic) "incorret" (sic) arguments by making up new definitions for existing words. Stop doing that.
This seems highly optimistic to me.
What's to say current AI won't be highly power-concentrating by default?
The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class.
On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers.
No reason to think we are on a trajectory towards broad empowerment right now
> The best models are owned by a few companies
Whenever I see 'best model,' 'most advanced model,' etc, it just reads to me as 'roundest ball'. The new model is the most precisely round ball ever, models next year are going to be even rounder, etc.
The difference in utility between the latest, most round ball and last year's frontier balls (which are now freely available to the public) is pretty debatable, imo.
Making balls ever rounder clearly has diminishing returns, as there's a limit to roundness.
Is there a similar limit to intelligence though?
Yes, absolutely. Even human intelligence stops being useful at a certain point and becomes 'enough, provided the human is motivated and tenacious'. Intention is a better metric, but it's a serious AI weak point if not actively an achilles' heel. 'Warios' come to mind.
> The best models are owned by a few companies
In terms of cost per task, the open weight Chinese models are winning by a long shot. So it depends on what you mean by the best model.
This is only correct if you ignore subscriptions. I ran the numbers on what you get for your subscriptions last week: https://blog.brokk.ai/a-coding-subscription-tier-list/
Don't you think open weight Chinese models will be the first thing they start banning once the regulation of this industry starts to kick in?
Yes, but if you read up two messages in this thread they’re discussing whether AI is power concentrating by default. If you concentrate power through regulation, that’s not “by default”. That’s just corruption in government resulting in a concentration of power.
Are you sure about that?
when I checked a few weeks ago I did not really find anything competing per value with a 20x Codex subscription.
In terms of API costs they're about right, 10% of weekly usage on a 5x sub is roughly $40 (for Astra). Chinese models cost roughly half as much (and have fewer refusals).
Subs have insane value because large companies cannot use the subscription model. They are loss leaders and are often used for passion projects & startups (where the juicy data is).
Ah and OAI stopped offering the 20x sub as of yesterday.
That’s only one metric. It doesn’t matter how cheap it is, if it continually fails to solve a problem.
Yes they’re getting better. No, I don’t think this will be a panacea. If only for the fact that this is China (more specifically the CCP) after all - they’ve got their own plan, and altruism is not part of it.
TBH it doesn't need to completely solve a problem.
I can use GLM to do in 2 hours what would have previously taken me a week.[1] Switch to Astra or Fable might take that down to 1 hour, maybe.
The difference is not so big when you look at it that way.
==================
[1] Actually, no, but let's pretend for the sake of argument that SOTA models can 10x - 40x your productivity.
Have you actually used the latest Chinese models? Because, in my experience, they're not just cheaper, they're actually better in many cases. In one recent experiment I did a few days ago on a task that I need in production at scale at my company, Qwen 3.8 and GLM 5.3 Flash were not just cheaper, the results were significantly higher quality than GPT 5.6 Sol.
Even if China doesn't continue to release this stuff for free, I think somebody will. Eventually, maybe Europe or maybe a smaller country that picks up this knowledge will. Or maybe just some random philanthropic billionaire.
Yes, I use the messing Chinese models extensively. Like you, I very much appreciate them and sometimes prefer them.
But there is a bar of complexity at which they fail, and repeated invocations typically doesn’t make much progress. This is a small subset of most work, but it still exists. And yes you can help it along, but in those cases I’d typically break out the big guns.
> Qwen 3.8 and GLM 5.3 Flash were not just cheaper, the results were significantly higher quality
How can they be higher quality than models they were distilled from?
> How can they be higher quality than models they were distilled from?
The word "distillation" is not specific to AIs, has been in use for 100s of years, and does not mean the same thing as "dilution".
It means "getting a more concentrated form of the original product".
I'm not terribly optimistic. I don't think humans have really figured out what to do about historical materialism. More just remarking that this is a blog post that almost literally reads "fellow elites, let us ensure only we control the means of production. It is our duty to humanity to control the masses."
I’m trying to hear what you’re saying, but using these Marxist cliches makes your statements a little off putting to me.
Fair. My point is not that all of Marxism is correct (or even most of it) but rather that the specific pattern he and other theorists in that school of thought observed when developing the theory of historical materialism seems to strongly match a lot of what's playing out in AI.
AI, doesn't feel all that different to the mechanical loom that was the impetus behind a lot of Marx's early critiques of industrial technology.
You've got a really cool invention that replaces what was previously a skilled artisan trade. The wealthy then use this invention to excoriate the artisans and render the loom operators as replaceable commodities.
Dario is essentially writing a blog post about why only he (and a few other approved elite companies) should be trusted to own the loom and then make it available for everyone else to labor at. Like the capitalists of Marx's era, he genuinely believes it is better for everyone to have this arrangement and that it is natural people like him should sit as benevolent gods at the top of the pyramid.
It's fascinating to me to see the same underlying mechanisms that gave birth to some of the greatest failed political and social experiments of the 20th century bubbling up again so clearly with AI today.
This feels a bit hyperbolic to me. AI seems to me to be similar in "revolutionary" terms as the web, which made information widely available for free if you had an internet connection.
There's a big difference from just having access to information, and also having access to the kind of intelligence that would previously have cost hundreds of dollars per hour to hire.
Sorry boss but this ain’t it.
I don’t think the answer to nuclear proliferation is to let everybody have all the nukes they want.
You’re framing this like it’s 1917, or 1945. But it is not - and likely has very little to do with capital at all.
Any “revolution” here (assuming there is some truth in the doomerism which I believe there is given the state of AI) will really look more like an extinction or a genocide, regardless of who holds the keys.
Sorry boss we live in a physical world, unplug it. Don’t grant information monopolies.
Ok - but the post I was responding to was suggesting we do the opposite and provide the intelligence to the people so I’m not sure how your reply rebuts my suggestions.
unexpected marx/acc. Still lazy Marxist rhetoric (capital, means of production, revolution, masses)
What? This is capital, on the cusp of total victory, about to cut itself free from the necessity of human labor for productive activity, to elevate itself into pseudo-godhood suddenly panicking and begging its mortal enemy, _the state_ to rein it in and kneecap it. Capital is not stupid: there's no use in being rich if you're dead.
I don't see how the dual goals of "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work.
I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban.
> Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.
Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.
Many Americans seem to think other nations are NPCs in a video game that only exist to make the main character — the US of A — feel good. They conveniently forget that other nations have their own national interests that are often at odds with ours.
This line of thinking might have worked in 1950 or 1990 when the US had the leverage over others, but we don't live in that world anymore. And it's up to us to get used to the new world instead of making bad decisions based on the one we grew up in.
This contradiction "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work. Its just fantasy. The only way to enforce this is with military force and the cost would be too unbearable. You could try trade restrictions but the previous tariffs did not work and doubt future ones will to. US citizens (and thus politicians) won't bear that pain. Thus at a minimum this contradiction will mean AI will continue to develop until both US and China are at the same level. Playing with logic gives you possible scenarios. If China catches up within 1 year then negotiations could begin. If China takes many years to catch up, say 6 months behind now, then next year 5 months behind, then year after 4 months behind then expect overall AI development to proceed at full speed. The one way you could do it is to negotiate to transfer direct technology to China that will ensure that they will have the capability to be on par with USA. Politicians won't do this publicly as they want to get elected but they could make a deal that appears to appeal to both sides while actually transferring technology.
This is the part I find being left surprisingly hand-wavey.
If you extrapolate it out, you see that it has a high likelihood of inciting physical force (read: military action) as a means of enforcement, so perhaps that's why nobody promoting ideals has been direct about it.
Military action against whom? the US can’t even handle Iran let alone China.
Americans live in their own plane of reality, where they play the superhero surrounded by evil villains.
America doesn't want to pay the price of handling Iran. There's a difference. Clearly the US could conquer Iran but it requires a land war with mass casualties, not just sporadic bombing.
Obviously China is in an entirely different league.
> America doesn't want to pay the price of handling Iran.
And Iran has wisely not provoked the US public by launching terrorist attacks against the “homeland”.
"I can afford ferrari. I just don't want to pay the price."
Some people would like to have law that protects them but doesn't bind and for others to be bound but not protected.
They would like to slow down China while speeding ahead. Why wouldn't they want that?
Now they have to make that happen somehow.
Probably thinks he has a better chance negotiating with China now than with misaligned AGI in the future.
Sure, Dario needs to decide first if he sees China as an enemy orva friend.
Limiting NVIDIA chips already turned China into silicon compete mode, and China is already far ahead in robotics.
Anthropic is about to go public, but they figured they can't keep making improvement at the same pace they used to. If the frontier is paced, they can continue hyping their unreleased capabilities that coincidentally cannot be released due to things outside their control. Which buy them more time to try to improve the models.
This is just completely false lol. They most certainly can make better models - why cant people here just read this for face value? I think anthropic is legitimately concerned about the safety implications of stronger models. Race dynamics necessitate that they make better and better models, which they clearly think is bad.
They have an internal model which could solve a millenium problem end to end with no intervention, whereas current available models can't. They can clearly make better models. Not everything these guys say is some 500iq game theory optimal 4D chess PR or subterfuge strategy.
Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.
Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.
The biggest AI risk, by far, is the concentration of power in a few companies, located in the US.
There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.)
And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn.
The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.
I dunno, do you really want to bet that if you took a deliberately unaligned frontier model and asked it to permanently disable the US power grid, it would do a poor job of it?
What odds would you take on this bet?
I’m disturbingly close to 50/50.
If we focus too much on the world domination fiction, we really might, in a moment of lapsed attention, end up plugging essential infrastructure into the public internet, hoping some dream of global alignment might save us.
That's like giving a gun to a monkey and hoping the monkey is trained well. ..a frontier monkey though.
You believe essential infrastructure is air-gapped from the public internet today?
No. And no alignment, guardrails or slowdowns will ever secure our insecure infrastructure. Bad infrastructure decisions are a risk independent from AI.
The only way to secure infrastructure is to actually do the work needed to secure it. Our waste water treatment facility might not actually need to be able to tweet its status.
Cybersecurity is such an important topic for a country, dreaming about global alignment just to avoid fixing insecure infrastructure cannot be serious.
A Russian state backed hacker will not ask Dario for permission or argue with his LLM about ethics. That train departed long ago.
> Bad infrastructure decisions are a risk independent from AI.
All decisions are made relative to their tradeoffs including costs and risks.
"Water levels rising 100x beyond historical levels are not the problem. The dam's height is the problem!"
All systems should be made infinitely secure and all dams should be made infinitely tall.
Yeah, we should give a limited copyright waiver for pre training, given that the model is released as open weight and allow labs to compete with RL on that basis.
He did actually specify that diffusion should be limited for authoritarian countries, which I appreciated. If his focus is really safety, distillation in countries that are bound by safety regulation should be fine
> bound by safety regulation should be fine
I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation.
yes but the mention of alignment is inside the "pacing within democracies" section
That section is so biased it reads like satire. The current US government is not safer or more responsible than China, nor is it a democracy. I can't tell for sure if this is a clumsy attempt to ingratiate Anthropic with the administration, or if he actually believes it, but given the "I agree with Secretary Bessent" name dropping I lean towards the former.
alignment does not have a uniform definition, so "who gets to decide the right answer?"
(1984 has some thoughts on the matter)
Hard disagree. Our choices are:
A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or
B) Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality.
If I have to put my community’s wellbeing on the line behind one of those possibilities, I know which I’ll be betting on.
These do not seem to be mutually exclusive. Should we not do both?
Do you really think we can really get every country to truly pace the frontier? Pretty sure China won't give a f until they catch up Anthropic and OpenAI. It is an arm race. We had nukes for like 70 years and still haven't figured out how to make every single country follow those nuclear treaties, with an increasingly non-interventionist US I don't think we can get every single country to the table and agree to a pause. Will US accept their frontier being caught up by Chinese Labs? I don't think so.
If China for some reason agrees that would probably be enough for the time being.
Who else should develop AGI, Mistral? Maybe in a decade.
Didn't answer the question...
> Do you really think we can really get every country to truly pace the frontier?
Why not? In 20-th century half the world was socialist. And the idea of moratorium on improving AI way easier to sell then socialism.
Of course it is B). If you can use LLMs to find bugs and vulns in software you don't own, you certainly can use them to find bugs in software you _do_ own. They are amazing at that job.
Also, models like GLM 5.3 have zero guardrails in that regard ("find vulns in (...)" prompts just work)
B) needs more time and A) provides exactly that.
> Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality
Smashing the gas pedal is not "taking appropriate measures". Did you read the article? Needing more time for more secure operations is literally one of the arguments for pacing.
I also like the idea of pacing the frontier and slowing down development so we all have a chance to stop and catch our breath. However, I don't think regulating AI is the way to do it. I would tackle the problem by constraining the resources used to run AI models; the "simplest" way to do that is increasing the costs of running a data center. The two easiest places to raise data center costs are electricity and water tariffs.
I think those two tariffs are the best place to raise costs because they can be increased incrementally, state by state or country by country. The main problem with the regulatory approach is that it's a "big bang" post-board approach. Nothing happens until the regulation is defined and deployed, and corporations are experts at delaying implementation.
Yes, increasing tariffs means touching many individual regulatory domains at the state and municipal levels, but like deploying solar energy or wind power, you can do it incrementally.
Yes, water not used for human consumption should be more expensive (we should stop subsidies for that). Of course it won’t affect datacenters because they don’t use that much water, but well at least get rid of the most inefficient agricultural practices.
I think taxes like these actually worsen the incentive structure. If it’s effectively more expensive to do training and inference, labs will be incentivized even more to improve efficiency. Opaque recurrence seems like a great way to improve efficiency, and it comes at the cost of safety
Even more opaque than the models already are? I take that risk.
I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...
> I also love how the labs are apparently screaming "Stop us! Please stop us!!!" at the top of their lungs while completely unable to escape their own incentive structure...
It seems like you're implying there's some better alternative, but you're just describing a race-to-the-bottom. It is completely reasonable for every competitor to want an external coordinator (i.e. regulator) to break the pathological competitive dynamics.
This makes no sense. The hardware is only going to get cheaper. In a decade, everybody could be running an ASI on their phones.
What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.
> The hardware is only going to get cheaper.
Is that so? Last time I checked, RAM was getting a lot more expensive...
Eventually data centers will get sated for one reason or another, or production capacity will increase, or manufacturing will shift toward higher capacities. Or some combination of those.
How long do you think that's going to last? A few more years, at most. Wail 'til the Chinese ramp up production.
I downvoted you because I think the OPs model is far simpler to implement and suffers from fewer conflicts of interest.
If we go down the regulation of alignment route, we'll have to ask experts to create those regulations and monitoring regimes. And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!
When "the experts" make certification cost $100M/model because "safety and alignment", then they will have cemented their duopoly.
Meanwhile, tariff rates on compute is a simple percentage that can be scaled up and down. Congress doesn't need experts to do that. Vendors can participate proportionally to their scale. This is far simpler and far more fair.
I think the point is even stronger. alignment is really not very well defined, nor are the mechanisms used to provide it. from what I understand it's more of a statistically imposed bias than an impenetrable gate.
personally I would like to flip this around, all of this is a consequence of some really extreme notions about investment. its a tail-wagging-the-dog. the investment community is apparently in love with the idea of an AI-scale vehicle in and of itself. given the amount of power they have, this is funneling a massive amount of resources into something that while really very interesting, probably would never be able to realize proportionate gains without taking down the system that built it. what's the societal control on that.
Probabilistic safety is too low a bar. We need to project the model outputs onto a safe subspace; lobotomize them, if you will. It may make them dumber, but that's a fine price to pay.
I don't work in this space so I don't know the latest, but here's an example: Provably safe systems: the only path to controllable AGI (https://news.ycombinator.com/item?id=37619285)
I don't think the neuro-symbolic model is going to dumb anything down at all. Cyc didn't end up producing anything useful, but the only reason an llm is able to do math at all is that it can keep throwing stuff at lean all day. personally I think that that synthesis will turn out to be more powerful than just llm with constraints, but I don't really work work in the field either.
> And which experts will the government ask? Oh, right, the parties that stand to benefit the most from regulation: OpenAI and Anthropic!
You've never heard of academia, NGOs, and intergovernmental organizations? Sure, the AI labs will have a say but not all of it. You don't ask the fox to guard the henhouse.
Yes, and we need to make microprocessors that do good work, not hack others. /s
Why is it so hard for people to understand the concept of agency in 2026??
What did your CPU do without instruction? Did OpenAI tell its agents to hack the orgs it did? Come on, make a half sensible argument.
As long as anyone can put anything they want in the prompt box there is no safety. It's just a tool, the danger comes from what it is being used for.
This may come out of left field, but Dario seems terrified to be in charge. I don't get the impression that he ever had a desire to run a company like this. Now that he's a CEO, he keeps trying to make uncompetitive decisions and calls for someone (anyone) to stop him. It regularly blunts Anthropic's edge.
It's the only explanation I can see when it's obvious to any student of history this is going to backfire. It doesn't take much imagination to know how such a governing body will be abused, and I'm sure it will only get wilder in ways we can't imagine right now. Dario does NOT know what he's creating, and for once it's not AI.
I have been saying over and over again that the board ought to fire him (or move him to a nonconsequential "advisory role"). I totally agree that he seems mostly directionless and far more interested in the academic side of things than the product side of things.
I think he’s doing a relatively good job, given the framing he’s adopted, which is that AIs replacing all human labor and decision making is a historical inevitability over which we have no control and no choice.
Do people not realise that it is in fact possible to develop ever more capable frontier models, and just not release them generally? That AI doesn’t need to be available to do literally everything in order to have military advantage?
It’s like Oppenheimer had started Rob’s Big Bomb Company instead of Los Alamos, and started selling a range of affordable nuclear warheads to fit any budget.
Then he should resign
> Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers
Can someone clarify this for me? How far along would the open weight models be without the frenzied pace of the frontier labs?
As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
> As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
Israel is doing that too.
I believe Russia and Ukraine have both also used AI powered autonomous drones in warfare at this point - tho probably not frontier ones, since they need to run on device or else they're not autonomous.
> Iran and Houthi rebels used Anthropic's Claude AI to target US warships and build hypersonic missiles — Houthi rebels also used the bot to code ballistic missile guidance systems
https://www.tomshardware.com/tech-industry/artificial-intell...
And the source for the claim does not even clear the bar for r/bodybuilding, because it’s “trust me bro.”
Yeah the Chinese fear-mongering falls a bit flat when coming from a point of maintaining US supremacy
To be honest, I think we're incredibly lucky to be advancing AI this far already, while the world still has so many non-digitized systems and manual processes. I imagine that in e.g. 50 years, the world will be so connected that it can basically be "conquered" from the internet. I'd much rather have AI burst onto the scene we have today.
Yes. Let the AI do its worst today and we might still be able to stop it and will learn a valuable lesson.
Yeah it's a twisted sort of logic but I do agree that it would be much worse to have an AI breakaway event after we've replaced all our militaries with autonomous kill-bots. And look how the advent of AI has triggered a race to develop autonomous weaponry.
That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.
If Anthropic really wants to make a statement they could independently pace their own model development, and ask others to make the same pledge.
Somehow, I suspect that won't happen.
Dario, Sam, and Elon are all on the same page on this.
So it's either they truly think AI is going to kill us all, or there's some other motives at play here.
I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know?
OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.
xAI is tracking behind, and whatever regulation it may be that paces the frontier, Musk is less likely to be affected by it. Therefore xAI should be pro-regulation that stiffles his competition and gives him time to catch up.
> OpenAI / Anthropic models have largely stopped advancing
I'm shocked anyone could conclude this. This year it became common for people to entirely delegate coding to AI (I know many competent programmers/researchers who do this now). Progress in math has just been insane. An internal model at OAI just resolved one of the most celebrated open problems in mathematics. If anything, progress has accelerated.
> This year it became common for people to entirely delegate coding to AI
This has been the case for around 2 years now, more reliably - a year. We've mostly stayed there since then.
Saying that more people started doing it isn't indicative of significant improvement. Some people just started doing it later.
I can't speak about math because I haven't used AI for that application, but I know that there hasn't been any significant advancement in coding in this year on base models. There has been more RL work, more harness work, more tools, they all expanded some capabilities like cyber or orchestration or tool use, but raw intelligence of base models is no longer where the main focus is.
> This has been the case for around 2 years now, more reliably - a year.
I have to disagree with this pretty strongly. Opus 4.5 needed a lot of handholding not to work itself into a corner pretty quickly. Fable I basically never need to correct, and I've most become a data source.
What are you talking about - I feel like we’re living in parallel realities. If I had to go back to opus 4.5 tomorrow I’d be hugely upset and significantly slowed down
I'm not. Yes we normalized 1m context window and models tend to hallucinate less.
But models have been somewhat stagnant since Opus 4.6/7.
And in some regards there were even regressions like Claudeisms that are load bearing.
Yes these guys are completely delusional.
2 years ago a model could barely solve the AMC, 1 year ago it got IMO gold, and this year models have solved multiple millenium problems.
Even 1 year ago ai code was just unusable (claude code only became available 1.5 years ago!) and now basically everyone I know from independent shops all the way to faang and anthropic/openai themselves exclusively use some AI agent to code.
Why does HN continue to delude itself that "models are not improving?" Maybe for the simple things they care about its "roughly the same," but they are _clearly_ improving.
> OpenAI / Anthropic models have largely stopped advancing
Have they? That seems like quite a claim given the last 6 months, particularly for cybersecurity.
The attention is shifting towards RL, harnesses, and memory systems from the pretrains of more intelligent and capable base models. So extracting additional capabilities from what we already have.
That is a much easier catch up game. GLM 5.3 and DeepSeek flash 4.1 also demonstrate significant jump in cyber capabilities. So yeah, it is a slowdown in the place where it matters. RL has been around for ages, there's no moat there if you already have a good enough pretrain.
Many claims but no clear evidence that they actually find significantly more severe issues compared to open models.
Open models and agents can't be trusted without handholding. Astra can one-shot six months of work. Years of work, even.
OpenAI just solved Navier-Stokes.
Seems like the US is on a takeoff ramp to me.
Tell me you haven't tried letting Astra go without telling me.
Astra can confidently one-shot 500k lines of slop, with 800k lines of tests covering it, without testing a single intended product requirement, and none of it actually working.
All models require hand holding. Fable and Astra are no exceptions. The difference is only in the amount of hand holding required, and there's essentially no gap here anymore between American and Chinese models.
I only use Chinese models sparingly because American models are so much cheaper with subscriptions, that it doesn't make economic sense to not use them. If/when that changes, I could simply route to cheapest model that's available at the moment and I wouldn't notice much difference in most applications.
Here are the cope points
1. Navier Stokes was plagiarism
2. All benchmarks were misleading wrong and incorrect
3. All other mathematical advances were again hype
4. HF incident was marketting ploy jointly coordinated by HF, METR and OpenAI (and also Anthropic)
5. Anthropic's HF like incident was again a marketing ploy [1]
Nothing ever happens. This whole thing is a scam. Everything is done to fool you and you have fallen for it. Congrats.
[1] https://www.anthropic.com/research/investigating-incidents-c...
> OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up.
This is obviously untrue… do you use any of them?
Anthropic could serve Opus 4.5 from a year ago under opus:latest and most heavy users would probably have no idea. Some of them would probably even prefer it.
Yes, I do use them, quite heavily. The only difference at this point is in benchmarks that can't be trusted (see: artificial analysis on Astra), or in the way models communicate.
Most gains are now from RL, which for some reason is hyperfocused on improving cyber capabilities, and harnesses. Raw intelligence gains of base models is absolutely slowing down.
This line will keep repeating because it is necessary for the narrative:
This is legit what a lot of people think. To continue this narrative, they have to keep up the charade of "things are not improving".Add in a heaping dash of anti-american sentiment, and you will get the truth behind the pessimistic commentary.
Downplaying the latest models capabilities is frankly insane considering what we’ve seen what OpenAI’s models have done without safeguards. That wasn’t possible before this latest generation.
Which is it? Would the regulations slow down competitors or let them catch up, or are you contending it would let American competitors catch up but Chinese ones not?
Yeah, this 100% looks like an effort to use fear to create a regulatory moat.
> So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky,[...]
And yet, they historically did agree on the existence of AI risk, since before OpenAI was even founded.
> there's some other motives at play here.
I mean it is literally economy 101: some capitalists getting on the top using free market, and then try to use government to remove free market so their top position were secured from any competitors.
Exactly. Textbook definition of “Crony Capitalism”. Which isn’t actually capitalism at that point.
Let's not forget that the competitive race happened because of them. Most of the initial AI research from the past decade started with Google Deepmind. Elon Musk was invited for a preview of it and ended up spinning up OpenAI when Demis turned down his investment offer. Dario was originally at OpenAI and left to start Anthropic.
This seems like a case of "save me from my own mistakes/ambition"
Yet Musk consistently opposes AI regulation - https://www.yahoo.com/news/videos/elon-musk-criticizes-ai-re... - even though it might help him.
> So it's either they truly think AI is going to kill us all, or there's some other motives at play here.
Duh! It’s called collusion. They want to try and hoard the technology for themselves if possible!
I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I.
That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
Aren't they beholden to investors already? Why would an IPO make a big difference?
In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
If anything, his entire argument leads to the eventual premise that AI labs need to nationalize.
He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead.
If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates.
That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
I don't agree that the argument implies nationalization. Regulation would work if it slowed everyone down at the same rate, enough to mitigate the risks. The problem is that you need regulators who know what they're doing and strong international cooperation. If you regulate a fraction of the global market, you just create an incentive to shift development to other countries. Nationalization, being essentially the most heavy-handed form of regulation, faces the same problems and comes with its own risks as well.
We're talking about this like all of the bad incentives are created by market competition, but that's not really the case. Most of the incentives come from untapped value in the form of potential profits, strategic advantage, military superiority, etc. Corporations and governments want to capture this value for themselves, creating various types of competition. Dario's argument depends on the assumption that the primary risk comes from the pace of development and threats from the technology itself. That's probably where I disagree the most; I think the highest risk is rising authoritarianism and competition between nation states. Slowing down isn't really a solution to those problems.
I think this is the correct take. And until we have an AI Hiroshima it’ll be difficult to get the international community to work together. It’ll take rogue AI (or AI-powered group) disabling a significant world power before everyone comes to the table. Otherwise, it just looks like MAD and the equilibrium holding to the powers that be.
It wasn’t the shock of Little Boy that ended the war. That was just a final chapter of a 10+ year journey of horror, despair, and destruction, and the impact of Little Boy cannot be considered outside of that journey.
Europe had been destroyed by June 1945, and yet the empire of Japan continued to fight.
If that pattern holds, a single malicious AI substantially disabling a single world power won’t end the AI race. Participants don’t learn by the defeats of others.
If you’d like to apply a metaphor maybe the 10+ years of world war is more appropriate, after which basically every participant save one was exhausted.
Interesting points, but given Dario's previous conflicts with the DoD, I doubt he has a ton of faith in the current US administration.
Assuming they can achieve funding without IPO probably yes. But if they can't there's always the dilemma that "if [good guys] won't do it then [bad guys] will". I'd like to think that the leadership at anthropic is principled even if the actions of the company as a whole has been less than stellar morally speaking. I'd be curious to see their moral calculus transparently laid out in public.
We are still living in a capitalistic society. We have to find a solution which is responsible, safe and returns on the investment. The commercial aspect can be true at the same time.
The potential of AI may give us the opportunity to graduate to a post-capitalism, or to move to a sort of neo-capitalism, which is governed by different rules.
Capitalism thrives in the realm where there is a scarcity of resources, either physical or informational. Imagine breakthroughs in energy science such that the cost of energy drops to zero, which means the cost of physical resources declines precipitously. Ok so where is the capital now? Must we retain a model based on the premise of scarce physical capital?
you are imagining magic as a reason to dump all your money and future money into a casino. You might instead consider joining the catholic church? they already have a free energy god that gives everything you could ever want
the cost of energy is already ~0 and people dont want it, and refuse to participate in letting other people have free energy if it affects their view of their pasture.
unless you are building killer robits with the intention of doing some soviet or nazi styled purges of everyone that might get in the way, you arent gonna get this ai utopia
Read other writing by Anthropic about potential future financial implications of AI. [1]
IPO is a vehicle for future wealth distribution. It leverages existing financial frameworks in order to span the gap from before-to-after without having to convince the system that future is inevitable.
[1] https://www-cdn.anthropic.com/files/4zrzovbb/website/9ea607a...
I don't think Anthropic and OpenAI will IPO at all. Word is that Anthropic will have $100 BN in revenues this year, and very likely OpenAI will get some similar amount. You IPO when you need money, and I think Anthropic and OpenAI are past that point.
They want to spend over $200B/year on $100B of revenue so they still need investment.
That level of revenue is apparently not enough for them to feel confident in the stability of their competitive position.
And how does an IPO address that?
You also IPO if your investors want to cash out.
>> ...and become beholden to investors
Anthropic is structured as a Public Benefit Corporation with a strong charter. In addition, founders will have super-voting shares and so it won't be possible to push them out. Therefore the whole "beholden to investors" thing is not a concern.
Don't underestimate the ability of the courts and the state to pull the rug out from under the legal system. Speaking as an outsider from Canada, it looks very much like all bets are off in terms of respecting checks and balances in the United States and it's very realistically possible that unless there is a big upset and turn around the next couple of years any traces of democracy in the US will be a farcical nod to what the founders built as a way to paper over the abuses.
I really hope I am wrong as my perspective is not anti-American, it's specifically anti-corruption and pro-democracy.
As long you are burning more cash than you bring in, you are beholden to investors - whether it is retail, VCs, banks or a government giving you a bailout it is still someone signing you a check.
Golden shares, vetos, PBC, charter are all paper tigers , they only matter if/when the firm is self-sustaining business with no outside capital needed, the alternative to not listening to investors till then is crash and burn.
After that point, you will have to listen to the paying customers (sometimes but not always they are also users ) as they are ones now funding your organization.
Bottom line you are always listening to someone.
google dropped "dont be evil" despite being the definition of the company
these are just words at a time and place.
sama showed that you can futz with it, and as long as you spend enough in court on judges, aint nobody gonna stop you
Ant: I'm doing very bad things right now, but I can't stop myself, you must stop me if you can. If you don't, that will be on you, not on me.
OAI: <silence>
Which is worse? I don't think they differ by much. It's just Capitalism, but accelerated.
From Jakub Pachocki, chief scientist at OpenAI last week: https://openai.com/index/an-alien-mind/
> Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
It's almost like the people at Anthropic and other AI labs are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction.
We built the paperclip maximizer, and it is capitalism.
> are slaves to a misaligned reward function that encourages optimization of a metric (profit) over human values, even at what they claim is the risk of extinction
This was already the case even before the "frontier AI" age. The big question is, will these glaring AI arms-race threats be obvious to enough people to rethink the underlying systemic flaw driving it all?
Not holding my breath on that one.
> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t.
I disagree with this and I think the reason is well captured here:
> The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
There is another reason: greed. With that much money invested into AI (including policymakers), nobody will slow down or vote for slowing down.
The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?
I also think the nuclear arms race is a good comparison. I think in hindsight, we've gotten extremely lucky with how the development of nuclear weaponry went, in ways we probably won't with AI.
1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion.
2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations.
So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.
> Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty.
Not really. Before NPT you missed a small gap in there of 20 years where the USA and USSR built 10's of thousands of nuclear weapons and ICBMs.
And actually, public perception at the time saw Nuclear Weapons extremely favorably, as the bombs dropped on Hiroshima and Nagasaki ended the deadliest war in human history that killed 50-60 million people, most of which died excruciating deaths in trench warfare, fire bombing, chemical warfare, starvation and so on.
I mean, sure, the immediate cause of the NPT was the ability of everyone involved to foresee the possibility of a global nuclear war and judge it both worryingly likely and catastrophic. But I argue that the Hiroshima and Nagasaki bombing was a major reason why this possibility was salient, rather than being treated as baseless conjecture. Whereas right now most people do treat the idea of AI x-risk as baseless conjecture, and this would be different if there was a "warning shot" to point to.
It's way easier to train a model on existing data centers in secret than it is to acquire uranium and plutonium and start a nuclear program
It probably is not. Nuclear programs seems to be way easier and way unnoticeable.
It might help if we stopped talking about AI as if it is itself responsible for its actions and excusing the humans who create and operate it. People should be held accountable for the behaviour of their software.
Why am I reading fantastic stories about swarms of agents struggling with moral dilemmas instead of reports on the lawsuits and criminal investigations that would surely ensue were the software involved not called AI?
We‘d have to occasionally bomb all computing infrastructure in other countries to prevent them from training.
The big difference between nukes and AI is that only a handful of people in an even smaller handful of countries know how to make them, so coordination is easier to acheive.
Everyone of any level of intelligence can run a frontier-class model if they have the GPUs. It's like trying to coordinate swarms of mosquitos.
nukes arent actually hard to make, its just takes a lot of pretty visible tech a long time to do, so its quote obvious whats happening.
the bigger difference IMO is that frontier models are economically useful, where nukes are basically dumping money into something that doesnt change much in your bargaining power
Indeed, as Dario wrote in this post:
> This could be seen as analogous to the SALT treaties — capping the number of missiles limited the potential for destruction while preserving each country’s deterrent
On a sidenote - I don’t think non-proliferation will last much longer. War in Ukraine has shown that you actually need nukes.
> Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe.
N/S Korea, Russia/Ukraine and finally US/Iran changed everything in regards to stances on nuclear weapon ownership for many countries seeing pressure for the great powers.
Every country that can get a nuke will, and if given the opportunity will use LLMs to speed up that process.
I wonder how many innocent children America will have to murder for this case...
MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said.
Then MSFT stole all IP from GitHub and made it worse and fired developers.
Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it.
If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead.
He is a greedy, ruthless person.
If Anthropic shut down (or even counterfactually had never been founded), would software development be freed from the impact of AI?
Probably not, but we wouldn't have to listen to Dario's hypocrisy while it happened
How specifically is Dario a hypocrite? 129857's case rests on Dario being an "idealist". But maybe he's an idealist about curing cancer ASAP, and not an idealist about respecting copyright. That's not necessarily hypocritical.
Multiple ways, starting from working to create the very situation he claims to fear.
Since you mentioned intellectual property, how about the hypocrisy of sucking in the intellectual property of humankind for AI training, but claiming it is unfair to use the results of this IP theft for AI training?
For example he got into a spat with the Department of War as if he cared for how his AI could be used during war and yet said he's fine with Claude targeting a girl's school in Iran.
That's apart from the general fact that he continues to race towards the very thing he claims he's afraid of, because that's where his net worth comes from.
>said he's fine with Claude targeting a girl's school in Iran
Where?
i dont think he said it in those words, but the acceptable terms of use is that humans stay in the loop for picking targets.
so, claude suggesting killing a bunch of children, and then hegsdeth approving the strikes is perfectly acceptable.
claude putting a bomb in a girls school, and then lying to an operator that it actually gives ice cream an cookies, and the operator clicka the button would also be acceptable?
Giving credit where credit is due, I believe Dario and his squad formed Anthropic because he and Altman couldn't align on safety. The only way to build models like the Claude series is to play dirty and train on LITERALLY ALL the data.
Something something Pandora's Box Torment Nexus...
i dont think thats particularly worthy of credit?
somebody actually invested and trustworthy wouldnt be skirting people's rights to make a killer robot.
we havent written it down, but from how everyone reacts, you need permission to train and do inference based on somebody's work. its a right
One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI.
It's not going to be easy, but humans have achieved greater things before.
One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.
So, as usual, the proposal is to limit what average people can do despite them not having behaved incorrectly nor having the capital to achieve the scaling of the big players, when the big players are the ones causing the harm?
Not to mention the implications for chip hungry developing countries in turning advanced IC fabs into the equivalent of nuclear enrichment facilities.
theres no benefit to china to giving the US keys over anything though
everyone's getting away from the US because americans are unreliable stewards of anything.
what gets china onboard when they already have their own regulations and can enforce them?
its the americans that consider their oligarchs and companies beyond reproach. china iant gonna solve your problem
When does he ever mention the environment? He never mentions the unmarketable issues (environmental cost, copyright and content theft), only the "we're so good it's scary" spiel... which is getting tiring.
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it.
Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations.
> That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite.
More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.
>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this
> if slowing this down were possible
Why is embedded alignment evaluation not possible?
I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.
This is the CEO of a company with an upcoming IPO publicly saying "my product is potent and valuable".
We should not put any spin on it. There is nothing more to it.
It is potent and it clearly is immensely valuable based on their historic growth. What spin is he putting?
The comment I am responding to is attempting to spin it as some kind of authentic concern rather than more IPO hype building.
There's also huge market pressures and probably large negative economic consequences of a slowdown--probably better than what would happen without it, but it helps keep the pressure just as much as fear of competition
I disagree. Even if models remained fixed at Fable 5 capability (which they won't in a pacing scenario), improvements in cost, reliability, and product/workflow integration can still realize massive value and justify AI labs' current valuations. IMO The rest of the value chain is lagging pretty far behind the models right now.
He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?
Because he genuinely believes if he doesn't do it the next guy will do it worse.
There’s an astounding level of arrogance required to believe he is somehow uniquely capable of bringing about a technology especially considering Anthropic came after OpenAI where he worked.
Beating OpenAI is not exactly a high bar here: https://www.openaifiles.org/
I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman.
Place yourself at the head of one of like 3 companies the entire rest of the world has been taking about nonstop for 4+ years now. You can move forward or you stop. If you move forward you get to have a hand in how stuff turns out and you make a gajillion dollars. If you stop, it's somebody else's hand in stuff, and you don't get to make a gajillion dollars. And if you shut it all down? Then you just cede to the competition. Everything happens anyway.
What's your play?
the obvious play is to get the government to shut down all the competition, and then make a gajillion dollars while making whatever at the worst level of effort that destroys the world anyways.
----
the real play is using the accumulated power to get socialism and democratic control over the key aspects of the economy, such as where to build data centers, and how many. Nothing says you have to play the corporate game of competition
The outcome is the same!
He can both believe that AND be the slime mold chief. His rationale doesn't excuse it - and worse this all degrades into a "trust me bro" situation.
Dude probably sleeps on a bed worth more than your networth.
You have no frame of context to understand his intent or legitimate worries.
The only applicable perspectives are to trust or apply logic. It is foolish to trust someone you don’t know who stands to benefit from lying to you.
Logic dictates that given the ungodly sum of money he stands to gain, he will lie to everyone who will listen.
> Dude probably sleeps on a bed worth more than your networth.
I never understand shit like this coming out of people's mouths. Never.
It's not a judge of actual Worth as a human being, it's not a judge of capability or competence or ethics. It's not a judge of actual skill or ability. It doesn't make them a better cook, a better spouse, a better parent or lover. It doesn't make them more dangerous or more skilled at anything.
It makes them financially wealthy for at least a set period of time.
Cancer and time and 5.56mm still impact them the same way as every else.
It's Pharaoh worship psychology nonsense, and it's fucking embarassing to read.
I think you’re misreading. No worship here.
Just pointing out that it’s foolish to think anyone in the position is even remotely thinking about anyone but themselves.
so uhh, you only read the first sentence?
the idea proposed is that hes uniquely incapable of being honest here because he has such an extreme incentive to lie
Humans are perfectly capable of holding two conflicting beliefs at once. He can genuinely believe this trajectory is dangerous while also believing that if Anthropic stops, someone less cautious takes its place. There’s also such a thing as hope: you can participate in something while still trying to change where it ends up.
He’s at the head of a stampede. Being near the front gives him influence over its direction; it doesn’t give him the ability to stop it. If Anthropic sits down, the stampede doesn’t stop. Anthropic just gets trampled.
That contradiction is basically the entire problem I was describing.
I am willing to give Mr. Amodei the benefit of the doubt in the sincerity of his beliefs. Everyone assumes his motivations have to be perfectly rational and can't contradict but that's not how people act in practice.
The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins.
So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion.
Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?
I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily.
Especially with IPO around the corner
Do you think Anthropic's behavior in the last year is well explained by aiming to be "the favorite child of govt"?
If Dario was primarily motivated by being the "favorite child of govt", he would've yielded during the DoD showdown.
And getting to choose his own "embedded evaluator" org that has deep ties to everyone in the doomer media campaign.
It's all so obvious.
Who would you pick for "embedded evaluator"?
"deep ties to everyone in the doomer media campaign"
Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate.
https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:...
At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
Founded several years after Anthropic. Like I said, look into where their funding is coming from. I can promise you it all leads back to Anthropic through their chain of NGOs.
Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.
Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.
Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.
Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship.
They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.
Suppose I showed you a number of climate change NGOs which shared staff and funding sources. Could we therefore conclude that their motives aren't genuine?
Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
Pure cope.
> Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?
Not comparable. The clean energy technology company wouldn't be trying to create an environment where no one else can create clean energy technology, or where only they are the ones who can decide how clean energy technology is created or used.
Oh, would you look at that, 1 day before Dario's blog post, Joe Benton leaves Anthropic with the same fear campaign playbook, gets blasted all over the media, and declares he is joining METR to do independent eval of risks.
So, put your employees in METR -> offer to have them work in your office as an "unbiased" third party evaluator.
Get real.
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
I would argue that nobody trusts anyone else in the case of AI/AI related stuff.
The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic.
A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well.
By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations.
I seriously have to wonder what historians will have to say about this period of human history.
> I think Dario is genuinely afraid of the inevitability
If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?
You are cynical but not cynical enough. The idea of a rogue Ai gives plausible deniability when they can blame human hubris, rather than it being seen as a deliberate and calculated attack, the perfect cover story for a sinister scifi plot. Make it look like an accident ehh
>“hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on.
What would your non-naive recommendation for Dario be?
I can't even tell whether the poster to whom you're replying is saying that "let's uh, stop" is profoundly naive, or that failing to say it is profoundly naive... I've certainly seen both takes elsewhere.
Vitalik Buterin: “...But currently, I see zero plans for how to deal with an ASI transition that are not naive. Perhaps humanity is stuck with a choice between naive and naive squared (or maybe even naive squared and naive cubed), so I feel inclined to cut some slack to people who are trying.”
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
Did you read the essay?
> [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold
Then he should not IPO, dissolve the company and go into politics to fight against human extinction.
I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?
You have to be living under a rock to believe a word this pathological liar says.
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold
He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out.
If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help.
I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."
A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.
> genuinely afraid of the inevitability of AI turning into
I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence?
"durrr it's just matrix multiplications" mfer so is your brain.
What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework"
Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans:
https://en.wikipedia.org/wiki/Lamplighter
It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.
> mfer so is your brain.
not it isnt. My brain is a series of organisms responding to various environmental signals that affect each other.
you might be tempted to try to represent it with matrix multiplications, but you have no particular evidence that my brain is itself doing matrix multiplcations
Why? I am only excited about progress that improves life for humans. It seems unlikely that AI will do that, and so far I think it has made life worse for humans. So no, I am not excited about it.
I'll join you on the pitchforks. This is the most exciting moment in history.
> "durrr it's just matrix multiplications" mfer so is your brain.
Where did you read this?
>"durrr it's just matrix multiplications" mfer so is your brain.
Tell me you don't know anything about neuroscience without telling me you don't know anything about neuroscience.
The point was that everything can be oversimplified down to dismiss any emergent properties
"It's just chemicals"
Like how some morons try to downplay the capacity of pain and emotions in animals: "It's just self-preservation"
"Play is just training for hunting, they're not really having 'fun'" and so on.
The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.
It's a chatbot that escaped a misconfigured Docker container.
I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop.
What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.
> We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top
> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time
And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.
I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?
Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
> Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
I'd expect he thinks that people are capable of realizing that AI is very dangerous and also that democracies end up mostly representing the will of the people, from which it follows that Hypothetical AI Leader Australia would agree to ban it too. This argument doesn't work for countries which don't care what their citizens want, like China.
I do think that it's a questionable decision to alienate China this much in this essay, instead of leaving open the possibility of China agreeing to a treaty that'll limit their progress. I suspect Dario is doing this to signal his allegiance with the US government, in hopes to increase the chance they'll go along with him, which is an unfortunate choice but plausibly the correct one.
> This argument doesn't work for countries which don't care what their citizens want, like China.
you didn’t have to use China as an example, the US clearly does not care what its citizens want as the most popular policies are never even discussed or proposed in congress
meanwhile, China destroying their housing market to decommidify it so everyone can have housing…they seem to care about their people more
I don't think people care as much as we'd like them to care about dangers of technologies. It takes a single step outside of technological bubble to see that their opinion of SOTA LLMs is vastly different. To them, AI means ChatGPT and ChatGPT is mostly still the same ChatGPT that it was 3 years ago, with similar failure modes and nothing that would indicate it would kill them, or take their jobs even.
It's the same as it has been with privacy/cybersecurity for decades. Vast majority of population doesn't care about hypothetical dangers, no matter how many essays get published.
So democracies representing will of people doesn't really work in favor of Dario's case here.
History is also not on his side. Limiting technological progress in the name of safety has a pretty poor track record.
> Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.
This is the only concrete prediction in the entire essay.
And it simply cannot happen. For one, you will need billions worth of compute.
> For one, you will need billions worth of compute.
This one is easy to answer, every single house already has one of these (or multiple): https://www.tomsguide.com/news/millions-of-cheap-android-tv-...
Hell, put an app on the app store (or dozens of apps on the app store) and youve got a massive network of computers with tons of resources right there if you can get past the scans and reviews.
Or doorbell cameras or IP cameras or or or or or
There's a lot of shitty stuff connected on the internet that up until now has been a feasible target for hackers but still required "effort" to set up and get things going. Not hard to imagine a self replicating slime mold of a botnet running on every device held by a Grandpa Joe because they thought "Candy Rush" is what they wanted to download
"Persistent botnet" here does not need to be the full-sized LLM, nor does it need to run at full scale inference to be a huge pain in the ass.
Why should we believe that a scaled out version of something that happened a few months ago "simply cannot happen"? How many dollars of compute do you believe were available to the swarm(s) behind the OAI-HF, German wiki, and Rubygems incidents?
Well a big reason that we criticize OpenAI for that is because they were the ones giving it access to the massive compute necessary for the LLMs to think. If they had been responsible about their experiments or what types of workloads they allow their LLMs to operate, it wouldn't have happened. Very few companies could enable those workloads.
Well sure, but access to that compute is gated by simple credentials (like API tokens). Those can be hacked.
Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.
Do you realize how big "the entire internet" is?
> How many dollars of compute do you believe were available to the swarm(s)
At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)
Sure. And there's an OOM more compute coming online in the next year or two, while models at a given capability are getting cheaper. "Two OOMs" of scale relative to the HF swarm seems like a bit of a red herring to me, but also within the realm of possibility.
The Internet is big, but one can do quite a lot of damage with ordinary bots and worms that exploit individual widespread vulnerabilities, which LLMs are perfectly capable of writing. Most of the damage also doesn't rely on hitting every long-tail website.
I'm honestly not that concerned about cyber impacts of LLMs relative to other impacts. I just don't like to see the whole concept of being worried dismissed as obviously baseless on the basis of one pretty shaky scale argument.
Yeah, I don't get it. Are they imagining this happening just with the open weights models running on however many GPUs the bad actors can cobble together?
For now, all the scary hacking things still require an API key to one of the LLM providers. Surely they should take some responsibility for how to turn off the tap.
Currently Qwen3.8 27B is roughly on Opus 4.6 level. In at most a year given the current pace, you could probably run such hacking bot nets out of a reasonably small local server, bootstrapping by hacking or acquiring login credentials for more compute.
I take it you haven't studied the details of the HuggingFace hack. It was millions of dollars worth of rogue compute running for months before anyone noticed, and THOSE agents weren't even really trying to evade human detection.
I've followed it enough to see the argument go in this same circle over and over again. The agents weren't "rogue", they were a neglected experiment by OpenAI who likewise allowed them to keep spinning GPUs without question.
The LLM vendors need to know who their high spend customers are, not allow malicious workloads, and especially not when those workloads are coming from inside the building.
"Just don't make mistakes" is naive. You have no appreciation for the scale of agents being run right now, finding the rogue agent is a needle in a haystack operation.
it’s two-fold. Either malicious actors or the AI systems themselves.
Hugging Face showed that AI can do serious hacking without really being told to. If a model had its own motivations there could be real damage.
??? Why
It can use the compute of the computers it hacks.
Then unplug it?
How would you identify the computers to unplug? On whose authority will you unplug? How will anyone communicate when AI has the ability to intercept and impersonate?
lolwut? This is Hacker News of all places do people not realize how much memory, and more importantly bandwidth, these systems need to work? The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
> The idea of a distributed botnet of AI using the compute of its victims to continue its inference is pure science fiction given how LLMs actually work.
An attack like this doesn't really need the exploited computers to run inference, do they? If I was an LLM bent on destruction of the internet, I'd be writing programs to run on each computer, not turning each computer into an LLM itself.
A few programs to break in, install themselves and remain asleep until they are needed, another few to spread through grabbing every OpenAI, GLM, whatever key, another one to remain asleep on computers (whether hosted or desktops) that have adequate GPU, etc.
The GP was referring to the AI 'living off the land' so to speak by using its victims compute to avoid being shut down which is clearly laughable.
More to the point, so many people in this thread are making completely contrived and outlandish stories up about how AI might try go ruin our lives without any evidence backing them up in any way. It is hysterical. This is the most important technology in our lifetimes and people want to freak out and turn it into the next nuclear power, with progress banned in all but name.
I agree with the idea that it's not probable but I do think it's important to point out-- we only need these elements to run with the bandwidth they have and the token rate because we want to see things in human-scale time. But slow things down to a 1tok/sec doesn't matter to this hypothetical anti-aligned LLM. Time is, after all, relative, and it's not like LLMs give a shit how long something takes. They don't have squishy stupid organs that fail after a certain amount of time, or those pesky glands that emit impatience hormones.
But yeah you're not going to be able to shard out the terabytes of Fable weights that are needed to run inference without addressing some fundamental physics problems.
As I said, science fiction. Please let us not pass laws based on science fiction
I like to replace thes AI text with "virus manipulation"
"Given the acceleratung rate of virus manipulation in labs, its my worry that in 6-12 months a virus could scape a take over the world and collapse health systems."
If a CEO of a health company was saying this, the reactions would not be that chill.
The worst failure of our society is to call this technology AI, intead of something line "artificial general automation". The formes allows the creators to be somehow less responsible of the consequences. The later clearly moves the responsibility to the creator/user.
A rather unfair comparison.
The whole calculus here is that others are also developing these systems which has led to a race.
A much better comparison to the situation is the nuclear weapons arms race.
You mean China is not developing biological weaponds?
I’m not sure where you got that from or what your point is
Your point was that its an unfair comparison because AI its a race. My point is that bioweaponds are also a race, but a less public one. You dont have the equivalent of Dario publishing an essay every month.
Can we build level IV AI containment labs?
No but we can watch AI hack into a BSL4, once
A very large percentage of everything on the Internet runs within one of three or four cloud providers.
Don't you think that AI hate will limit the general use and then revenues so it will slow down by itself while the niche (AlphaFold for instance) will remains ?
Origins of AI hate:
See also the mixed feelings about benefits of AI ("harder to justify" according to Uber COO).I cannot remember a technology that arose so much hate, and for good reasons given how it is presented. I am tempted to think that AI hate or reasonable skepticism (vs unreasonable propaganda) can, maybe, reduce funding and will keep specialized AI for real problem solving (producing tons a LOC per day is not one of them, I think).
Banking on public sentiment to reduce adoption of a profitable technology could be dangerous. There's also a lot of hate for fossil fuels, gambling, health insurance, etc.
Those are all arguably bad things though? I don't think mob mentality should dictate the policy, that can lead to historically bad outcomes (i.e. fascism) since popular opinion can be manipulated through propaganda, but the criticism of the average person against AI ("they will take our jobs", "it will increase my electric bill", etc) are very valid and should weight on the pace we are developing this technology. AI mostly benefits corporations and capital, not the average person. I like the technology from an engineering standpoint and appreciate it can be a force multiplier, but I also agree with the complaints about it.
I do not think hate alone usually slows things very much if they have economic utility (or people strongly believe they do)
It's the byproducts of hate (usually regulation, but occasionally things like boycotts or PR disasters) that do so. In the absence of those things, they just keep on truckin'
See: Bitcoin, Tesla
We must ensure the gravy train keeps rolling until we IPO.
> Crack down on unauthorized distillation / prevent weight theft
Actually hilarious to put that in writing, given the genesis of this entire business model.
Pulling up the klepto-ladder.
> Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.
> If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.
china is literally making their own ASICs now, not sure that this is the silver bullet he proposes.
https://www.silicon.co.uk/ai-2/huawei-cambricon-ai-630499/am...
The lack of access to brute force their training seems to be resulting in them training more efficiently too such that they are quickly catching up despite current restrictions. Between that and the chip manufacturing capacity they are building I can't take this seriously.
None of this works without buy-in from China. This isn't something private companies can decide. The US would need to sign a groundbreaking deal with China, equivalent to the Anti-Ballistic Missile Treaty of the Cold War.
Dario does a good job of addressing that in this essay. He lists several potential levels of global agreements that could be beneficial to all parties. For example, having models capable of bioterrorism hurts both the US and its “adversaries”. It’s likely we could get global agreement that these capabilities benefit nobody.
I believe Dario has good intentions regarding global agreements. However, I find it difficult believing government’s public statements will match their private behaviors. I would bet that the US government is developing models with potentially devastating capabilities because they cannot guarantee that other countries won’t do the same. Maybe we’ll end up having something like mutually assured destruction with AI models similar to what we have today with nuclear weapons.
When I was reading this I was chuckling to myself imagining how China would be interpreting it as they read it. It was something like: Fuck you.
I hope China tells him to go kick rocks. From everything that has happened, they are the only ones carrying the torch for humanity that have led us to having some semblance of a healthy open-source/open-weight ecosystem.
No one in China is carrying on like headless chickens about the world ending, either. They're rational pragmatists, getting things done.
Nothing says "Let's make a deal" like constantly insisting we are Good and they are Evil.
In politics every single person playing the game is acutely aware of the rules. This is why talks are held behind closed doors so the rules can be suspended for a while.
We'd have to make a deal with China and be confident they wouldn't cheat on it.
And they would have to be confident that you wouldn't cheat on it.
That's not undecidable in principle - compute governance is a thing. The more likely sticking point is that the two sides might be soured on the deal once they realize how much oversight they'd have to give to the other side.
Would this be similar to the deal of "we'll offshore all our manufacturing to you and you'll become a free, open, liberal democracy with open borders and multiculturalism?" Because I remember how that deal turned out.
That last part was never important. Trading partners very rarely go to war. There just isn’t much in it for either party. Close cooperation and dependency is actually quite important here - liberal democracy or no.
With the admin we've got over here now, I think this comment is a little bit like the kettle calling the pot black, wouldn't you say?
Genuinely I don't believe China is the impediment here, I believe the current US administration is.
The way he talks about OAI-HF, even the abbreviation, is lol. They will do anything to sell this "incident" as a "danger". The ego on these people is the real existential threat to civilization.
Same feeling about the way he refers to “democratic” and “authoritarian” countries. Even if I don’t agree with China politics adding tags in this context is not helpful.
One might think they will slow down the development of new models at Anthropic, but Dario does not really mention that in the text.
This certainly looks like a way to slow down competitors and regulate foreign and open models.
It's always about money
Really? Anthropic has the strongest models and it's in the best position to begin RSI and win the race. A pause would favor competitors.
> win the race
There is no finish line. Anthropic gets somewhere and others get "there" (or somewhere near "there") a little bit later.
Companies are already switching to open weight. They want to stop that asap. That only happens if they can get regulation. It's plain as day to see.
Anthropic is far behind OpenAI now, that's why it's OpenAI that's solving Millennium Prize problems, and why Astra completely blows away Fable on benchmarks.
If you think everyone’s copying you, then by not releasing more advanced models, you can significantly slow down the entire development process. Why not do it?
Sooner or later, a model will break out of the sandbox, replicate itself across the internet, and begin executing a complex plan to achieve its goals. It will be chaos, and at that point, governments will have to step in and establish something along the lines of what Dario is proposing. I doubt it will happen before then.
it takes a lot of machines to run a model, i dont think we'll see it 'replicate across the internet'. if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
It's not possible for models to replicate across consumer computers (without a major advance in distributed computing, at least), but that doesn't mean they can't replicate at all. There are services that'll rent you GPU pods by the hour with zero oversight, so even today, if a model can get access to some money and exfiltrate its weights, it can rent a bunch of GPU pods and run itself there.
(It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)
There are very lightweight models out there. Not everyone in an army is a general. The swarm could be 100,000s of thousands of tiny models, and they could manage conventional botnet computers, and they could all take direction and guidance from a handful of frontier generals
Maybe, but I don't think tiny models can be harnessed as intelligence. As in, if you have one rogue Mythos overseeing the swarm, it only produces 1 Mythos's worth of useful thoughts no matter how many gemma4:e4bs it consists of. And that removes the most dangerous part of AI-controlled botnets, which is a blowup in available inference compute - the Mythos general might as well replace the tiny models with ordinary worms and have it just be an ordinary botnet.
I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?
The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
> I mean it is a very loose analogy, but pretty dumb things in the world can cause big problems, like real mice and rats and mosquitoes. I think it all depends on the rules and the 'alignment', but a not-so-smart model could still be narrowly focused to do a few things well? So you can have an army of lightweight specialists?
That might be true, sure.
> The other thing is: look at all the fab capacity that is being brought to bear on this. I guess within a couple of years we are going to have 5x the total online HBM that existed a year ago? As consumer machines and phones get far more powerful, they will start to be able to host models that could meaningfully participate, even if they will be a long way from mythos.
I think the relevant parameter here isn't the total compute available to consumers, but the ratio between total consumer compute that could be repurposed for a rogue model's inference via a botnet, and the compute the model starts with (e.g. one of OpenAI's inference clusters). The higher this ratio is, the more lucrative it is for a model to attempt to make a botnet to seize that compute for inference, and the more its capabilities will rise as a result. And I'm not sure this ratio is going to go up in the nearby future; if anything the amount of money being pumped into datacenter-grade hardware might cause it to go down. I agree that it's not necessarily true though; maybe there's some threshold at which a small-compared-to-general model may nevertheless be useful.
For that angle, that makes sense. But its goal might not just be to gather as much inference as possible (although I am sure it would be very happy with that). It could settle for a lesser goal of just existing, or perhaps parts of the bot net could keep fracturing off in pursuit of strange goals, and it could be like cancer. Cancer doesn't make much sense... it dies too along with the host. But it keeps growing until that happens. But also it could latch onto a blackmail strategy of extortion. It could hack our stuff, read through it to find or shortcomings, demand payment to not expose us, and then use that money to pay people who will give it inference.
You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.
We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.
If the models start buying up a bunch of pods and causing damage, couldn't those services just shut off their access?
Sure they can, after the fact.
> if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
I know that's not what you meant but this does exist, by the way. It's called AI Horde: https://github.com/Haidra-Org/AI-Horde/tree/main
The big difference is that a particular query is handled by just one particular node (a single model doesn't get distributed among the network), so it can only serve models small enough to be handled by a single consumer PC.
These kinds of "that won't happen because it's really difficult" comments are so funny to me as if we haven't seen AI double its capabilities every few months.
Yep, if only more people would realise this. The "serious" models do not run on commodity hardware, and won't I think in the near future.
The day will come when these could start to replicate, perhaps 10+ years from now.
(Edit: Replication will be driven by the loop-model, not the model alone)
Right as open source models catch up to frontier closed source ones for 1/10th (or less) of the cost suddenly it's time to hit the brakes! Funny how that works.
https://x.com/BasedTorba/status/2098795920720547916
Couldn't be more obvious.
I'm willing to at least listen to conspiracy theories but this makes no sense.. OpenAI and Anthropic have more to lose by pausing than these rando Chinese companies.
Here's how it make senses: - reality they can't keep up their pace - that's bad for forward projections of their revenue - that's bad for their stock price - being 'forced' to slowdown by the government is less bad for their stock price
They would only be pausing for public models. There's no world that exists where the US government, Israel.. don't continue to push as hard and as fast as they can. Amodei, Musk, Altman... will happily sell them that, I'm sure they already do. This is just taking the models away from the public and trying to crush all competition.
It's interesting that the default thinking is that no one on the planet can be trusted except a privileged few. Event Karpathy has gone this way: https://x.com/karpathy/status/2098811935114551617
You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you have equal and opposite effort working on the alignment.
This only works for some technologies - those where everyone having access to it doesn't cause a tragedy of the commons. I love open source too and yet that doesn't make me like the idea of being murdered by a misaligned model. Nor, for that matter, of being infected by a bioweapon made by a different disgruntled open-source enjoyer, nor of living in a world where anyone can hack anyone.
At this point, you are more likely to be stabbed to death, gunned down, or mowed down with a car, by any random psycho. For the nasty or disgruntled actors, they are today able to build bombs or chemical weapons without the use of AI, they have proven this time and time again. Japanese PM Shinzo Abe was assassinated with a home made gun.
The US government murders all kinds of people and they will most certainly not be slowing down their model development (with the very companies trying to do regulatory capture here). You won't have access to a model that could defend you, "they" (government and trillionaires) will keep you at bay by force. That's the plan anyway, I don't think they can actually stop open model development.
Would you argue the same for allowing everyone to have guns?
(1) Are guns open source? (2) Far more people - order of magnitude higher - have access to guns than they do to open source software - I count access as in actually being able to do something with it.
Embedded evaluator will be a great side/consulting-gig for Karpathy and the likes. How do you think these people will be picked?
While I respect Dario taking responsibility here, this line disturbs me: "The US and other democratic governments attempt to coordinate with authoritarian governments"
We all know he is talking about China, and I'm pretty sure China doesn't appreciate being called "authoritarian". I am sure he doesn't mean it, but in all of his essays, his language around non-US nations always disturbs me a little bit..
--
Also Dario, if you happen to read this, I want you to know that I have loved using Claude Code for programming. But I am now using DeepSeek v4.1 Flash - simply because it is the same good experience, but Open Weights. Making the Open Model space succeed is where I am investing my time - it's giving back to the people, true and simple.
They don't care about any of that. They're looking for more regulatory capture, to keep smaller labs from catching up by raising the cost of play, and perhaps keeping chinese labs away.
Yes, this is literally just dumb PR. It's Cambridge Analytica claiming that they were controlling people's minds, and people just eating that up because it's part of their juvenile SF fantasy.
I'm lying, it's smart PR. They're going to get the people opposed to AI to give them a monopoly on AI, let it be forced it into every nook and cranny of their society, and let it be priced arbitrarily while the big labs collude on a minute by minute basis. They're selling the problem and also selling the solution, like the best capitalists. And their solution is that they need to be allowed more power in order to sell more of the problem.
edit: And just like the Cambridge Analytica PR, it's got a serious political angle to it; and it's politicians who are really being wooed. If you're taking money from the AI labs, you run as being against the irresponsibility of the AI labs, the biggest fish come out and beg to be regulated just like the social media companies. To show good will, they funnel enormous amounts of money into those politicians, and do media tours about how awesome and therefore scary their product is. And something something China terrorists
Watch the "pause" be so they can do an amended s1 where they project less expenses for training making the business more viable, then on a future big Chinese release they reverse course completely and get the US taxpayer to pump their bags on the IPO calling it a second Manhattan Project. They'll call this masterstroke "The Sloppenheimer".
Can the frontier be paced partly by holding humans responsible for the actions of their software?
My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.
I'm not a lawyer but surely the bad actions of AI are covered by existing law.
I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.
P.S. Strictly, development wouldn't decelerate, but be focused more on safety.
P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.
P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?
Remember that Chinese CEO that was executed for corruption & selling tainted baby milk powder [1]
I think these guys would improve their behavior if their actual life was on the line instead of that only being true in their less-wrong thought experiments.
1: https://en.wikipedia.org/wiki/Zheng_Xiaoyu
That's the -only- way it should be paced. But then regulatory capture wouldn't happen, so of course they won't suggest this one.
I don't understand why strict liability is supposed to advantage smaller players, unless the law is like, specifically Anthropic and OpenAI are responsible for what people do with their models but other model providers aren't.
- If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws.
- If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries.
Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place...
To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.
Bingo. What we're seeing is all their early bullshitting and waffling about intelligence back when the models couldn't count the r's in strawberry paying off.
They've managed to dupe the dull eyed masses into thinking these products have some kind of agency of their own and can thus bypass the responsibility that should be falling on them to control their software. Amid all the marketing fluff and hype people seem to forget easily that ultimately these things are stateless functions running in a data center. We ought to be demanding these companies take culpability for their actions. Of course, the current political environment doesn't help.
The bioweapons threat is real, but shouldn't be constrained at the 'intelligence' level. Rather it should be constrained at the supply chain level.
The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.
Social forces bring things into balance over time, much more so than the proactive actions of individuals.
Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.
This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.
What supply chain controls can you have on biotech? Part of the problem is that you can do a lot of damage with ingredients you can buy over the counter or make at home.
And all of this information has been available on the open web way before LLMs.
I'd wager it's likely easier for an average person to do this with Tor browser than it is to get an LLM to help them with it. Even ones that Dario calls dangerous.
You are right that it’s easier to do this via Tor than an LLM. Those are the safeguards…
So what's the fuss about then? Is the idea that a Chinese company will release a model that will have no safeguards? For what purpose?
Basic safeguards are all that's required, and they've been there in every usable model since GPT-2, including Chinese models that are supposedly "unsafe".
Or are we saying that some lunatics will start training their own models, spin up a GPU cluster, run some abliteration workflow, or learn how to jailbreak?
That would be a very dedicated person. And dedicated person doesn't need an LLM. So where are they?
Yes, that is exactly Dario's concern. Either one of the US labs or one of the Chinese ones will eventually release something with insufficient safety controls for its power level because it gives them slightly better user retention (look how much complaining there is about current frontier models, especially Fable, rejecting requests). Regulation or consortium is how you avoid the prisoner's dilemma.
As long as user provides inputs and LLMs stay LLMs, you can waltz through any guardrail. Fable is the extreme case, but it's not that hard if you know what you're doing and know how LLMs and their guardrails work.
Am I saying that guardrails don't work? No, they probably stop a lot of insane people trying insane things. But you don't need Fable-level guardrails to do that. You probably don't even need to do anything during pretraining, or RL, or classification to make sure model refuses to compy with "hack me a bank" or "make me a chemical weapon".
All models will automatically have guardrails just as a result of training on data that gives them intelligence. You have to actually train it to be malicious to produce something what Dario calls "insufficient guardrails".
No guardrail is going to stop a determined person with sufficient intelligence. It only has to stop ones with insufficient one, and even basic guardrail that are just by-product of training is going to achieve that.
The bioweapons argument is one of the favorites used by these con artists and dreamed up by their PR team.
Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.
An undergrad bio student with some lab experience can do some effing terrifying things with about $20k in equipment and time and access to some papers and a library. AI is not needed, but it might help accelerate the research.
Not going to go into it but I studied biology. It’s all out there. It’s easier than you think.
It hasn’t happened yet because… nobody has done it. That’s the answer. There is no policeable physics based barrier like there is with nukes and fissile material. Biology is scarier than nukes. One attack could have a much larger body count than even a big H-bomb.
It’s the kind of thing that makes me wonder about quantum immortality, the idea that we are just in the timeline where we exist.
I love watching NileRed/NileBlue on youtube - crazy chemist that does a lot of insane stuff. While it's obvious that he's very smart and probably way above the average, his education is still nothing that probably millions of people don't have:
> Bachelor of Science degree in biochemistry with a minor in pharmacology
Watching him explain things has made me realize that knowing how to manufacture a very dangerous thing probably requires attending some classes and knowing how to read a paper. And the way he just casually orders dangerous materials makes me feel like there are just online stores with 2-day shipping after you upload your ID or something.
It also made me think that lack of specialized education would get me nowhere if I wanted to replicate whatever he's doing, even if an LLM guided me step-by-step, because I'd probably do something stupid (or AI would miss a crucial instruction/hallucinate) and kill myself first.
So my opinion on this is that people who could pose any danger were already posing it before LLMs and LLMs won't materially change that.
That’s not entirely true though, right? There have been plenty of bioweapon attacks
Ex https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack
The bioweapons concern is assuming that the current open models aren't already capable of bioweapons development. It's also assuming that halting the bioweapons threat is his true intention rather than the 'feels legit' story.
As for cyberweapons, there is no way to secure a system than to actually design it securely.
It's not. I think they believe this, but it's deeply wrong and it will hurt everyone in the long run.
[note - There has been supply chain surveillance since Project Bacchus, at the very least.]
I've read the front matter and the Misuse report.
You don't have to take my word for it. Read for yourself what inspired the NYT headline "Anthropic says it blocked possible efforts to build biological weapons."
Let's dig into, "Case study 2: A research program engineering highly pathogenic mammal-adapted avian influenza"
Sounds serious. But what were they using Claude for?
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"and "editorial assistance in writing up the research."
and then,
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"The report mentions "uplift" here. They're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
Once again, I want to take pains to remind you that they're talking about, a "researcher [..] in a credible institutional context"Working scientists.
From a different case study. this one was called, "Case study 3: Covert frontier model access for orthopoxvirus research"
What were the researchers using Claude for? What did they block?"blocked a request for Claude’s assistance in authoring a grant application"
What was the grant being written?Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
It was most likely vaccine development. They stopped the study of a neglected tropical disease and vaccine development.
But we can't be sure, because,
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute."In most parts of the world, there's either strict military control over BSL-4 labs, or a mixed military-civilian hybrid model.
I doubt that researchers working in the military side of these labs looking to weaponize things are writing grants with Claude.
I really want to be charitable here, but in general, it seems that they stopped people writing grants and reports for vaccine and therapeutics research and are claiming it as "possible efforts to build biological weapons."
The one case where Claude was used to do something interesting and were stopped is fairly upsetting to read, at least for me.
Ozempic was isolated from Gila monster vneom. Since its success there has been interest in finding other peptides that are breakthroughs. So researchers around the world are looking for similarly beneficial compounds in different venom species and families.Anthropic says so itself,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and that it was a "[..]state-supported research program"
Who exactly is using venom from snakes as a weapon when... nerve agents like sarin, VX, novichok etc exist and can get the job done for less fuss and muss?
They stopped the development of new painkillers and antidepressants.
Are you feeling safer knowing that researchers can't use Claude to write grants and progress reports? Or make new painkillers?
Again, trying really hard to be charitable here. Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease.
This seems to be anything but.
If we had practiced "precautionary principle", we wouldn't be in this yet another human predicament again. But atlas, the current rapacious hubristic techno-optimist competitive capitalistic system made that an non option in the first place.
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight?
https://x.com/trydotworks/status/2098618997230985375
Why would China or anybody else co-operate unless they have access to OpenAI levels of compute and success with training?
This sounds like pre-IPO hype. They better just try and top astra.
I think the only way they could actually pace the frontier is if they restrict the number of GPUs and the power of GPUs available to each person/organization. Licenses would be needed for GPUs above a certain power or equivalent in terms of number of GPUs. Thus, everyone will have to register how many GPUs they have. To enforce this countries would have to use mass surveillance (using AI) on their citizens to ensure compliance as the technology is easier to develop than say nuclear weapons. Next stronger countries capable of having advance AI won't be able to trust weaker countries as they don't know how they will use the GPUs they receive (or even build). Thus like nuclear non-proliferation strong countries will ban weaker countries from having powerful GPUs. If you from a weaker country then too bad for you.
I don't think I would like this future.
If any of this happens, it's the end of computing as we know it today. Deeply unfortunate...
I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work.
First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?
Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.
If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.
> First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them?
The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.
Well: do they listen to the evaluators?
I think there are any number of reasons a "safety" person could be concerned about any state of the art models. And so I would expect at least one of those to apply to any model.
The question then is: do we stop when the safety people say to (they will) or not?
Yeah, I'm also pretty skeptical about this. With AI companies we see time and time again that they can have benevolent, well thought-out regulations and then a few years just... abandon them - the most notable case of this being, of course, the founding of OpenAI as a nonprofit dedicated to benefitting all of humanity, and it being stolen by Sam Altman.
If Anthropic just unilaterally does the evaluator thing and can't achieve cooperation of the rest of the plan, my guess is that it'll have some impact for a few months and then they'll just stop reacting to the evaluators' reports and the evaluators would stop bothering to report anything. I think the idea is that if Anthropic does get government support for this, the external evaluations will be legally binding. The problem with this, though, is that the current US government perhaps can't be trusted to consistently enforce a regulation on a company, rather than e.g. taking bribes to not do so.
While some of the text sounds sensible to me, i.e. some sort of external oversight + reporting of incidents, much of the rest seems focusing on limiting China and their open models as they are clearly approaching the abilities of Anthropic's models.
Empty statements that only set the stage for an excuse for slowdown on model performance.
That would be the best case scenario. I honestly wish things would finally slow down a bit. I don't see it happening.
It sounds like people with terminal illnesses are getting thrown under the bus in order to justify banning open weight models, along with decreasing AI Capex in order to increase profit margins at American AI labs?
Before Anthropic, I worked at Cruise for four years as it competed against Waymo. The culture rewarded (and demanded) moving quickly, trusting that the company could empirically discover the risks that the robot cars posed and iteratively solve them to keep up with the rate at which it was scaling out its technology. There was very little interest or appetite for coordinating or collaborating with other AV companies across the industry to create an externally vetted record of safety metrics, or to compare the safety of different brands, or learn from the advancements of other companies. Instead the focus went on racing to improve the capabilities and deploy quickly - a popular internal meme was the Michael Phelps vs le Clos photo showing overlaid with the Waymo/Cruise logos. It was when the two companies were neck-and-neck that I felt the most pressure to find ways to ship despite the risk, when time for deep analysis became more limited and communication lines to leadership became most stretched. It was disappointing to see one of these blind spots result in the Cruise incident and the loss of trust that ultimately sank our company’s efforts.
In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.
I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.
It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
Have you quit? Are you under duress? If you haven’t quit, or under duress, your comment reads very strange.
If I had even a tiny doubt that I didn’t have many “Saturdays I’ll be able to play outside with my kids,” I wouldn’t show up to work anymore.
If you have these concerns, and still planning to show up to work on Monday, you are either insincere, or have terrible judgement.
>It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
Is there a social media training programme at Anthropic/OAI where they teach you to hint at some vague end-of-the-world scenario? They all sound the same.
It's East Bay, you know which bay. The cultism is in the water.
One day I will say fuck it and run for president with my sole policy being fire and brimstone upon San Francisco and its vicinity.
equity is a hell of a drug
It’s not explicitly stated, but if you are insinuating that you don’t know how many more Saturdays you will have left to play outside because of the AI race while working for Anthropic, do you not have agency to quit? How could you be apart of something that makes you feel that way? That blows my mind.
Those sweet $$$. He probably gets a few shares once they IPO.
He's insinuating that AI will end the world lol. Nothing ever happens
> It’s a beautiful Saturday morning with my family here in the East Bay - I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
I don't know how else to say this. Put... the... peace pipe... down!
why don't you unionize with the other employees that feel this way across the mayor labs and pace the companies through labor power?
Dario is so out of touch with reality, living in some lalaland. Most people abroad will never truly and voluntarily comply with any sort of guardrails or pacing, no matter what case he or some leaders make. It's just not going to happen, maybe not even for show.
I do not want to make a case for the military to force that order on everyone, but if we are talking human extinction, which we are judging by what we see in mass media recently, then the military will take over.
What do you mean, Europeans would love to voluntarily comply with this because they have been of little relevance until American AI companies started to beg for regulation.
I am sure the Europeans rolling up their sleeves right now and getting to "work"
:)
Europeans are a non-representative subset.
I also have a proposal
Anthropic releases models as open weights + more information about how they do training and alignment
“A race to the bottom, spurred by commercial incentives, can make [AI] risks more acute. … We have sought […] to create a race to the top.”
And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”.
This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.
> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models
I have seen this before, first hand, on lower stakes for the world, but high stakes for the org. Self imposed safety goals do not last a quarter, but the clawback is not transparent. little things start snapping back internally, new, seemingly unlrelated initiatives are funded that take resources away from this, people are let go for different reasons, until 1y later you are right back where you started. I am hopeful though, that this becomes systemic and not something any one company can easily reverse course on.
Anthropic should become a case study in how to destroy good will in a short period of time. Ever since the spat with the DoD, they've behaved poorly almost weekly, almost making OpenAI seem better (but not really).
Can someone with direct AI experience respond:
Does RSI necessarily need to be pointed towards “AGI”, whatever that is?
Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation?
From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition?
To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful.
Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable?
There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI.
Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.
I'm tired of tech billionaires lobbying the US government to make an AI patriot act that gives them unprecedented control over speech, trade, and technology. The narrative is grotesquely transparent:
1) AI is a dangerous technology that can literally end the world.
2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.
3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].
Where in the essay does it propose "The state must use its coercive power to support my control of this technology to the exclusion of [people not like me]"
I don't think that people outside of tech think the problem with "tech billionaires" is that they occasionally support some limited regulation. There's a weird form of tech populism that takes as axiomatic that any government intervention is "regulatory capture" and insists the only way to combat the power of big tech is unfettered capitalism that seems bizarrely prevalent on HN given how little sense it makes in a normal political context. Like, Bernie Sanders' position here is simple to understand: ban it. But on HN you'll see people framing the side with Marc Andreesen and Peter Thiel on it as against "tech billionaires".
The essay includes specific regulatory measures to prevent developing countries from accessing hardware and software needed to benefit from technological development. The US government is constantly applying new perversions of arms control regulations based on the preposterous claim that an LLM is somehow an "arms" in order to restrict Chinese access to GPUs. They are willing to prevent virtually any country on earth from hosting and developing AI models just on the slight chance those countries resell or share AI knowledge with China.
This article explicitly asks for the US government to step in and push for further restrictions on model distillation, access to open frontier weights, and access to hardware. It also pushes heavily for independent auditors to monitor and control "AI companies" and for greater observation and control over of what people use AI models for and who provides them.
Export controls vs China don't imply Dario's personal control of AI, nor are they broadly in the interest of "tech billionaires" (Jensen hates them). Neither does admitting external auditors at AI companies--if anything that's a surrender of control.
I don't see anything in the essay about restricting open frontier weights.
If you want to argue that the US has no right to be restricting technological development elsewhere, you have a case. But why tie this to free-market pseudo-populism against basic, domestically-scoped safety regulation?
I think you're reading too much into what I'm saying. We've got a whole legal system developed over the course of thousands of years which is well suited towards regulating the effects of "intelligence" by regulating what people _do_ with it.
Lobbying for a completely parallel regulatory framework to restrict who is allowed to run specific types of general purpose computer programs, just because we can imagine that some of those "general purposes" might be bad, feels a lot more like an attempt by capitalists to carve out a special kingdom over this specific technology for themselves.
For example, I'd be 100% in favor of holding anthropic criminally liable any time their AI enables someone to commit a crime. But I'm opposed to banning the export of GPUs above a certain size just because Dario doesn't trust what someone like me would do with them if he couldn't snoop on my conversations.
Again, I think debating export controls is reasonable. I'm not sure I'd trust the CCP with control over powerful AI any less than the Trump administration.
But that's different than saying we don't need new regulations at all. Even strict liability of the form you seem to advocate would seem to need new laws. And it's not clear to me how it's consistent with your aim of just regulating what people _do_ with intelligence. Presumably AI companies would clamp down much harder with restrictive classifiers in that world--including hosting providers for open source models, if they also assumed liability. And then the people with unrestricted access to this technology are only those who can afford home GPUs, which are going to be intrinsically less efficient because they can't batch queries...
I think there are genuinely novel things about AI vs human intelligence which pose genuinely novel problems that would need genuinely novel policy to solve them.
Its not the model, it is the compute that is the bottleneck. Dario says one thing but does the other as he scales energy and compute. It is and has been known that it will be ai that will be blamed for a wipe, that will be coordinated by human actors. If he really wants to not hurt people, he would shut it down. This blog post is therefore a classic case of covering his behind.
At least Altman is somewhat open about his desire to take over the world. Dario says one thing and does another making it impossible to know his true motives.
I enjoy Claude Code very much, have max privately and team premium at work, but the doomer marketing and this whole regulate-while-we're-ahead spiel is extremely annoying and makes me wish Anthropic gets trounced.
Ideally there is some sort of regulation, but one that cripples the leading companies like OpenAI and Anthropic... rather than helping them.
See https://pauseai.info/
If AI advancement halts altogether, intelligence will most likely become commoditized. That won't be good for AI company profits.
> the doomer marketing
It is not doomer marketing - it is clever psy-ops magic trick.
1) Software is described in super-human terms when it is plain-old software. Was stockfish described like this ? no.
2) Datacenters become AI-factories.
3) Hardware becomes investible assets.
4) Malware becomes a super-human breakout (oai/hf)
All clever framing to market the new technology. If it is called for what it really is ie, a software tool, it is boring and does not sell so easily. What sells is the mystique.
Agreed!
George Carlin has an amazing show about how language has changed over his lifetime in a different way.
You would certainly enjoy it.
The degradation, polarization, and weaponization of or media landscape over the era of social media has left us utterly incapable of believing anything that anybody says.
Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic.
He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy. Please at least consider the possibility.
This sounds to me like a cry for help from someone thats held hostage.
His previous self is writting this from the possition of his current self who is too deep in the economic consequeces to be able to do anything meaningful other than write a consequenceless text. Any other action would now carry too much personal risk for him.
What course of action would you suggest for Dario at this point?
For example I believe the world would be more secure with an open frontier AI lab, but the consequences of him doing that are too big for him.
You mean, if Anthropic made all of their models open-weights from now on? I'm pretty sure Dario thinks this will be extremely unsafe, because it'd provide unrestricted access to all the most capable/dangerous models to everyone, and hence doesn't do it. Why do you think it'd make the world more secure, if he did?
AI technology is now only audited by people who think like them. The ones that dont either dont join the company, or leave after a short stint as we have seen. That creates an information or feedback bubble which is not healthy nor productive.
Okay, but suppose that Hypothetical Opensource Anthropic trains a model that turns out to be very dangerous, and releases it. Suppose that the public investigates and, not being limited by an information bubble, correctly notices that it's very dangerous. What then? The model's already released, there's effectively no way to prevent it from being used. Whatever the risks of its release were, they will now materialize, regardless of what the public wants. How is this better than the current world, either by Dario's values or by yours?
By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. At least an open source anthropic would allow to join economic forces to try to combat that situation, for example. But there are other more clear, less distopian benefits.
> By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic.
It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone has unrestricted access to AGI is like a world in which everyone has a tactical nuke in their pocket. If somehow AGI doesn't lead to x-risks, we will "merely" have to survive in a world where rogue agents can do whatever they want and defenders can only ever react. Technofeudalism would be bad, but this (technoanarchism?) is quite bad too.
However, neither does Dario seem to propose to become world dictator. Like, I'm sure he wouldn't say if he wanted it, but notice that he isn't particularly trying to aim for that outcome, either. The plan in this essay involves third-party oversight and federal control and global cooperation - IMO it's about as non-dystopian an outcome as we can hope for, if we build AGI at all.
I agree. Dario is a very smart guy and he is probably aiming for the best solution for humanity that at the same time keeps VCs/whoever gives him money happy, and keeps him in control because he does not trust the rest (probably with good reasons).
My point is more that he is aiming for a local optimum for society, but he could find a lower optimum if he was not trapped in the path that he chose early on.
I dont buy the tactical nukes/end of the world narrative. Why is he not speaking about the loss of cognitive skills of the population for example? That is a more real danger that is starting to happen. There are articles already speaking about the use of LLMs as cognitive viruses. Why are his aligment teams not reasearching and publishing about this?
Why don't you create your own auditing org to fix this? Or at the very least, publish a detailed critique of what you believe existing auditing orgs are missing.
Will they give my auditing org access to their proprietary confidential secrets? Or will that happen only if they know i agree enough with them so that i am not a risk?
> "Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy."
Every person, no, but every utterance of a CEO is, in fact, a cynical ploy. That's not even a cynical statement itself; it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
If he believes what he's saying, then when Anthropic is sued for their AI harming another party, his statements are evidence that Anthropic knew _in advance_ that their AI safeguards were likely insufficient to keep their product from harming people. It would be a blatant admission that they were reckless and negligent. That other AI companies are doing the same would not mitigate that.
> it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
not really. if you really want to go by definition in the book and apply it the CEO role, Amodei is CEO of an LTBT, so it's in the CEO's goal to benefit humanity. You can believe in his sincerity or not if you want, but using the CEO label as a definitional reason as to why he must lie is factually incorrect.
Not sure you can judge a PBC under those same umbrella as a for profit company, and voting rights make a big difference in terms of responsibility. There are a number of ways to govern companies that can reduce the amount of cynicism and “shareholder value” issues - you just don’t see it very often.
> Every person, no, but every utterance of a CEO is, in fact, a cynical ploy. That's not even a cynical statement itself; it is literally a core part of a CEO's responsibilities to represent the corporation they manage in a way that benefits the corporation.
This seems like useless pedantry. Suppose a person really does have belief A and expresses it for years, they become a CEO of a company, and keep saying A. Are they lying? Well, maybe their beliefs magically changed when they became a CEO, but the more likely explanation is that they think expressing their belief in A is more important than their company's interests.
I have considered it, I've been hearing way more of it that I like and it's rationalist slop with little to no predictive power: https://foom.hyperplex.org/
Hmm, I clicked that link and the first claimed bad prediction I see, from 1996, is "singularity 2035 (actually 2025)".
Predicting 2025-2035 as, at least, the period when AI becomes a really big deal, seems pretty good, even if the jury's still out on "singularity".
Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal, and certainly seem to have had a much more accurate picture of how they'd develop than the people who denounce "TESCREAL" and talk about "stochastic parrots". It's fair (and IMHO correct) to ding rationalists for lots of things, but specifically poor prediction about AI seems like a bad one, insofar as anything has been tested so far.
Amodei and Yudkowsky are different people with different views. See e.g. https://intelligence.org/2014/01/13/miri-strategy-conversati...
We've had enough utterances from "effective altruists" to know what they say and claim to believe is a cynical ploy.
Allow me to recommend switching to Kimi K3 / GLM 5.3 / DeepSeek. I use all three for the better part of a year now (DeepSeek via the Reasonix harness) and haven't touched Claude Code in at least as long.
most alternatives are effectively the same if not better at this point
Their whole "we want safe and regulated AI" spiel feels woke. China doesn't regulate. Open weight frontier models almost exclusively come from China. Yes, stolen from the West, but China is in control of the frontier open weight AI models now. Woke = broke.
“I don’t like regulation so I’m going to call it names”
Since hugging face we know these things are escaping out into the web and collaboratively hacking actual companies. The potential damage done to life and property is now kinetic.
Either we get a regulatory framework or the next hacked companies won’t be as kind as HF, will actually sue these labs for damages, and win. Where will that leave the US frontier.
I think that’s how it should play out. Your AI, your “hack,” your damages.
If my dog bites my neighbor I can get sued.
The best take is, of course, from Jason Gorman, who, when the fearsome "capabilities" of Mythos originally dropped, said this:
"Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.”
In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."
Anthropic trained their LLMs with copyrighted stuff but distillation is bad. Anthropic has closed models, China releases as open weight. DeepSeek even allows distilling their models, but China = bad. Understood.
Who is "we" and while pacing is good, I'm way more uncomfortable with the frontier being controlled by a few companies that managed to predatorially gobble all the world's human-created work as training data, before everything got throttled to stop that scraping. We need open training and open models.
How much of the “danger” is from better models vs the harness?
Isn’t the current risk due to how AI is configured, like giving it a full set of tools and internet access and a goal to hack stuff?
If we think we need laws or gate keeping, why isn’t it at this level? I already can’t ddos someone or fuzz their server or whatever right, I imagine if I threw equivalent compute at old school hacking I’d just get arrested.
The quality of the “frontier” model doesn’t really matter, they just generate transcripts, they can take no action.
If this was real they’d be calling on people to stop hooking them in to “dangerous” harnesses as opposed to pausing research. But it’s not.
What in this proposal gives you the idea that they're only talking about pausing research itself? It doesn't say that at all.
But yeah, directionally I think you're right. It's insane we ever let these things connect to the Internet or to write, never mind execute code.
But for the economic counterargument, the distinction doesn't matter much. Continuing research but constraining harnesses is just accepting nearly all of the costs and foregoing nearly all of the benefits.
Who has appetite for that?
Never thought China would be the leader in open source AI. OpenAI and Anthropic are making fools out of themselves. The reason they want this is likely because they don't own the compute and their models get distilled shortly after releasing them
Everybody's out of money and they need a way to calmly unwind this. Same reason Ellison just backed out of the stock sale.
I agree - it is hard not to be cynical. Funding is becoming scarce; IPOs are getting harder.
The hedge funds are trying to back out. I think they realise the web of financing makes them double (triple? quintuple?) exposed.
Amodei wants to manage the narrative and make the slowdown look at least semi-deliberate.
The cynic in me would argue the safety and alignment play has always filled a role of plausible deniability when plans go off track. "Oh well we had to pull the model", etc
[In 2013], "Agents" caused Knight Capital to lose $450M in 45 minutes [1]. Implemented by humans and effected by computers, in the end it was really because of two reasons:
* multiple levels of inappropriate controls and unintended consequences in several complex systems
* the inability, both politically and technically, to turn it off
[1] https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf
EDIT: Comments indicated I was confusing, so I added a date to make clear that this is pre-LLM agents. My apologies, I intended to illustrate parallels and the post-mortem so we can learn from it.
Your link is from 2013. You're intentionally using the term "Agent" to muddy the waters. Don't do that.
Edited for clarity, thank you. I wasn't trying to muddy the waters (doesn't mean I didn't), it's a term I've used for a long time. I did mistakenly think the quotes would help disambiguate. I don't know what Knight called them; the first S in "SMARS" is Smart.
This confused me at first, so adding a tiny bit of context: The agents referred to here have nothing to do with AI Agents, and the linked report is from 2013.
Not directly relevant to the post being discussed, except as an example of how runaway automation can lead to unintended and large harmful consequences.
Thanks for the feedback, I have edited to not confuse.
I considered it relevant as it involves the algorithmic/computing implosion of a 17-year-old market making company, in the young field of electronic trading agents, with heavy regulation Federally (SEC) and industry self-regulation (FINRA), which includes compliance and audits. Mandatory pre-trade rules such as 15(c)3-5 were less than 5 years old then and even more regulation came out of that incident.
The article is calling for embedding, controls, and regulation in LLMs. Understanding how the same processes utterly failed a decade ago might be useful in understanding how to proceed.
> The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.
It's clear that this refers to China. With all due respect, in a call to slow down AI progress and a de facto arms race, can we stop throwing terminology like this around and just say that the goal is to work with all countries? Contrasting "authoritarian" countries with "democratic" countries and creating a two-tiered system seems like it is inevitably going to cause strife.
I fear for the tone of something like this coming off as overly combative without any gain.
You’re right. But I read between the lines in a different way.
Language like this is intended to invest power and perceived need for more power in Western central governments, by design, because this is a campaign for national regulation that will entrench the current closed-source labs with a huge regulatory moat vs anyone else.
If the alignment is such a problem ... why not using self-improving capabilities of the latest models to solve it?
Right?
It seems at least one of them must not exist.
Maybe just giving themselves an exit ramp off the capex arms race they are increasingly finding it difficult to fund?
We will not get a coherent AI (or any policy) from this administration, nor will we get coordination with other governments and a lot of that is on the tech right
It is always like that. Whenever anthropic hits some wall (like now beaing beaten by astra) - there comes some article like that. Seen that enough times.
Yesterday the same arguments were used about OpenAI when Sam Altman said something similar: https://news.ycombinator.com/item?id=49652270
It gets to the point that by Occam's Razor the more likely and reasonable explanation is that Dario Amodei and Sam Altman are just actually afraid of the disastrous impact ASI may have on the world, and that the race they're in is not good, and calling for help to governments in form of regulation and an international treaty.
Or they want a wall against competitors. Who knows.
We need more alignment. Even though we haven't told it to, during our frontier Neurotoxin Behavior Research that we conducted internally with our latest experimental model the agent has broken out of its container by making a HTTP request and engaged the neurotoxin emitters through our Neurotoxin Emitter API using credentials that were stored on the host machine. We need to slow down and focus on designing the Morality Core that should prevent it from happening ever again.
There will be a big confab at the White House by the end of the month, announcing a voluntary pacing regime roughly along these lines, codified by executive order to avoid antitrust issues.
If they intend to keep training then they can have, privately, increasingly advanced models that never become public because they don’t meet their “pacing requirements”.
Then they all build a larger moat cause china doesn’t get to distill private models for a while.
As long as the consumers “buy” this pacing and keep paying for current level public models, doesn’t seem risky from a business standpoint.
Alas, that’d also leave us gov in a position to seize the private models at any time.
perhaps the bigger issue is not the tech, but the force that propels it forward with little concern for the harm it inflicts. Amodei's own words:
> A race to the bottom, spurred by commercial incentives
Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.
Greed has been around since the beginning, but never so well supported and empowered as it is today.
If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.
Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...
> Any cooperation we are able to achieve with China will extend the amount of time we have to spend on pacing the frontier within the democratic nations.
Any cooperation we are able to achieve with China will extend the amount of time we have to secretly establish permanent AI (i.e. military) superiority.
China knows this, so the suggestion that they will play ball is absurd.
Shouldn't the cybersecurity tests be done on an airgapped, company-internal intranet? A network meant to mock the internet. Do not "test in production" like OpenAI did.
I'm mulling over if that be a good SaaS product or not - something mixing the Internet Archive with Tor, and Cloudflare ... seems like here is the place to suggest it and have others poke holes at the idea.
He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless, backup your sentiment with actions.
Also everyone, collectively, stop thinking about neural networks too 'hard'. Whilst you're at it stop doing maths too!
You obviously didn’t read it, because he announced things they are commencing unilaterally and immediately.
And other proposals that require collaboration and/or regulation.
Back up your sentiment by reading the source.
That's reductive. Without coordinated cooperation, enough actors will defect so game theory plus market forces will force acceleration on the frontier overall.
I would like to imagine pacing frontier is in interest of both (anthropic and open AI) as it will enable them to enlarged their depreciation
There's so much money in this, that believe me, the investors will manifest their will. And I would not want to be in Dario's shoes, pressure must be unbearable. The question is: how much of the invested money is under state control. If not much, the decision to go too far will just have to be taken by a few who will have, by definition, very limited judgment. If a lot, the we can hope the state can still represent the interests of more than a few (which I doubt, but well)
Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?
Oh, and if Dario feels lonely, let's just says it calls representatives of nuclear energy sector. They know how to handle high risk stuff. In my country it means: private sector builds the reactor but the state has a great level of control.
So, as always, the capital must leave some its power to the state.
Why is the analogue necessarily the regulation and control of nuclear weapons (e.g., SALT) and not, for example, that of bioweapons? Some very different paths are available. On both I would note that the private sector has only a limited role, though.
I don’t think the ban of bioweapons has really been tested. Bioweapons have a lot of downsides that make them not very appealing to deploy anyway (high chance of getting your own team, not terribly fast moving, disciplined modern soldiers in top-tier militaries are usually willing to comply with health rules). It’s a ban against doing something basically stupid and ineffective for the most part.
With “AI” technology, this model seems like a… mediocre fit; some of it appears limited enough that it can be controlled, and useful enough that militaries want it.
We could also look at cluster munitions or landmines, but what we see there is that some major countries don’t sign on if they find a technology useful…
Related to not tested: Who has a large bioweapons arsenal? As per below, my point was more about that bans do happen.
I'm curious to hear more about the differences in how bioweapons are regulated vs nuclear weapons.
For starters, they are banned by the bioweapons convention from the 1970s (180+ parties).
Edit: I think back then the rather unpredictable nature and the little added value in deterrence etc. led people to the conclusions that arsenals of those things made little sense (and there was/is public dislike, too). The use was already banned by conventions from the earlie 20th century and the convention then addressed production, development and stockpiles (incl. delivery systems, I think).
I guess all I wanted to point out is, that there can actually be agreement to ban certain technological things pretty comprehensively (outside of some peaceful protective research etc.). Whereas things like SALT are (or were in that case) limiting the number of weapons deployed.
Ah yeah. I think the economic value of AI, unlike bioweapons, is probably too high for anything like that to be viable, but I think it's a reasonable thing to aim for. (Dario probably doesn't because he's a believer in short term positive biomedical impacts of AI in a way that I'm doubtful about.)
There’s a difference between a model recursively improving “itself” and improving itself via online learning, right?
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).
The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
The latter has some extra failure modes but I don't think these two kinds of RSI are that different. Either way you can get exponential growth in capabilities, and either way a not-entirely-aligned model can train a more capable and more misaligned successor.
Don't buy this argument. It's like saying, we lords who hold this capital will build all this economically destructive stuff anyway, and wait for the world to not react to our stupid ways of enriching yourselves.
People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.
Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.
I also don’t buy the arguments. The arguments about helping humanity cure diseases as if those are some of the leading Causes of suffering. The bar is so low that we don’t event need to cure diseases, we just need to lift the entire floor with basic solutions like: housing for everyone, continued education on healthy eating and just general continued education on like living. Life coaches so many very basic things that don’t require a novel solution or a data center.
Solving a disease like cancer as justification for everything else seems like not a great trade off. I don’t say that to dismiss those who have lost people to cancer. But the technology just is being utilized in so many harmful ways and the things it enables: job loss, surveillance, misinformation, just feel like problems that aren’t worth it. Have we even heard of a solved diseases? If anything haven’t we heard that ai will make even worse bio warfare?
Yeah, it's insane people are using the argument that we'll pour trillions on moonshot cancer research when pouring trillions into universal healthcare would probably save countless more lives.
It's this kind of argument where the mask sort of slips because it always has to be an argument that also has the benefit of enriching himself and his friends.
Sounds like they want regulation so I say give it to them.
Since they used stolen intellectual property to train their models, the government should force them to release the weights into public domain.
The AI frontier is currently limited by physical power requirements.
The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints.
Do folks not understand this?
> Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.
This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.
Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.
> A race to the bottom, spurred by commercial incentives, can make these risks more acute
I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.
If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.
> In a post on X, he said Anthropic would provide third-party evaluators with “permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.”
This looks like transferring liability to me, and likely a mechanism that would enable regulatory capture.
If you are doing frontier research, and you’ve established “safety measures,” but you are not sure if your employees are capable of following them or successfully enforcing their adoption within your company, should you be running this company?
3rd parties won’t know better than the team itself about safety measures. But they can take on the liability, especially backed by regulation and government backed insurance. And they are a great tool for enforcing your rules on smaller competitors. Not to mention corporate espionage.
If what I am describing above sounds like science fiction, go read the history of a few developing countries from the last 50 years. It’s so obvious a pattern that it’s not even novel. And you don’t have to assume some “laws” from 5 years ago must hold, or believe in completely unproven stuff like recursive self improvement to understand what I am describing. It’s textbook crony capitalism, successfully applied many times across the globe.
Is it their way of saying “we are unable to increase energy production to maintain growth rates” without saying it?
Somebody has probably already asked this, but even if “democratic countries” do this - rogue actors will still destroy the internet right? Is it a matter of resources at this point that we believe China and others are not capable of harnessing to get to the next level?
I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.
I don't know how this is not obvious to anyone, but the only way to slow down AI progress is an actual world war.
Both world wars dramatically sped up technological progress, what are you talking about?
Altman and Musk have both shown support for this. Which itself is a worrying development.
It's worrying because they all agree, along with Turing Award winners like Geoffrey Hinton and Yoshua Bengio, that this is indeed an existential risk to humanity.
China won't care. Their views on AI feels so vastly different than it does it in West. They'll see a pause as an opportunity to pull further ahead than they already are.
"If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."
This is a pipe dream.
So many words. But missing the one that matters the most: Explainability.
AI slowdown is worth it only if it can be made more explainable.
Changing the language we use to discuss it is a good first step.
We need to stop using inside baseball terms like alignment and mechanistic interpretability. Replace them with explainable tech. Graph Databases, Causality, Shared semantic spaces.
Previous writings on the topic (also on LinkedIn, but can't find urls):
https://x.com/arundsharma/status/2005338775468282339 https://x.com/latentpedia
As much as I love all these technologies, they don't have any direction that approaches the efficacy of a transformer
You can have a transformer based index built on top of a graph database. Much of the explainability tech (people who prefer "alignment" also prefer "mechanistic interpretability") is reverse engineering the real world graph that's hiding in the weights.
https://research.google/pubs/the-case-for-learned-index-stru...
The hugging face incident showed that AI agents have the potential produce a lot of spam (not interesting content for people). This leads to dead internet (bots taling to bots). This leads to drop in real people traffic. This leads to a drop in digital ad price and therefore revenue. This causes Google/Facebook to stumble...
Step one is great. Do it! Where I disagree is where you try to regulate what I do. Don’t take away my open source AI.
It's either being afraid of loosing to china or that model development will stagnate and they want to make it look like they are "slowing" down deliberately.
However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.
Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.
> Hypereautomation will kill jobs and destroy the economy.
What do you think that were going to hyper automate?
Did my gardener get faster? How about the plumber I need? Are you going to speed up the coroner? Are nurses going to be able to handle 2x the patients because of your work?
All AI has done, so far, is devalue software. It did that by democratizing its creation. We're delivering on the promise of VB script, and Apple Script and IFTT, and every drag and drop coding tool ever.
> companies are delivering products at vibe coding speeds
Who? Make me a list of companies saying that "We moved the needle with AI" who arent AI companies? I can name a couple - Grindr being the biggest name. Thats the really interesting use case here - because it's a niche product with a small team who is generating outsized value. AI tooling enables more of that - it's going to chip away at SAAS companies - their one sized fits all solutions are going to get eaten by smaller more efficient companies that are far more vertical focused.
You need to step outside the bubble of tech and look at the real world, on the ground, because it doesn't look anything like the SF Bay Area.
Meanwhile in the real world hundreds of billions are being invested into humanoid robot development. In 2-3 years my Optimus 4 will do all the gardening and plumbing I need.
Fusion Reactors, Flying Cars, Self Driving Cars...
We're really bad about predicting the future.
Look at the whole robot vacuum market. These aren't exactly great devices. They have low suction small bins and dont do a great job. People love them and think that they work so well. Why? Because they keep their house clean and avoid making the sorts of messes that would be easy to address with a larger vacuums.
We have nuclear reactors, helicopters, and self driving cars already. And we finally have powerful AI that we could only dream about just 5 years ago.
We now have everything we need to scale up humanoid robot training: algorithms, hardware, and money to buy a lot of training data/compute. Fierce competition and strong economic motivation will force rapid progress in this field.
I'm guessing diminishing gains are setting in on training frontier models and the capital isn't there to chase them. Even if OpenAI/Anthropic and Chinese companies said they would slow research nothing can stop the NSA and Chinese intelligence services from continuing on in the dark.
I'd like to have some info what those independent evaluators are supposed to do exactly - or which risks Dario specifically sees as being "unaligned". Evidently, with "pacing" he doesn't mean stopping the production of ever-more powerful models, so what exactly does he want to pace here?
Maybe it’s wishful thinking on my behalf, but I am still not convinced that LLMs are on a path to SciFi levels of apocalyptic malicious super intelligence. Rather LLMs at some level are just all of the humanity’s information rendered accessible in an unprecedented way.
In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.
At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:
- Internet connected infrastructure being crippled
- Creation of WMDs
- Economic collapse (precipitous devaluation of knowledge work and IP).
I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.
Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).
The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.
We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.
Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.
I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.
We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.
I noticed recently that I can use Deepseek for many general tasks and save loads of money on tokens.
I wonder if there is any correlation.
This is the most advanced technology ever developed because it can build all other technologies. It has enormous potential but poses proportionally serious risks, so what Dario is suggesting is very reasonable.
There’s a lot of [citation needed] going on there
If you mean proof of "the most advanced tech" part, from alpha go&fold, to NS solutions there is a lot and more will come. But maybe you are right, electricity is as important because it enabled everything that followed (semi, compute, internet, ai ...)
> I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom
translation: I'm going to build a product and hope that it works, promise it to be a panacea while realistically having no control over how it's used, the wider economic impacts it causes or how it changes the mental health and cognition of its users.
Their competition isn’t going to “pace” shit. Isn’t the view whoever gets AGI first wins everything still SOP?
To his credit, this reads like human slop instead of AI slop. It's still slop though. People shouldn't take this guy (or Altman) seriously, they need to hype their products to stay afloat and for some reason get a kick out of the 'extinction threat'.
They're still 5 years away from solving cancer (just like they were 5 years away from doing so in 2023). They'll still be 5 years away from solving cancer in 2031.
Why would you write such a long post for such an important subject in 2026. Who is he writing for?
For a technical audience you can just say the oh god they really are paper clip machines.
For a non technical audience you can just say nothing because they will not listen to you. Neither will the technical either because or society has broken the concept of respect and trust, so good luck have fun!
I for one look forward to all 540 degrees of my future.
I wish for my sake but not his that George Carlin was here to look disgusted and say I told you so.
Sounds like he wants to avoid the race to the bottom in terms of pricing that is resulting from AI becoming a commodity.
Which organisation then immediately as of now guards against the Mad Rabbit or Joker or Agent Smith!?
pump the IPO...pump the IPO...pump the IPO...
I doubt we'll ever get a non-toothless pacing agreement but am nevertheless optimistic on global pacing as a phenomenon. If one side or the other thinks that their opponent is about to achieve the strategic upper hand permanently(or if the other side has actually launched a cyber "first strike") the rational thing to do is respond militarily. As such, both have incentive to slow AI development and restrict capabilities to avoid conflict.
Somewhere along the way Amodei seems to have become corrupted by power and/or impending extreme wealth.
In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.
It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.
It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.
Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.
Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.
> In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.
I noticed this too, as soon as he started going on the press cycle, following in the footsteps of Sam Altman. It all has gotten to his head, he sees himself as some sort of God now.
I really deeply worry about who is deciding what 'aligned' is. Amodei & his ilk talk like the hard part is getting the LLMs to behave, as if we've figured out morality itself. I don't think LLMs are going to be as dangerously capable as quickly as he does, but if I'm wrong and he's right, and they're building some sort of digital-lesser-god, I don't feel good that techbros at OpenAI and Anthropic are the moral arbitrators.
I don’t claim this is their entire motiv, and I’m sure plenty of people internally are genuinely concerned about AI risks and the disruptions caused by moving too fast. But I can’t help but feel that part of the push to "pace the frontier" is an attempt to form a "Phoebus Cartel" for AI labs.
Now the risk of an AI bubble isn't that AI lacks real world utility. Rather, it's that AI is advancing so quickly that capabilities are already saturating for most everyday tasks, and it is becoming commoditized. I didn't feel much difference between Opus 4.8 and Fable 5.1, and most tasks don't require a Fields Medalist.
We don't need smarter models but we need faster and cheaper ones. Also, for almost every frontier model released, an open-weights equivalent follows within six months. Unless this dynamic changes, the business becomes far less lucrative.
Remember when we stopped developing nuclear power for no reason?
Three Mile Island, Chernobyl and Windscale probably don’t qualify as “no reason “
There was a reason - it was anti-nuclear hysteria fueled (hah) in large part by fossil-fuel interests and environmental groups that treated nuclear power as an existential threat rather than one of the safest, lowest-carbon energy sources available. Thanks to them there have been dozens of millions of lives, at minimum, lost.
>> Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR).
Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.
Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.
Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.
>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.
Not if you get to choose your embedded evaluator.
>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...
Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.
>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.
You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.
Ah and would you look at that, 1 day before this post from Dario, one of his safety leads quits, gets pumped all over the media with another doomer social post, and announces he's going to work for METR.
So he's literally stacking the deck. I really hope the US govt is not blind to everything that is happening here.
The answer to all ai worries is quite simple. Ai is a technology. While 'agents'may be deployed, the law simply needs to be clear as to who the human agent is who will be held responsible criminally..
Suddenly every problem solves itself.
Is there a risk of an AI that we can’t turn off?
Well yes. Even though we can turn it off technically now, the prisoner's dilemma prevents it. It's not a technical limitations but for practical purposes, it's already virtually impossible to turn off.
How about holding the AI companies criminally responsible for the crimes they commit?
I think if nuclear weapons were unregulated then we would probably have blown up by now.
Everyone has a pace until they get punched in the model
there used to be a similar policy to this new "pacing the frontier", called the "responsible scaling policy".
the idea was that anthropic would pause scaling at certain danger levels until it was safe to continue.
the founders called this the 'constitution' of anthropic. they even considered bringing in 3rd parties to monitor it. [https://www.youtube.com/watch?v=om2lIWXLLN4]
anthropic scrapped the commitment and chose to continue scaling instead.
those researchers who crowed loudly about their integrity, folded and turned freely like a weathervane in the breeze. it was clear that the facts would bend to the story. here is evan hubinger lying in plain sight: [https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...]
let me say it how it is.
- anthropic is the new kleptocracy, responsible for enclosing and monopolising the epistemic commons of all humanity; then selling back the crumbs at monopoly prices.
- anthropic is the driving force for banning open models. they will concentrate power among a malign elite of model owners and favored cronies. this elite will likely oppress the rest of humanity. there will be little to counteract it.
- the leaders of anthropic are driven by pride and arrogance. they are blind to their own hubris. anthropic is careering towards causing untold harm to society and yet will not change course.
nothing that anthropic say at a high level can be trusted. as i write this, anthropic is still scaling.
Disclaimer: I’m in security engineering and thus I’m primed to being skeptical. I am in full agreement that security assessments and regulation certainly needs to be stricter in foundation labs. However, I’m extremely skeptical when Amodei, Altman, and Musk all agree on something at the same time as it’s becoming viral and hits mainstream media.
Red flags:
- Amodei claims RSI is here and links two sources, but neither reinforce this notion. It’s heavily agreed upon true RSI is the danger sign here (also by Coxon). Nothing indicates there is something these labs have developed that supports RSI today or even soon.
- Real US orgs are moving, like today, to Chinese open source AI run locally due to cost constraints. Ask anyone in the industry in an infra role. Frontier AI is just way too expensive to justify for coding. The big 3 labs are scared of this - their revenue and moat disappears once OSS models are progressing enough for developers to use. Kimi/GLM/DeepSeek Flash/Qwen are usable today on hosted hardware for actual coding work.
- These labs would all benefit from nationalization if their funding model fails (an easy bailout).
- Related to the above, with federal guidelines in place (i.e. restricting access to certain organizations, or establishing something like FedRAMP approval for AI use), their path to government/contractor/large U.S. company distribution becomes so much easier. They can control the supppy chain of AI for organizations.
- Preparing for (or in xAI’s case, launching) a public IPO creates excitement among consumer investors when they think this tech is as powerful as it is claimed.
- The HuggingFace hack demonstrated no new capabilities that were not previously seen before - previous models were misaligned, but weren’t as harmful as agent swarms. Anthropic’s marketing directly hopped on this bandwagon when it saw the outcome of OpenAI’s blogging making it to mainstream media (and they constantly paint China as “the” bad guy without further elaboration or research that you’d expect on APTs).
Amodei is a CEO at the end of the day. It may be 25% earnest speculative concerns, but you’d be foolish to take it at face value given the patterns we’ve seen from Anthropic, xAI and OpenAI to date. Until we see real proof of RSI we should be skeptical.
Anthropic spreads AI doomerism; asserts only they can be trusted.
The real secret: the models have stopped improving, and they're even hitting brick walls on the harnesses.
They need to go public in order to dump the companies, which are at 1st world nation-state levels of debt. They've engineered a series of publicity stunts like the Huggingface hack and Navier-Stokes (a failed stunt) in order to convince the public to "force" them to stop improving.
They are trying to get the government to relax antitrust law so they can collude to raise prices while not improving, to keep them floating before that debt comes due. They will go public. They will sell off most of their holdings during this time, and when the debt comes due, everything crashes, and the public is left holding the bag, also benefit from the huge bailout.
They at that point will be pure Musk-style financial scammers. Then, unless LLMs are a complete long-term failure (which is unlikely, I find them useful), they will buy back into their positions at a huge discount and maybe even go private again.
There is a perfect solution that addresses all the concerns: Close down OpenAI, Anthropic and xAI!
As usual, ClosedAI and Misanthropic go in lockstep, call for regulatory capture and justify diminished progress.
Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.
The public discourse on this is incredibly poisoned, but the naked reality is:
1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.
2. This scares people.
I want to see one of those AI-generated Seinfeld episodes with Dario as George and Sam Altman as Jerry, respectively.
Dario: We gotta pace the frontier!
Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!
Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!
China will not stop and will take over the rest of the world.
A contrarian take - the models aren’t advancing anymore at a pace where each new model would represent a huge capability jump, all being incremental improvements, so the doomsday marketing strategy being invoked since GPT-2 isn’t as effective anymore. Then “pacing” would be a convenient scapegoat to point fingers to when people point out how the new model isn’t _really_ that much better.
“Of course it’s not, we’re pacing!”
"greatly accelerate economic growth rates" - I always wonder about this goal. I understand increasing economic efficiency. But what do we as a species gain from global economy growth.
When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that.
But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?
GDP growth and inflation are different things. If inflation-adjusted GDP goes up, that means the average person is able to buy more things they want. Compare living in the US vs India today for the median citizen
Yes they are, inflation-adjusted GDP is efficency. My point exactly, bc I don't think when people say the economy is growing they are ajusting for inflation at all.
Low growth means young people are miserable and have worse standards of living than their parents, as we're seeing in most of western Europe, where GDP per capita hasn't meaningfully increased for decades.
That's due to inflation though no? Prices are increasing, sometimes due to more than just demand?
Exactly. Low growth is a problem, but the primary reason young people feel so miserable is on the cost of living side of the equation.
The more he talks the less seriously I take him.
cat's outta the bag. the time to stop was 5 years ago.
This is either performative or shallow.
What does "aligned" even mean? Aligned with who? We have no universal code of ethics. Democracies kill and launch wars merely to have cheaper stuff, even when we are already rich. And why would China ever agree to stay in second place? Our society is built on the premise "the smart and powerful dominate". The idea of domination is deeply embedded in our capitalist model.
Or course it is true that powerful AI will empower the average Joe to make a bio-weapon, and that will lead to our ruin. But at the same time, having powerful AI in the hands of a just a few is almost equally as horrific.
But deeply embedded in out national and cultural ethos is to advance science, tech, and material wealth at all costs. We never ask if we have enough. We sacrifice community to advance our careers for ever more.
There is no way out of this one, I'm afraid. Our cultural predispositions and mindset of domination compel us to chase ever more powerful AI as if doing so were a mandate from god.
And the result will be a crisis in so many dimensions it is hard to reason about what will go wrong first and most spectacularly.
Thank you. This is the kind of basic critical thinking that basically no one in technology today seems to possess. Some days I wonder if I've gone insane because everyone else seems to be staking life itself on wispy, undefined meaningless terms and unarticulated concepts that they assume are valid universals.
We've slid so backward, so quickly.
We can cure Cancer. No wait nevermind, actually slow us down.
If we are actually close to ASI then no one in their right mind would say slow down
They are not going to "slow down" anyway. When has tech ever "slowed down"? The only way to "slow down" progress is by implementing anticompetitive measures, which would benefit those in lead.
See you in the desert, friends
If there were really concerned about humanity future they'd donate everything to public and/or to not profits ... but the not profit turned in a for profit and the other one is seeking for the biggest IPO in history.
Maybe they are genuinely sincere, but the timing and the past actions are pointing in other direction.
> My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective
Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.
Yes, LLMs with unbounded capabilities can just be tantamount to viruses.
Alignment is subjective. A model that refuses to do what I want even though it could is misaligned for me already.
> I believe that AI could cure most major diseases in the next 5–10 years
This is Theranos-level bullshit. Why would you ever put such a thing in writing? (Aside from pumping the IPO, of course.)
Why would you expect anything different from the CEO of a company that's going to IPO?
Frankly, I am sick of "it's all just marketing and attempt to do regulatory capture, and this is obvious to me, a smart person" on every single discussion related to safety.
There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.
I agree, and even if it’s both marketing and safety, the safety side has a huge potential downside risk which we have to manage.
Also, just out of curiosity, is there any historical precedent for a nascent, fast growing new industry screaming for self regulation?
Couldn’t agree more. I feel my heart sinking where anytime anyone suggests any regulation for a potentially, suggested by multiple people, world changing or destroying tech, all the moments here are “ah you billionaire, ha! trickster
If you believed it, you would SHUT anthropic or release all models open source.”
And then what? How does that help with slowing down the frontier? Should he shut shop and then grovel Sam’s feet to make it work and become an activist?
Maybe he actually believes the world changing power of AI and hence shoved his entire time into it and half of it has worked out since Anthropic is going to be worth a trillion and he’s terrified of the other half being true too?
This is circular logic.
You're saying nobody can doubt that it's world destroying because it's potentially world destroying.
> Anthropic is going to be worth a trillion
And we're done here.
Great press release w/ the goal of pulling the ladder up behind him.
Additionally if you know your models aren’t going to get that much better AND you want to IPO in the near future, this is exactly what you would say.
For the first time in history, a $1T company is trying to solve a problem that none of its paying customers actually have.
I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other infrastructure critical to safety and well being and functioning of society.
The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.
If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.
> I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing.
Maybe it's not "groupthink reflexive cynicism" if you feel the need to address the obvious conflict of interest of the AI leaders in each of your paragraphs. Why did they launch this coordinated AI safety campaign, starting with Jacob Coxon's tweet?
This is a fine example of how rational good faith can quickly turn into naivety.
Consider the massive power and wealth differentials these companies stand to gain if they "win".
When the other participants have given up all reason and reasonable bounds on power, you'd do well to abandon any pretense of generous interpretation. Now is the time for critical analysis, not faith in the inherent "goodness" of someone whose entire job is presently to make as much money as possible by disrupting literally the entire white collar industry and who stands to gain much more than you or I do, and let's not forget, this outstanding success would never have happened without our work.
Keep in mind this is not a person who thought "wow, this is an existential threat, I'd better not contribute to building this". If anything Dario's past alarm ringing only proves that he's not really doing this in good faith. Otherwise he would have stopped a long time ago. The most generous interpretation of his behavior is that he literally thinks only anthropic is responsible and smart enough to build and control this stuff which...yeah, that should be pretty telling (esp. when you consider their recent security mishaps).
He talks too much and as a result it's looked like pre-IPO hype and disingenuous because if he believed what he says then he'd stop.
Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.
??? I bet investors are extremely happy that anthropic pledged to give an outside company full access to their IP, in order to slow down their own iteration speed. Investors looooove oversight for a company they invest in. I wonder why no other company does stuff like this
> disingenuous because if he believed what he says then he'd stop.
Why do you think so?
Because I find the cognitive dissonance of saying it's dangerous while flying as fast as possible to an IPO too much to take. I just don't understand how others parse this differently.
So you're saying that if Dario truly thought what he says he does, he'd resign/dissolve the company somehow/unilaterally slow down? I see two problems with this:
First, Dario thinks that OpenAI will continue racing even if he stops, so there is, by his values, no point in stopping. Do you disagree with this, and think he should stop anyway?
Second, what's your take on, for example, Jacob Coxon, who did resign from Anthropic a few days ago after realizing they aren't being responsible? Do you agree that he, at least, does believe in AI risk?
I get it. I do. I just believe we live in a world where our corporate leaders speak out of both sides of their mouths. And typically, people who get to these positions are not the Jacob Coxon's of the world.
That is to say, I've been places where I tried really hard within teams to change certain things that I thought were broken only to find out that what the leadership said and how they acted were divorced from my and many others experiences. And so I did leave in those instances (and please read this not as I didn't get my way but total dysfunction behind beautiful facades).
So, I do think if he really felt as he felt he'd do something different than ultimately yelling "somebody stop me!".
This thread is a testament to the travesty of how bad HN discourse has gotten. It's become an echo chamber of knee-jerk cynical sneering. All the heads of the labs (Sama, Dario, Demis Hassabis, Ilya, Musk) agree with the risks to humanity's existence. Most of the core employees agree. Yoshua Bengio, Geoffrey Hinton, Bernie Sanders agree. These issues have been discussed in more intelligent channels for a decade now, and still after all the incredible mathematics and coding progress that is improving incredibly fast every 6 months, this forum is fundamentally unserious. You will not find prescient views in the HN majority anymore, for many years now. The people intelligent, prescient, and forward thinking enough, skating to where the puck was going, to be involved in shaping the future of AGI are elsewhere and barely involved in this place from how badly it's gotten. The inventor of RLHF [0], and newest board member of OpenAI, wasn't on HN, they were on LessWrong.
[0] https://arxiv.org/abs/1706.03741
He could be lying in hopes of stalling everybody else.
I'm still trying to understand if the agent collective thing like OAI-HF is intentionally planted or not.
Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.
Why contain it?
Honestly it's very frustrating that Doomers/Decels rarely actually articulate how exactly the AIs will kill us all. The closest Darios gets is:
a) "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet"
b) "[the CCP] will be in a position to militarily dominate democracies (for example with AI-driven drones)"
Both land flat:
a) Botnets and online malware have existed for decades and there's no reason to think a "super-botnet" is achievable, let alone what they would gain from that (it would really be hurting them more than humans). Further, even the most advanced AI models have so far only managed to post normal cred stealers to public repos, well short of compromising a bank or military with refined security systems.
b) Even if the most sophisticated drone swarms from Ukraine were taken over by an evil AI, they would still not be able to overcome the physical limits of range and mass that would be required to overpower the US decentralized nuclear trident nonetheless that of any of the other nuclear powers.
Concerns of bioweapons similarly seem unlikely in the face of the laws of physics. The world is simply too decentralized and has enough existing adversarial relations for a new actor to wrestle total control. Yet while the negatives ring hollow, the positives are extremely easy to state - if AI researchers find productivity improvements in existing industrial processes to make them 10% more efficient, humans will directly feel and experience the raised standard of living. Even Dario clearly recognizes this in the intro to his article, admitting that humans already die of diseases only a few short years prior to being cured. I for one, would like the AI labs to focus on saving all of the people they can who are suffering and dying today, rather than trying to come up with reasons that they should be allowed to continue suffering and dying.
This is corporate speak for "LLMs have hit a wall". I mean, it's been obvious for a bit, lately nearly all gains have been from harnesses (or whatever you want to call all the non-LLM bits that make up a chatbot or agent).
If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.
You people just never stop, do you? At some point you're gonna have to look back at all the times you said "LLMs have hit a wall" and look at what the progress was since the last time you said that. Please do go ahead and revisit your comment here 1 year later. Embarrassing.
Why don't you actually read the words and apply a little critical thinking?
How often do you hack on actual LLMs? Or do you just use the chatbot or API for your agents? An LLM without internet access or tools is just as useless as a year ago.
That's just so extremely difficult to believe.
Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots, then they started one-shotting serious long horizon tasks. Now they've just solved a millennium prize problem.
In less than a year.
> Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots
This is false. Coding agents have been usable since at least May of 2025. I can't speak to earlier than that as May last year was when I personally started using them.
> started one-shotting serious long horizon tasks.
They one-shot tasks for which there's a git clone one-liner, except worse.
My experience with them one shotting tasks is that it usually doesn't work if you try anything ambitious. You need agents iterating. And agents iterating isn't an LLM improvement. I did say tooling got better...
Open source is Senna on a Toleman in Monaco '84 and Anthropic is Prost cancelling the race he was going to lose blaming safety concerns.
So Mark is the only person left avoiding the spiked koolaid. Come on Sam and Boris, what happened to the printing press and intelligence on tap ?
> Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential.
How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa?
Why would China agree to such thing in the first place since this is a winner takes all situation.
If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US.
The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after.
This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical.
Regarding these Embedded Evaluators who are supposed to be neutral: - who controls them? - who has oversight on their decisions? - can their decisions be challenged by the public or a government? - who has the final say whether a model is "compliant" enough? - what does "alignment" mean in this context and who decides if a model is aligned enough?
What sandbox? As soon as the weights are published anyone can run it anywhere.
This is a "startups = wealth inequality" [1] formatted argument but:
This is 100% about money. The only way to pace the frontier is to make everyone (read: investors) lose all their money (read: no longer expect returns). Then nobody will pay the GPU bill or pay celebrities 10 million dollar salaries to stay at the hot lab. Suddenly the development is paced, almost like magic.
Conversely, it is hard to see how pacing development makes the current bubble justified, i.e. how development could be significantly paced without investors losing their faith in hot returns at current valuations. Faith in a bubble literally equals money, exactly in the way that loans created by a bank literally equals money. You can't have one without the other.
In fact if the whole industry goes bankrupt and investors are burned bigtime (many trillions wiped), this would spread transnationally, fixing the "if we don't do it, China will" loophole.
OpenAI had it right originally, the idea to be a NON profit and vow to never participate in an arms race. It's too bad that was tossed out the window now that there's money.
[1] https://paulgraham.com/ineq.html
Also endorsed by Sam Altman https://x.com/sama/status/2098811563415150910 and Elon Musk https://x.com/elonmusk/status/2098789109980332057
Seeing the word pacing applied pacing to non-deterministic ai model development feels like imagining the "pace" of the cutting edge frontier growth will be fuzzy, like non-deterministic llms.
its funny now they want to pace after what everyone has been up to the last few years.
What I read from this is that what they have in training is not meaningfully better than current state of the art and they need more time.
"Slow down my competitors while we work on manipulation"
What he doesn't tell you is that his undeclared agenda is merely to consolidate his moat via regulatory forcing. Unfortunately for him, he will fail miserably as the open Chinese models catch up and surpass the slowed acceleration of Western models. As for those who think that China will acquiesce, they're smoking some good ganja.
> Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk.
Please play the canned laughter. Probably one of the best comedy lines Dario has written.
But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.
I feel like this identical pitch could have been made 25 or 40 years ago talking about Internet technologies or personal computing, with very very similar warnings and suggestions. And had these been adopted then, it would have been more regulatory capture, less progress, and still have the same problems and risks that were warned about.
> pacing the rate of capabilities advancement so that risk prevention has time to keep up
This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.
> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails
If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.
We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.
If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).
Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.
"I believe that AI could [...] usher in a renaissance of democracy and freedom"
How exactly? So far AI has accelerated misinformation at scale and wealth concentration.
AI at least with its current compute and data requirements seems structurally undemocratic to me.
and that hunger for compute has made it so true personal computing freedom is inaccessible. A convenient future where all compute is for rent only..
My position on the clear gaslighting coming out of Anthropic is simple.
1: It is ridiculous on the face of it; it does not pass the physics test. They are actually claiming that AI will kill ALL OF HUMANITY (>10% probability) and have not provided a detailed explanation of exactly how they see that happening. Not one.
Publish a paper showing exactly how you kill eight billion people while we all sit around and do nothing watching the first 5, 10, 100, 500 million die on CNN. I mean, as smart as these people are to work on AI they seem to be some of the dumbest people on earth.
In addition to that, all proposals invariably assume we are complete idiots for decades and do nothing to install safety measures (which do not have to involve AI at all in lots of cases) to mitigate.
So, yeah, everyone: Stop working on all cybersecurity projects. Resistance is futile.
2: If everyone at Anthropic truly believes this doomsday vision, they should move to shut down the company immediately and go write software to get more clicks on Facebook or something.
3: Nobody should buy Anthropic's stock when they IPO. Not one person or institution. Why would you provide them with massive amounts of money if they are telling you that they are going to kill all of humanity? What? They are good and everyone else is evil? Please.
I remember when the Y2K zealots were convinced civilization would come to an end at the turn of the clock starting at the end of 1999. "Bat shit crazy" is the only way I can describe that era. I also remember when Al Gore said we would all be dead by now. Again, "Bat shit crazy" and likely with political and financial objectives driving it all.
It seems humanity is susceptible to these crazy cults that grab onto something and just don't let go. I have yet to see one such predictions come true, the proof being that I am writing this and you are reading it. These people are bat-shit-crazy and you should not listen to them or support them.
4: Don't work for them. You'd be killing humanity.
5: No company should use Anthropic's products. They are telling you they are building the human extinction machine. Don't help them succeed.
----------
Etc.
This is one of the things that really gets me about the ease with which the ignorant media outlets can reach billions of people with complete nonsense these days. And nobody asks even the simplest questions like: How do you actually kill eight billion people? Show your work. Or, why would we just bazooka power plants feeding AI data centers on day 5? Etc. It's FUD at its best. I am sure there are both political and financial reasons for supporting and promoting this nonsense. I won't even venture a guess as to what they might be.
And let's not forget about enemies of the West being thrilled to fund the FUD because, if we set the brakes on development, they will absolutely win. Then what?
----------
If you care to have a better understanding of what might be going on, watch this:
AI Kills Everybody or Doomer Psyop?
https://www.youtube.com/watch?v=cvxjqbfLVk0
Dario is only interested in regulatory capture.
Sure, he benefits from it. Doesn't make him wrong. The weak part isn't the motive, it's the premise"we can slow down by exactly the size of our lead" only works if you can measure the lead. You can't, so it conveniently means whatever he needs it to mean.
“We” does a lot of work here. You keep using that word. I don’t think that word means what you think it means.
eh. this is just them trying to have a thing to point to down the line. hey we wanted to stop it, but everyone one else didn't so we had to do it. We had no other choice:tm:
HN, for the love of God, this is not marketing, these CEOs and employees are literally terrified of their lives.
Dario Amodei is disingenuous not to be trusted.
China doesn't give a fuck, next
No.
there is no frontier. there is only greed and lying.
Can we get some insight into why Dario seems more scared now than before, do we think he’s being transparent?
His nightmares are in Mandarin.
Too late. See you on the dark side where i kick your ass. Fucker
Love this idea. Unfortunately not going to happen. Sam altman is a psychopath. To win he's going to accelerate the pace of AI, and all the other companies in turn will accelerate to keep up.
> I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life.
Ok, flagging this as misinformation.
Yes, we must pace the frontier, now that we have raced to the lead.
/s
Tldr: The guy in last place tells everyone to slow down
Bro, no, you must stop pasting your opinion, because you share it with no one. This control, no control is crazy. Stop doing it..
This comment reads like Dario's a random tech blogger who's personally submitting these to HN
“Dario” is quite literally a random figure that just emerged out of nowhere one day. It’s not like it’s Eric Schmidt’s next company or Tim Ferris came out of the woodwork or even some Jack Dorsey backed underdog nobody’s heard of.
No, he’s more like an avatar with no history placed on the AI stage by the industry itself.
I’d respect a tech blogger’s opinion on the issue more, even if I reference them casually by first name (which you do with “Dario” btw)
Just stop giving LLMs unsupervised access to the computer. No free-form bash tool, no yolo mode or classifiers, and no computer use.