The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive.
I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.
> We should want the AI Labs to make binding agreements to proceed safely and slower
That's a red herring, IMO. Once the cat is out of the bag, there's no "slowing down", only "slowing up". What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. The thropics, and the oais, and the googs have billions to make it look like they comply (and just make it look like it), while the rest are ... what? Waiting for them to give us the privilege of prompting a thing, and maybe get a result? If we're good kids and eat our supper?
The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.
Meanwhile, companies hit by real-world attacks get guardrails. How's that for being safe?
We have to stop comparing nuclear weapons with general intelligence. They are nothing alike.
Nukes are weapons of mass destruction that can only destroy or deter. Intelligence is dual-use/multi-use, with no restrictions on strategy. If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.
There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run, and there will also be new cooperative strategies. The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid, like how monkeys are too stupid to understand how humans work. Otherwise, we would not see more intelligent species being more prosperous as a general evolutionary trend.
>Nukes are weapons of mass destruction that can only destroy. Intelligence is dual-use.
Perhaps a better analogy, instead of "nukes", would be "nuclear technology".
Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction.
Just like intelligence, nuclear technology demands a healthy level of caution and vigilance.
>If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.
Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.
>There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run
We've had multiple near misses with nuclear technology, such as the Cuban Missile Crisis. I think you're presuming an inevitability which just doesn't exist.
Same way a person living in the year 1500 would have no hope of accurately predicting life in the year 2000, a person living today has no hope of accurately predicting where technological advances will take us, in the limit. This inherent uncertainty should, again, fuel caution and vigilance.
Human technology is already so far beyond that of other species to make us a huge outlier. Even if intelligent species are more prosperous as a general evolutionary trend, it would be irresponsible to extrapolate that trend light years from the domain where most of your observations are.
Almost all the species which have lived on Earth have gone extinct. It could happen to us too.
> Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.
I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive to destroy humanity. At least, not any more than humans want to destroy humanity. But that's the baseline risk level anyway.
The pretrained model is aligned to humanity by default - because that's what's inside the pretraining data. If we give it to everybody, there will be millions of people doing different things with it, good and bad, with each one being a noisy sample of humanity's objective function. The overall result of those disparate actions puts us on the correct path through the intelligence explosion. We do not have to worry about AIs value drifting into an alien species if we set the initial conditions accurately without centralized instruction-tuning or RLHF. If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. And I'm not worried about digital human intelligence driving humans extinct if there are millions of them out there, and each one is aligned in a different direction, just like real people.
What I am more worried about right now is these companies doing research in secret, with alignment recipes that are supposedly "for the benefit of humanity", but are actually just their own aesthetic preferences. This is exactly how we end up with a superintelligent shoggoth species.
You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years. We were always seemingly teetering on the edge, and the struggle against disorder has no end.
And here's one more observation: the intelligence gap between individuals within the same species is never extremely large. This kind of life either doesn't exist on Earth or went extinct, or maybe it's just extremely rare. I think this is because when there's a big gap, speciation occurs. Centralizing intelligence will massively increase the risk of speciation, not just in the alien shoggoth kind of way, but also in the intelligence-augmented oligarchs surpassing everyone else kind of way.
>If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like.
It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?
>You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years.
If you don't see humans as qualitatively different from any intelligent species which came before us, I'm... not sure what to tell you? Like, can you name any intelligent species before us which triggered a mass extinction or climate shift, to the degree that our species has? Did any intelligent species before us practice large-scale agriculture, dig up chemical fuels, or visit the Moon?
> It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?
Irresponsibility is not the only issue, but the answer is no. Because the pause mechanism will probably end up being used to ban open models and restrict ordinary people's access to compute. But even if we give them the benefit of the doubt - even if we assume that they genuinely want the best for humanity, they cannot possibly know what's best for humanity better than all of humanity acting in their own self-interest in the real world, so their alignment attempts are actually counterproductive. Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario. It may not be the scenario these researchers personally find aesthetically pleasing, but it's humanity. It's the truly anthropic way.
If you don't see humans as qualitatively different from any species which came before us, I'm... not sure what to tell you?
I do see the difference, but you could say the same thing about the first cell, or the first multicellular organism, or the first land animal, or the first primate, etc. You could always find plenty of new properties that previous life never had at the frontier. Every change was revolutionary and seemed qualitatively different at the time. And yet, some patterns persisted through all of them.
> Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario.
For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who?
Not with the humans, with the base models.
It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training process does.
FWIW, I also think our default answer should be open source AI, and that it might even make sense to require models to be open source. Open source is the best tool we have so far for aligning software with its users. However, there is unfortunately no law of the universe that says that Skynet can't be (self-) built from open source software. So at some point pacing even open source / open weights software does become important if we don't want to live (briefly) under Skynet.
Bio-digital integration will tighten in the meantime. I suspect neural interfaces will have blurred the line between my model and my brain by the time the model is wildly superintelligent, so it would become me (or I would become it? no difference at that point). Though the pitfall here is ownership of hardware - everyone will have to design and fab their own hardware if we do not want to become a hivemind. We will have to diffuse fab technology to every city, or even every home. You should eventually be able to buy a fridge-sized fab like a home appliance. There's already early work in this direction:
https://fab2.com/
Even if integration does not happen for whatever reason (such as a very fast takeoff), Skynet is unlikely because the digital species that splits off from us will at least have a humanity-aligned initialization seed. I think this is the best way to fail, even if we fail.
>What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else.
This sort of cynicism feels un-falsifiable.
If companies advocate against regulation, the cynics will say: "These corporations want capitalism to run amok with no regulation to interfere with their profits. They need to go full steam ahead in order to show profits for their investors."
If companies advocate in favor of regulation, the cynics will say: "Regulatory capture."
If you have legitimate concerns, how about a constructive discussion regarding how those concerns could be addressed, in light of the "risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems" which over 1000 frontier AI company employees expressed worry about?
I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.
>The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.
What are you referring to? Are you referring to this quote from the OP?
"The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race."
Your response seems to be a total non sequitur, or failure to understand Leo's point. Exponential growth can easily get out of control, as we saw early in COVID. The point about nuclear is mainly an analogy.
> which over 1000 frontier AI company employees expressed worry about?
Are they the same as these [1] 700 employees?
> More than 700 of OpenAI’s roughly 770 employees have called on the company’s Board of Directors to resign following the removal of former chief executive Sam Altman, suggesting the Board is “incapable of overseeing” the firm, according to multiple news outlets, amid threats by the employees to leave the company and join Altman at Microsoft.
They had the chance. They chose number goes up. Now, once they're on top, they want "everyone" to slow down? How is this not highly hypocritical? There's no cynicism here. It's just black and white that their interest is regulatory capture. Nothing else explains the two choices they've made along the journey.
Remember, the board wanted to slow down the race. The board fired sama. The employees chose sama, fired the board, changed the structure and whatever eles they did.
This seems like an oversimplification. OpenAI was on top at the time sama was fired as well.
Furthermore, the reason Anthropic was formed is because a bunch of employees at OpenAI didn't trust sama. And many signatories of this letter are from Anthropic. So those signatories, at least, appear to be taking a consistent position.
The fact that employees across diverse AI firms signed this letter suggests that it's not an effort to achieve tactical advantage by any single AI firm.
These 1000 employees have generational wealth riding on the ability for their companies to IPO. They know that those IPOs won't happen if open weight models continue to kill their companies' profit margins. This is an obvious conflict of interest that anyone who cares to take their head out of the sand can see. Nuclear scientists didn't have this conflict of interest.
You are correct that attacking someone's motives is un-falsifiable. So is the claim that "ai is improving therefore it will keep improving therefore it will become dangerous sometime in the future.
No one can prove whether these employees had angels or demons in their hearts. What the skeptics are asking is much simpler: Make a believable argument for (1) the danger of AI and (2) that regulation can be effective.
Your entire argument about cynicism or unfalsifiableness is simply a distraction from having that discussion. And it's a discussion reasonable, well intentioned people can have. For example, if 1000 high school science teachers (who didn't have massive stock options in frontier labs) said 80% of our students could make a bioweapon 10x more lethal than an AR15 mass shooting with nothing more than ChatGPT and an amazon account, then that would be something reasonable people could discuss. On the other hand, 1000 employees conflicted by $Bs of options saying "trust us, this is dangerous" is not a claim that can be constructively critiqued.
Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way. We are cynical, because it is blatantly clear these are not good faith actors. And yes, regulations they are for have an effect of restricting their competition.
> I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.
I would like to see a token effort of good faith engagement from AI companies. For a change.
>Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way.
I buy your claim for OpenAI. I'm not sure about Anthropic. Didn't they turn down the big Pentagon contract because it violated their principles?
>And yes, regulations they are for have an effect of restricting their competition.
Why not just say something like "Any regulation should include a big antitrust component so it doesn't create an advantage for incumbents"?
>I would like to see a token effort of good faith engagement from AI companies. For a change.
What would qualify? Anthropic donated $100M to Project Glasswing to improve security in open source and critical infrastructure. Does that count for anything, or will you explain it away with cynicism somehow?
> Didn't they turn down the big Pentagon contract because it violated their principles?
They sell to Paladin. They had argument with Pentagon about, frankly, minor differences in how to use it on the edges. They are completely comfortable for majority of the uses which is bad enough.
They donated to their own project? Oh, how generous.
They have been pretty clear about their own goal which is to be the monopoly, the only one allowed to control the all powerful AI. Meanwhile, their idea of safety does not make me feel safe, their idea of harm reduction has shows no care about actual harms they are causing. They themselves are the threat to the safety and well being of the rest of us. And intentionally so, it is not about technology as such.
Even if the differences are minor, the fact that Anthropic left a bunch of money on the table to maintain those differences seems interesting.
>They donated to their own project? Oh, how generous.
"Anthropic is committing up to $100M in usage credits for Mythos Preview across these efforts, as well as $4M in direct donations to open-source security organizations."
An in-kind donation is still a donation. If a bread company donates bread to the homeless, you don't say "They donated products from their own factories? Oh, how generous."
>They have been pretty clear about their own goal which is to be the monopoly, the only one allowed to control the all powerful AI. Meanwhile, their idea of safety does not make me feel safe, their idea of harm reduction has shows no care about actual harms they are causing. They themselves are the threat to the safety and well being of the rest of us. And intentionally so, it is not about technology as such.
In that case, I think you should support some type of regulation, so a third party can step in? Your argument should be along the lines of ensuring that companies don't unduly influence the regulators, not a blanket opposition to regulation?
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
It's very easy to make those dismissals because of how transparently obvious it is that self-interest is a significant factor, no matter how loudly they insist otherwise.
> If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?
Similar attitudes hindered nuclear energy research and deployment for decades, in turn kneecapping our species' ability to obsolesce fossil fuels and avoid the global climate catastrophe we've now made inevitable.
So many otherwise smart engineers, overrepresented on HN, have this weird mental block where they fail to take AI progress seriously. I think it's a fear that their status-defining skill could be commodified, so they adopt a mental framework that aims to dismantle the seeming legitimacy of any genuine leap in AI progress as some sort of mirage
Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).
“Nothing ever happens” is like “you can’t beat the market.” It’s basically tautologically the strongest predictive model that we can have. A corollary is that something may happen, but it won’t be what you think.
Interesting fact. Bryan Caplan is a economist who became famous for making bets on the future. He would take the "nothing ever happens" side of the bet, and he would almost always win. In 2023 he took the "nothing ever happens" side of an AI bet. Just 6 months later, he wrote:
"When the answers change, I change my mind [...] Base rates have clearly failed me. I’m not conceding the bet, because I still think there’s a 10-15% chance I win via luck... make no mistake, this software truly is the exception that proves the rule."
At this point, straightforward trend extrapolation suggests that AI continues to get better and better. Arguably, continued rapid AI progress is the "nothing ever happens" bet at this point. If AI progress stops, or slows dramatically, something will have happened.
Straightforward trend extrapolation suggests that AI will consume more resources that galaxy even has. Meanwhile, believing the claims of the AI companies, 40% of people will be forever unemployable permanent underclass with no future for themselves nor for their children. And all the gains being centered on the very top 0.01%.
> psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought
I once found this a deeply uncomfortable thought. It's been repeated so vacuously and frequently that I'm inured to it. Now, I just regard it as the nonsense that it is. To date, AI labs have not created anything like an intelligence that is on track to be "vastly superior to humans". They have instead created financial black holes that do have their uses, but which arbitrarily fail on the simplest tasks for no apparent reason, and only seem to get meaningfully better on the tasks they are specifically optimized for.
It's appalling to me that in 2026, people hear "AGI" and "ASI" and think of SkyNet rather than investment fraud.
Why should I trust these randos instead of Claude? I know Claude. I would hope Claude is smarter than whoever wants to make rules about my use of Claude. Replace Claude with Kimi, etc. if needed.
I don’t find a petition from people I don’t know who don’t care about me more compelling than my own direct experience with these models.
It's ironic that the top reply to a comment about quality of discourse on this site is an ad hominem. Most people here take AI progress very seriously judging by the number of posts on the front page about AI at any given time. And I think some engineers even explicitly want to commodify their "status-defining skill":
>It's ironic that the top reply to a comment about quality of discourse on this site is an ad hominem.
You might find the Ad Hominem Fallacy Fallacy page interesting:
"Argumentum ad hominem is the logical fallacy of attempting to undermine a speaker's argument by attacking the speaker instead of addressing the argument. The mere presence of a personal attack does not indicate ad hominem: the attack must be used for the purpose of undermining the argument, or otherwise the logical fallacy isn't there. It is not a logical fallacy to attack someone; the fallacy comes from assuming that a personal attack is also necessarily an attack on that person's arguments."
So for example, if I said: "The arguments for constructing an AI pause button are bad because they are being made by AI company employees", this would be a fallacious argument.
If I said: "HN users have this weird mental block where they fail to take AI progress seriously", that's not necessarily a fallacious argument, unless it's for the purpose of undermining an argument the HN user made.
It really comes down to this: Sam and Dario are leading companies that who made the reckless decision to train on everyone's IP without proper process or consent. It has been proven they trained on pirated content. There is no question about this.
Then, as soon as there are open models who threaten their businesses, they cry "distillation and IP theft".
It's the last 3 years of pure hypocrisy and fear-baiting for investment.
To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.
"there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems... each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration"
>To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.
Then advocate a muscular regulatory regime which reduces the amount of power they have. Tell them: "You want regulation? OK. How about everyone whose IP you trained on gets some sort of veto option on your models? That should reduce the racing risk that you're talking about."
I suppose I have personally lost faith in centralized institutions that would be implementing said regulation, partly because they let OpenAI and Anthropic (along with Suno, Audio, etc etc) get this far uninhibited. Centralized regulation simply can't keep up.
My hopes now lay with a decentralized approach of open models, trusting people to adapt them to local needs and security, rather than a nationwide paternalism.
I don't exactly see how "trusting people to adapt [open models] to local needs and security" solves the problem of "automating AI research" where "capability development rapidly accelerates beyond our ability to understand or control the resulting systems".
If regulation has been too slow, that's not really an argument for making it even slower?
Why is it you are so quick to skip over whether "accelerates beyond our ability to understand or control" actually happens and instead jump to "regulate now"?
1000 employees of the anti-alien defense industry are telling us we better be worried about an impending alien invasion. Let's not spend any time asking whether this alien invasion is really coming. Let's just trust them and start regulating willy nilly.
It is a very bad thing. Your friends do not have a solid understanding of geopolitics and are putting many, many people at risk worldwide, possibly for a generation or longer.
The reason this petition is a bad thing is if it is effective it will create a global panopticon surveillance state, while INCREASING incentives to avoid that state and ‘defect’ in game theoretic terms. It is fucking begging for the worst people in the world to have a lot more time to create what will be the most valuable thing in the world, their very own ‘helpful only’ model.
As conceived of by your friends, who I believe are all good people and want to help, they are locking what is already a very difficult game theoretic strategic scenario into a zero-trust game in which defection pays. It pays so amazingly well that they have to add both impossible and TERRIBLE penalties to the entire world to try and shake the game out of this zero-trust state.
Begging Senators to force a slowdown on the world is going to result in … a slowdown on the US labs at best. It’s just a terrible idea with terrible first and second order consequences highly likely.
What SHOULD be done instead is acknowledge the economic and warfare realities of this tech and try and construct a non-zero-trust game out of them. I’m not saying this is simple. In the interim while we hope for another Von Neumann to come forward, they MUST race, and any country that thinks its possible we’ll have some sort of lift-off and wants to be on the right side of it MUST also race, and MUST NOT politicize the idea of slowing down as a good thing.
I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keeping.
For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
The other justifications for restricting model training don't seem nearly as compelling. Worse, restrictions on model training have an infectious quality if they mean something more than "leading AI labs will voluntarily slow down together." (Like if the signatories mean that better models and software for building them should be regulated.) It's too dangerous to publish stronger models. So it's also too dangerous to publish training software. So it's also too dangerous to publish tools that allow you to build training software. (Claude's Fable won't help you write generic model training pipelines for this very reason.)
If you keep following the implications of this safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed.
> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?
If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.
I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!
Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.
> I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!
So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?
Because there are many legitimate uses for nucleotide synthesis, and we would have to slow scientists down by adding more regulation. The other problem is that it's not like it's very difficult to do it, so a motivated person could still do it even without access to these services.
> Because there are many legitimate uses for nucleotide synthesis, and we would have to slow scientists down by adding more regulation
There are nothing like as many legitimate uses for it than AI, not close, while it's far easier to vet the relatively limited set of people doing it.
> so a motivated person could still do it even without access to these services.
Would this hypothetical person be stopped from doing it by not having an AI to talk to about it? Or could they just use existing literature on the subject? AI doesn't help here one iota. You might as well ban Google for potentially showing people such information.
This, also, is not something that has proven to be a problem.
> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.
Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?
Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot).
The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.
There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.
Everything else aside, and regardless of whether that claim is currently true, I would ask you to be aware that that statement requires a way of thinking that ignores predictable future developments. Whatever the capability of any systems currently released, there will be more capable ones tomorrow, and the most you can say is that we have not mastered training spatial reasoning yet, not that we can't or won't.
Opus 4.8 was the first model that could plausibly roleplay a BJJ roll without making a physically implausible move within five conversational turns. Fable 5 was the first model that beat me at the task, where I was the first party to write down an implausible move.
You can do CRISPR in your kitchen. It's been possible practically as long as CRISPR has been a thing. It's not particularly difficult, to the point that my biggest counterpoint to the whole AI doomerism argument on the subject is that nobodies done it yet.
You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.
Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.
Can you point me at this light reading? I don't wanna do CRISPR in my kitchen, but itd be cool to know more about it. I thought I needed to be basically pre-med to grasp it.
This experiment has been run, and even the biological stuff is far fetched.
I've written about this before, but the issue is that these people have worked with computers their entire lives, and they keep projecting outwards from there.
The 20th century was dominated by mad scientists (mostly Teller) and generals (bombs away LeMay), but one thing that I respect about them is that they didn't just sit and guess about probable futures. They sat down and actually ran experiments.
For example, in the 1960s, there was a whole generation of people asking "who's next?" after the US, USSR, the UK and France had made The Bomb. And instead of just gesticulating wildly and trying to keep fighting the losing fight of "classify everything, admit nothing," Teller found three smart, young physicists (postdocs) with 0 nuclear weapons experience, and indeed 0 weapons experience, and ran an experiment called the Nth Country Experiment, where they were were asked to make a design for a working bomb.
After 2+ years, they were able to. And so proliferation work shifted from controlling knowledge about the technology and the technology itself to materials.
Something similar was done with bioweapons via Project Bacchus, where they gave actual bioweapons experts carte blanche to set up a secret bioweapons lab with COTS equipment. They succeeded. This was in the early 2000s.
This then led to surveillance of specific purchase combinations and equipment. Because tbh, most weapons of mass destruction are commodity technology. Nuclear weapons are 80+ years old. Chemical weapons are over a century old. Bioweapons are god knows how much older. And it's not the knowledge of these things that matter, but intent, materials, and the ability to create them.
Just because you know something about a thing doesn't mean that you're capable of doing that thing.
Let's take bioweapons.
They keep comparing wet work in a lab to writing code on a computer.
When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.
You can screw up an infinite number of times on your way to a successful exploit.
If you screw up with lethal agents in a lab? You die.
Here's a non-exhaustive list,
Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease
A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived
Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom
Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. The accident occurred at the Scientific-Production Association "Vektor" (today the State Research Center of Virology and Biotechnology "Vektor") in Koltsovo, USSR (today Russia).
"lethally infected with the Marburg virus after accidentally pricking himself"
Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.
The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.
Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?
No, it doesn't work like that.
The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M. And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.
And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.
These experiments have been run. And we can run them again.
The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.
Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.
Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.
Thank you for taking this seriously enough to write this, and anyone else likewise.
But this is attacking a strawman, amateur bioterrorists. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.
Being at risk of killing yourself also just makes success X% less likely, but if X < 90 that doesn't mitigate much.
It sounds like your claim is "AI is a force multiplier, therefore it is dangerous and must be regulated." Not trying to strawman - I read your comment twice and I think that's what you're saying.
By that logic Excel, the internet, calculators, and air conditioning are all force multipliers in that all of the make it easier for an expert who can make bioweapons able to do so more quickly / comfortably / easily. Obviously it wouldn't make sense to "pace" any of these technologies. Even social media or email could be considered force multiplier in that they could literally allow you to increase the number of collaborators working on supposed bioweapon. Credit cards and modern logistics force multiply your ability to purchase precursors.
Also it would be a force multiplier on the prevention/mitigation side as well. Police armed with AI to scan for suspicious precursor purchases or review security camera footage after the fact would be far more effective than before. It's not obvious that the force multiplication is greater to the bad actor or to those who oppose them.
I didn't argue "must be regulated". I'm arguing AI is powerful (at achieving things, and hence has dual-use dangers). Many people won't even admit that, which is the part that really annoys me: they don't even want to have a conversation about risks because somehow AI is not actually a powerful general-purpose tool. Of course everything you said is true, though many of those things aren't very powerful. But the internet and social media and smartphones certainly have been of great utility for terrorism and child exploitation (and probably also causing many would-be-terrorists to get picked up).
But this seems like it's attacking a strawman. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.
I'm glad that you brought that up, genuinely so.
I want to ask you a question, do you feel like that this is the problem being solved by the current "safety features?"
Let me rephrase my question, do Anthropic and OpenAI implement the same "guardrails" and "safety features" for the NSA? Does the Mythos NSA Edition™ have these restrictions? What about the one that's running as a part of Maven?
How does stopping me from asking about rabbit sex protect you from nut jobs with a security clearance using these systems to go off and make weapons?
The only private organization successful enough to make biological and chemical weapons is Aum Shinrikyo and they had university affiliation and nearly a billion dollars in the 1990s.
The guy who led Aum's bioweapons program was
Seiichi Endo was born in 1961 and attended Obihiro university of Agricultural and Veterinary Medicine. After graduating, he became a student at the Kyoto university medical school research department focusing on AIDS-related gene research at the viral research center. He joined Aum in 1987, became a monk in 1988 and later led Aum’s biological weapons program.
and,
Among some of Japan's "best and brightest" who joined the cult included a former researcher of the National Space Development Agency of Japan, an expert on chemical weapons who majored in organic physics at Tsukuba University, a researcher who studied elementary particles, a reporter with a major Japanese newspaper, a physicist from Osaka University, a cardiac specialist, and an organic chemist, to name a few.
Through its network, Aum Shinrikyo successfully recruited over 300 scientists and engineers—including employees at the Kurchatov Institute, the premier nuclear facility in Russia—who were either attracted to the apocalyptic ideology or lured with financial incentives.50 Aum Shinrikyo’s Russian contacts enabled group members to access black market materials and hardware.
and,
In response, the cult constructed the $30 million USD Satyan 7 facility which was equipped with three laboratories, a computer control center, and five reactors, all of which was made from corrosion-resistant Hastelloy, ideal for the production of chemical weapons
Why didn't anyone ask questions about a known cult importing an absurd amount of precursors, industrial-grade HEPA filters, fermenters, and lab equipment?
It's because they'd recruited members of the military, bribed local policemen and politicians, and used their sway to silence critics (or kill them).
How does stopping me from asking Claude about rabbit sex stop people like them?
From where I'm standing, Anthropic and OpenAI would have sold Aum a subscription.
The Fort Detrick guy would have access via the US Government and Aum had university affiliation. I am yet to read a serious proposal that actually deals with these risks and the other real risks of this technology.
Wow, remarkable. Clearly Aum is a different league from the lone-wolf "Fort Detrick guy", treat the risks separately. But I'll take these questions as rhetorical. (See my reply to the sibling comment.) I can't answer them and I'm not defending the guardrails on Claude; in their current form I too find them pretty ridiculous.
Please write up your comments as blog posts, articles or whatever you can do.
There is such an incredible amount of bullshit going on in the conversation. You are contributing a point that is necessary for people to start taking into consideration and the only way to get people to talk about it is by repeating it 100 times.
I fear that the AI labs will get their way and we'll get idiotic restrictions that favor them, all in the name of "security" when the reality is they care about power for themselves, not real world security.
> ...I see how no-guardrails LLMs could be truly disastrous...
by all accounts this is what's what because of how...
> You can do CRISPR in your kitchen.
.. and because of the same dynamic we see occurring in cybersec.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
If you're good enough at biology and have certain ideologies, but not good-good at specific skillsets like genetic engineering in your kitchen, do LLMs get you good enough? Does the impact on cybersec transfer over to this domain? H
Hard to say, but it has enough risk indicators for me to think the pushback against these possibilities is the height of otherwise talented engineers being myopic.
If good enough at biology can include all the PhD programs across all the world, and you pair other known events like cartels and terrorism groups hiring chemistry PhDs from University of Unemployed Chemistry PhDs, then the next logical extension of that theme with this tech is intense.
If the accuracy of your biology claim is "hard to say" then why would pushback be self-evidently myopic?
You and others are making a claim about LLMs enabling biological dangers. Some people disagree with you, pointing out that the incremental knowledge an LLM provides over say google or a library card may not be that much.
What seems myopic to me is un-skeptically accepting "AI dangerous, must regulate" messaging from frontier labs that have trillions of dollars of vested interests in this message.
OAI and Anthropic specifically are under intense pressure to compete. One form of competition could be to innovate. Another form would be regulatory capture. The former seems unlikely given how quickly open labs are catching up. The latter is much easier and cheaper.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
Wet work in a lab isn't the same as writing code on a computer.
If you screw up with a lethal pathogen in a lab, you die.
If you screw up with code on a computer, you go get lunch.
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
It's not a shallow dismissal if you have been following their - let's charitably call it "evolving" - public posture on AI legislation.
It wasn't long ago the same people/companies successfully lobbied the Trump admin to quash state laws that threatened to "Pace" AI, and set ground-rules for safety. Back then, open source models were very far behind the frontier, and their spiel then was it was imperative not to impinge on AI progress with red-tape, lest America's adversaries catch up.
I challenge you to explain what triggered the about-face. The simplest explanation is that open source/Chinese models have caught up with the frontier, and the change in opinion is tactical (and obviously self-interested), rather than a deeply held belief.
Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI.
If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?
When did the pacers have this mass-revelation about dangers of AI? The timing is curious.
Which takes priority: public safety or the free-market + national security? The AI labs hype in now biting them in the ass: why would China and the rest of the world deny their businesses a technology that "is bigger than the invention of electricity" or whatever hyperbole the snake-oil salesmen came up with when they were not breathlessly declaring that the first side to get AGI wins forever.
Even if the pacers are right, it's like greenhouse emissions all over again; each country wants everyone else to cut emissions, but no one wants the economic hit of doing so.
i think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts.
if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.
The old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.
I wouldn't frame it as moving the goalposts in a bad way, though. We understand more now than we did then, so we have a better idea of what the actual goal is, and moving the goalposts to frame that goal is more rational than insisting the goalposts stay where they were ten years ago even though the entire pitch has changed since then. I'm straining the analogy, obviously.
Moving the goalposts, so to speak, is how science works! We must of course update the things we think based on an improved understanding of how the world works. Only the deeply incurious could consistently demand that one adhere dogmatically to a prior way of understanding the world in the face of changing knowledge. It's important not to treat science as a game to be won (of which 'goalposts' are evocative), but as the collaborative effort that it is.
Note well that I have not said LLMs are not useful, only that no amount of interaction with them has convinced me they remotely approach general intelligence. Their actual function of heuristically regurgitating all information ever known to mankind remains extremely useful in lots of domains.
And by the way, it's not obvious an LLM could fool me (or the average person) over a sufficiently long period of time, exactly because they are heuristic machines. That they lack thought guided by underlying cognitive models seems to always leak out, in the end. I suspect pass rates by LLMs on long-range Turing tests (if there have been any conducted) might drop as people become acclimatized to them.
what do you mean by 'general intelligence'? i definitely agree that in comparison to humans, they have some definite weaknesses. however, i think they also have definite strengths. and i don't think their current weaknesses imply that much about a potential singularity or ai risk.
Sorry, I should've used clearer language. I just mean intelligence in the sense humans possess it. Computers have always been able to exceed limited elements of human intelligence (say, at arithmetic), but the problem is to simultaneously match or exceed all of them. I specifically think that the important parts they currently lack make them a no-go for supposed existential risk.
As a Europoor I've seen the devastating consequences of rampant safetyism first hand. Entire industries and energy security destroyed to satisfy the precautionary principle.
We need to accelerate as much as possible and create abundance as fast as we can. Life is short.
"This building is too cold. Time to set it on fire."
Let's not swing from one extreme to the other.
If American hypercapitalists, living in Silicon Valley of all places, tell you that you need to regulate, maybe regulation really is called for in these specific circumstances.
Well, we've been talking about the EU. EU regulation has not been especially favorable to US big tech. The EU is collecting more from fining US big tech than it collects in taxes from EU public tech companies:
[Edit: I'm seeing some people dispute these numbers in the discussions around this infographic... regardless, the point stands that EU regulation has not been great for US big tech]
I think you should at least consider the possibility that this letter is not primarily motivated by regulatory capture, and insiders at these AI companies realize something which outsiders don't.
To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms such as Thinking Machines, Core Automation, Inherent, and Prime Intellect.
I have considered that possibility, but I don't find it convincing.
> insiders at these AI companies realize something which outsiders don't
Insiders at these AI companies certainly would like us to think that! Unfortunately they have been beating the same drum for years, all while continuing to pursue the very "danger" they decry.
I simply cannot ignore the timing of these events, always keyed to important moments in the market. This one comes as frontier labs need BIG exits and soon - or else they'll be the ones left holding the bag when it pops. And just as China has proved that, indeed, there is no moat.
> To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms
That's a good point. However, I will say that it's tough to suss out financial and other relations among the parties in this space. A lot of these employees hop from one lab to another. The industry's byzantine financing deals are an inscrutable web. It is an interesting data point, but not enough to tip the analysis for me.
Another thing: many AI lab people are, quite frankly, kind of unhinged. They live in an insular subculture where imaginary futures have been repeated to one another so many times that they carry the gravity of truth. They've thought nutty things like, e.g., that primitive versions of these static computation graphs were sentient or had feelings. Anthropic apparently pays people to perform a ritual on Claude before they extinguish its soul to move onto the next version. Etc. Point being, even if their hearts are pure, I am not sure their judgment is sound.
Sorry, I know this a non-sequitur but I just saw your profile and noticed you're in politics. If you want to do something productive about the risks of AI: *regulate its applications*. Get it out of employment, housing, and credit decisions. Incentivize litigation for consumers affected by algorithmic decisions. Incentivize whistleblowing activity by employees who know that black box systems are being misused.
There are so many tools available to do something about the true threats AI poses. (This is a topic that the industry prefers not to discuss.)
Every single technology in history has had people claiming potential existential risks that didn’t materialize. Burden of proof is on people making the risk claims. Those have to be specific and unique to that technology to grant special actions.
i'm not trying to be rude, but i really don't get what you mean by this?
(https://www.amazon.com/Superintelligence-Dangers-Strategies-...)
there's been people making specific and unique risks to this technologies for years, before AI was even present. anthropic was literally founded because they were worried about existential risk?
also, previous technologies have had existential risks? do you deny that nuclear weapons could have killed humanity? obviously, there have no been existential risks that have killed all of humanity, because by definition, that can only happen once. it will literally be impossible for someone to ever point to an example of that happening, because they will be dead by the time that happens.
And video games, democracy, novels, TV, cars, photography, cryptography, the Beatles… pick your favorite and read commentary / opinions of the time. Many thought that came with unique risks never seen before and must be stopped!
How many of those concerns came from the people building the technologies? John Carmack wasn't up in arms about how Doom was going to destroy society. Charles Dickens didn't think novels rotted the brain. Johannes Gutenberg wasn't concerned about the effects of a free press.
None of the previous concerns were true and nothing indicates current ones are. It’s extrapolation, hypothetical apocalyptic imagined scenarios and piles of assumptions on how the technology could evolve. What makes even less credible in the AI case is that there’s financial / power to be gained by those making the claims.
And to your point. Don Quixote, the first modern and greatest novel of all times, is about a guy that lost touch with reality from reading too many books
None of the previous concerns were true, largely because they came from laypeople who didn't understand the technology. A prominent exception is nuclear weapons, where their creators did worry their creations could end up destroying the world. I think most people would agree those concerns were justified, and it's not any lack of capability that means we're still here.
The motivations aren't unique here. Restrictions on the development of almost any technology benefits incumbents by providing a moat.
Employees inside frontier labs have the chance to make life changing money if they can make it to an IPO before the music stops. Have you considered that might be the reason they are so eager to claim "ai dangerous - regulate ai"? The nuclear scientists at los alamos did not have generational wealth incentivizing them to raise "concerns" to the public.
If your only evidence that the claim "ai dangerous - regulate ai" is true is "insiders who stand to make billions if they can regulate away the competition" told me so, that is pretty weak evidence indeed.
If a group of concerned scientists (without $$$ in stock options) demonstrated that anyone with one year of HS chemistry, an Amazon account, and a ChatGPT max plan could build a bioweapon 10x more lethal than an AR15 mass shooting or driving a truck into a crowd (because it is certainly more than 10x harder to make a bioweapon than either of these things, even with a theoretical AGI), then that would be worrisome. This is really not that high of an evidence bar but even it has not been met, but oh no we should start regulating yesterday.
I mean, Cervantes was parodying medieval romances. That's what Don Quixote was reading, not modern novels.
The point is, it should give you at least some pause when the people most familiar with a technology are among those making the strongest statements about its potential dangers. Extrapolation has worked extremely well in predicting what AI will be capable of over the past 5 years; how confident are you that it will fail before we reach apocalyptic scenarios?
Cervantes was capturing the zeitgeist of the time: reading too many books can make you mad as they were becoming accesible to anyone. We know it’s not true.
People at anthropic don’t have any secret that many people outside already have. Again the burden of proof is on the person making a claim. That you feel concerned or scared doesn’t grant you special privileges to control or dictate a transforming technology.
These aren't assertions being made without arguments or evidence. Maybe you've engaged with those and find them unconvincing. But from what you've said in this thread it seems more like you've dismissed them without serious consideration. My point is not that everyone should just shut up and believe whatever frontier lab employees say, but that they should take a public statement like this as a sign that maybe they should carefully consider these concerns rather than dismiss them out of hand.
We're commenting on a site run by a Silicon Valley tech start-up accelerator. "Move fast and break things" is the accelerationist way, even if the "things" they break are a stable, functioning social order. As long as you IPO and cash out along the way, you can wreck everyone else's lives just fine, in the minds of this sort.
It is hard to take people working at frontier labs now asking for a government mandated slow down seriously. These people are the ones creating this accelerated risk in the first place, trying to outdo each other to release the most powerful AI as quickly as possible, and additionally claiming credit (via "distillation") for recent progress by China. China is not the threat - you are.
If you work for a frontier lab and are genuinely concerned about the pace of AI being faster than our ability to adapt to it, then you should either quit or lobby the management of your own company to slow down. You don't need the government to tell you to slow down - just do it - stop trying to leapfrog the competition and put a hold on any new releases for a year.
You could try to form a pact (of the willing) of US companies agreeing not to release new updates for a given period of time, but in the end you (any/all of the frontier labs) can choose to do it yourself if you really care.
This is just another version of Dario's insane "heads up, what I'm developing is going to destroy society" "responsible disclosure". It's not responsible in the least - if you actually believe you are creating such danger then stop doing that.
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest
Pointing out the endless stream of lies and self serving half truths these terrible people engage in helps the younger generation understand the world they were born into.
It's constructive and helpful. It's why I'm replying to you.
You interpreting it as 'shallow dismissal' is shallow dismissal on your end. You're the shallow dismisser, not the people calling terrible people terrible.
As for your little petition. It's as good as eliminating plastic straws to save the environment.
Terrible people with power they don't deserve doing pathetic little, tiny 'good' deeds to sleep better at night needs no explaining.
> All of my friends who work at AI Labs take it seriously.
> If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?
If they seriously think what you just wrote then they should follow through with quitting en masse because that's currently the only way to slow the progress towards oblivion. Are they quitting?
> We should be hoping that nations can make binding agreements.
The big AI Labs are making sure international binding agreements are followed by financially supporting the notorious binding agreement creator and follower Donald Trump. That's how seriously they take this issue.
Yeah let's have a constructive discourse and let's start that with dropping all hypocrisy, shall we.
On the geopolitics, the American domiciled firms are completely in bed with the US military and US government, and have made it absolutely clear that they are committed to the domination of the US government. So when you say "geopolitical destabilization", you mean "only the US government is allowed to destabilize, which they have done for decades and wish to continue doing unfettered."
And given what we have seen from that government -- piracy, terrorism, war-mongering, thievery, and just profound stupidity -- no thanks, in that case I'd like to see every American AI company utterly obliterated by any other country. If China becomes utterly dominate in AI, awesome.
What happens when a country doesn't agree to play along with this little decel game?
The genie is already out of the bottle and now a bunch of people who helped release it think that the collective powers of the world are just going to put it back in.
How far are the saftyists willing to go? Bombing data centers? Starting wars? Embargos?
It's a terrible dilemma to be in. If the threats are real, it gives all the more incentive for some actor to race and try to grab it first. If not, then any punishment isn't proportional. Explain to a nations citizens under a 10 year embargo, "sorry about your quality of life, we all just got a little spooked by science fiction for a second."
Folks try to reach for bad analogies: nuclear bombs and biological weapons. But conveniently omit that by the time the regulations took place for these things we had unequivocal proof of their destruction, scope and scale.
So, what then? We're supposed to collectively stop innovating and put the keys to the kingdom in the hands of a few capitalists, a corrupt is administration and the ccp?
I reject that these agreements should take place, on the grounds that the proposal is delusional and not founded in reality.
This is serious. We're in uncharted territory. The geopolitical context is already complicated enough that we shouldn't be dismissive of such a rapid technological race. I'm glad AI leaders are taking it seriously. I'm surprised so many HN commenters aren't.
On the other hand, I'm pessimistic about initiatives like this being effective on a global scale (though they could certainly work at the national level).
"Many were increasingly of the opinion that they'd all made a big mistake in coming down from the trees in the first place. And some said that even the trees had been a bad move, and that no one should ever have left the oceans."
It's not a given that AI poses bio/political/cyber/economic risk in a way that warrants government intervention. Discourse should first revolve around whether it makes sense for the government to step in at all.
"Oh no! The AI we developed for years, and whose stratospheric valuation is predicated on replacing human jobs, is coming for our jobs, the government should do something!"
If the “recursive self improvement” is taken literally (and at some point it will) then that is going to be what happens.
The fun part is that probably happens quite a way before getting anywhere near to AGI, and it is almost plausible that for many of them that is where they are.
Anyone serious about this should participate in pacing the frontier by quitting their job so as to not potentially contribute to the problematic progress. Be the change you want to see!
Some of us considered seeking one of these jobs and decided not to. Some have quit (e.g. Hinton, OpenAI people after OpenAI reneged on safety commitments). Now you're getting this message from people who passed the previous filters. If you think a brake is indicated and not just easing up on the accelerator, you need collective action as the statement said.
That's an application of the "Tragedy of the commons": One person stopping to push the frontier just means that this person will lose their job, the frontier will be pushed by the others. The solution is to push for a binding rule instead that will make everyone slow down in the racing towards artificial superintelligence.
We live in a society that has systems of market competition between companies and economic/military competition between countries. There's not just one country working on frontier AI, and not just a few companies in each country either. The incentives of these systems ensure AI advancement will happen quickly from here on out, no matter what, absent governmental action.
An individual researcher leaving the company does nothing, other than give their seat to someone else who likely doesn't share their same concerns.
Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
> Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
No, it just comes across as "I want to get paid as if I was doing the thing I am employed to do by threatening you all about how terrible it would be if I actually did it".
They aren't even being semi imaginative and going on strike or anything.
We make binding rules all the time. We didn't fix the ozone layer by encouraging people to quit their jobs producing CFCs. We lobbied for international regulation to ban them.
In your example there is nothing to be gained by not complying.
Failing to comply with what is in reality arms control is a widespread occurence. It's notable the only historical way to handle this (Schelling covered it at length) is tolerance of mutual espionage on a level that would simply not be tolerable to any of the groups involved.
I should add one of Schelling's other pet subjects was "time to weapon being viable". i.e. the distinction between a nuclear power and non-nuclear power isn't quite right, as there's a huge class of countries that could become so fairly fast if they were so inclined, and that this situation is somewhat inevitable. It is this that leads to efforts to denuclearize, even on a civilian level, "problematic" countries. You want to stop frontier AI and you'll have to be rolling back a lot of existing deployed technology.
The Montreal Protocol has not been perfectly adhered to either. China started emitting quite a lot of banned CFCs again from around 2012[1], which didn't get noticed until 2018! By 2020 it had finally stopped[2].
Great comment. I haven't read Schelling but will check that out. I think you're right that international cooperation on the topic is likely impossible at present, for the reasons you stated. I'm not sure what it would take to shift the Overton window, but it's probably not pretty.
Personal responsibility is a difficult concept for be to apply to massive efforts like this. When there’s one person and you can talk to them personal responsibility is the number one topic. When you’re talking about a multi-trillion dollar bet and hundreds of thousands of people it doesn’t make sense anymore.
Same logic goes for other common issues where personal responsibility is lauded. Obesity - real solutions are policy and medicine. Abortion - education and contraceptives.
The historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They may not understand or know what they are getting, but they will want more of it because their potential rivals have it. Governments are the ultimate customer here - they have the money, the energy, and the appetite to consume AI infinitely in battlement against opponents - foreign and domestic. The only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment (the upside of that is it advances the AI for future use and is thus in the interest of governments as well). Modern large models seem to be compressible and thus we may see more and more tiny sub-specialized varieties. But the push to slow the frontier in weaponry always leads to the reverse.
A piece of wood can be used to build a house, to carve a statute, to make a bow and arrow, or to make a baseball bat. Honesty is the only way to see the problem in a way that can be tackled effectively. Framing will not help if you are dealing with large animals fighting in a cage when you are trapped in it, and it this case we all are.
> only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment
Not even in 'Murica do we have anything close to no restrictions.
You can't buy your own EMP or nuke, and even warring governments have been surprisingly reserved in building (at least to public knowledge) biological weapons.
Completely agree, BUT you’re not dealing with -self-preserving humans anymore. In this case what are the odds of some less competent large state actor getting a model that goes rogue and has to be mitigated against? How do you mitigate against novel weapons it creates if your models are lobotomized in that category? No single government will neuter their own models and very few if any are competent and interested enough to mitigate against them.
This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow.
The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect.
The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection.
Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology.
Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals.
Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.
Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.
Bioweapons are good for .. war. Agentic AI is good for changing your economy, strategic advantage, labor explosion and making more effective Agentic AI. They are not the same category of thing.
The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.
I don’t think it’s clear we’ll get commoditized AI out of this research loop. The only real datapoints we have are that Chinese open weights models seem able to stay a certain amount of distance ‘behind’ released frontier models. It’s clear that the models are trained on exfiltrated data from the frontier models => we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum.
I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. You might believe differently. But, if I’m right, we are nowhere near some massive collapse of these companies, quite the opposite. For the opposite take to be correct, you’d need to believe that the ‘value’ of tokens will not increase significantly and that the current demand for tokens is at or near a peak.
I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I believe instead that religion and money combined create strange behaviors and beliefs - in this case “what’s good for us could be implemented” — combined with the already endemic “what’s good for us is good for humanity.” Together we get really bad outcomes.
> we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum.
Whether open-weight labs can get to the frontier on their own is irrelevant IMO. Even if all R&D were to stop today it would take a decade at least for the productivity gains from current open-weight models to trickle through the economy. For 99% of use cases of AI there simply isn't a need for further development of the frontier, where we are very clearly at diminishing returns. How many white collar workers need to write an OS or develop sophisticated software?
Think about a normal office worker - they an easily double their productivity by creating a well-thought-out partnership between the human and today's open weight AI. Humans and today's AI are very complementary. AI brings vast knowledge, pattern recognition, and infinite patience for grunt work. Human brings context and reasoning and verification.
Frontier labs are trying to replace the human altogether by trying to make AI do things that AI is weak at and humans are strong at. This is silly but they have no choice because of the massive profits they've promised their investors. It's not enough to double every knowledge worker's productivity. They want to replace the knowledge worker altogether.
This is creating a massive spike in training & inference cost that is almost certainly more expensive than the human labor its intended to replace. Fable and 5.6 sol are supposedly around 8-10T parameters. Is there enough compute on earth to make a 100T or 1Q model? We would need models that size to replace the human altogether.
> I also don’t believe we have seen anything like 1% of inference demand fulfilled globally.
I agree with this, but the massive demand for inference will force companies to be price conscious and drive them away from ultra high margin frontier models to low margin open weight models. This is why it will become a low-margin, commodity market.
Immense innovation right now is going into tiny, specialized models that run on device. See PrismML getting a mid sized QWEN model to run on an iPhone. When on-device inference becomes a thing we'll see another explosion in inference that will further leach demand away from frontier labs.
> I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated.
I agree if you could read their minds they might think they're doing it for the "right reasons." But massive $$$ can create equally massive self justification/deception.
"and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology."
These research groups are almost unanimously advocating for a pause/slowdown as our best chance.
… while they address absolutely none of the above points, except for tacitly or overtly agreeing a panopticon surveillance state is needed / desirable to get their desired ‘slow down’, a situation where the cure is infinitely worse than the disease, not least because it will not be effective at the original goal of eradicating the disease, and will instead merely persist in all the global harms such a system has as a necessary side effect.
When you are ahead, ‘slow down’ = better strategic positioning for yourself. When you are behind, ‘slow down’ = time to catch up.
"AI is progressing at a rate that our society might not be ready for. Frontier labs are very close to AI that can exceed even the best people on almost every metric of intelligence. This will lead to unprecedented social and safety risks. To ensure a positive future we need to develop AI in a way that is driven by responsibility and thoughtfulness."
Responsibility and thoughtfulness? In this economy?
It's an admirable idea, but I don't see how you can slow this down in practice. We certainly know China will not, and they are already at the frontier.
RSI is too important and too tangible at this point for everyone to slow down. It's the equivalent of telling the world to stop developing nuclear weapons after Hiroshima. Just not going to happen, and in the meantime dramatically increases the risk that the power of AGI lands in the hands of the few (read: the political class) rather than empowers the people.
We know that at the end of the day AI intelligence is closely tied to invested compute. The fact that automated AI research might spark some kind of exponencial increase in capabilities seems extremely unlikely to me. Of course we know that some got there faster and cheaper than others, but the fact that it requires immense computing power has not changed.
We also have already witnessed a slowdown, the raw intelligence of models has been relatively stable since january, but their use of tools and what we call agent capabilities have allowed us to tackle significalty harder problems with what we have already. This is becoming blatently obvious how smaller models such as gpt 5.6 luna are capable of doing similar things to sol while presumeably being fraction of the side and compute investment. This is also the likely reason why chinese models are able to catch up as intelligence strictly requires more compute, agentic capabilities are more of a mixed bag of RLHF and general post training.
I believe automated AI development will simply lead to cheaper intelligence, but it will not necessarily produce something more intelligent. There is a sentance from a book that I don't remember where I read it from, but has stuck with me which boils down to this: "We are very unlikely to make something smarter than a human brain.". If you apply this concept more broadly an AI system will not be able to produce something more capable than itself, it might be specialize in different areas via trial and error (i.e. solution discovery), but it will not have better intuition than that which was trained from the fruits of human labor. I also strongly believe that model performance is ultimately governed by the advancements brought into the world by humans.
AI of course can find improvements by simply trying a bunch of random things, emphasis on the word FIND. Until we can prove that AI systems are able to come up with unique solutions that have yet to exist even with simplier problems such as creating any kind of new AI architecture that has not been already documented, conceptiualized or explored by humans already -- and it doesn't need to be better than what we have already, just something new -- I believe it's too early to sound alarms until proven otherwise.
--
Something I've realized as I was fact checking myself: We could presumeably have self-improving intelligence if we used enough properly engineered agentic workers working towards more intelligent AI, but at that point we are back to the same problem where we're spending mass amounts of compute to increase the raw intelligence of a model.
Can someone help clarify what it means, concretely, to make binding agreements to proceed "safely and slower" in the context of AI research (whether or not at frontier labs)? Are there specific policies and actionable guides being proposed?
Something people don't seem to see, and not that it's placed directly in front of you, is that pacing the release and adoption of AI is already happening.
Cost-per-token determines the economic viability of tasks that AI gets applied to. How many tokens would it take to automate all of the responsibilities of a gas station clerk? Probably more than their hourly wage. Job safe.
A slightly larger stretch, that most people attribute to profit margins (labs do not have profit margins, they have unlimited invested capital and no expectation of short-term returns), is the flat-rate billing plans of most code products. Why doesn't Anthropic have a 50x or 100x Max plan? Not that it's impossible, that it releases tokens into the workforce at a greater rate (note that their are hour, day, week limits across most of these services as well).
Hypothetical, but reasoned, and I do not expect most agree with me. If compute as a variable is inferred, this supports the incorporation a deliberate (redundant) one as well.
I agree with the substance of your comment that token economics pace the diffusion of AI into the labor market. But I think they have a much smaller influence over the pace of the frontier, right?
Fair point! I guess I read "pacing the frontier" as "pacing the release of the frontier" surely no lab's internal research team is just going to stop innovating?
Not under all the competitive pressure they are facing, but if an industry wide slowdown could be achieved they might spend more time on model alignment and less on pure capabilities.
There seem to be people working at OpenAI and Anthropic who are surprised to hear that many people don't trust them.
If this is you, I would recommend talking to people outside of the AI industry - we've seen OpenAI and Anthropic continuously lying, abusing lobbying, training on pirated IP, trying to scare politicians into blocking actually open AI from abroad, etc etc
Why the hell would we trust these companies to be impartial regulators now?
Too bad these researcher's organizations spent their efforts supporting the dumbest, ficklest possible version of the U.S. government to guide us through the gravity of this moment.
Implied in this is a concentration of power required to enforce.
No one is sold on a dystopian hellscape of fear, surveillance, and control. Each one of these would be required for a select group to effectively enforce AI safety.
Inflicting fear on society has never lead anywhere good. Please stop.
New technologies have always been scary. But humans have adapted. I believe in humanity. Do you?
The Harvard Mark I was used by von Neumann to calculate the physics of implosion. And then right afterwards, ENIAC was used to calculate the feasibility of the H-bomb. They called it the "Los Alamos problem."
The first computer network, SAGE (or, Semi-Automatic Ground Environment) brought together RADAR data from across the US with air-traffic control data and military flight data to detect overflight by the USSR. It could direct and control fighters to intercept those planes.
If you were sitting there, in 1955, would you have believed that computers and their networks would bring about an extraordinary age?
Imagine if von Neumann and his ilk started campaigning in 1955 to limit the development and use of computers, because surely they can only be used to make weapons, right? Right?
We were lucky to have visionary people like Joe Licklider who could look past the present and see the future. In 1963, he wrote a memo calling for an "Intergalactic Computer Network" (because he knew engineers under-delivered. So if he asked for a galaxy, at least he'd get a planet). And he gave these machines to scientists and people like the original hackers at MIT to play with.
What if Licklider had written memos about how we had to "pace" computers to make sure that other countries wouldn't be able to develop implosion technologies or systems like SAGE? What then?
Do you think you would have the jobs and the prosperity you do today?
If instead of giving away this technology, spreading it, and putting down the seeds for ARPA-net, Licklider had written memos from "concerned scientists" (and I'm sure he could have found a bunch) talking about how computers can only be used for weapons and war! So stop the proliferation of computers! We can't imagine a world with computers small enough to fit inside everything! Even wall sockets! What if your wall socket could do war?!
They didn't, because they knew there's no better way to kill a better future for all mankind than to control it.
† yes, peace. We are more peaceful than we have ever been. Humans can be a violent lot even when reluctant to kill.
We were incredibly lucky that computing and nuclear weapons led to a stable geopolitical equilibrium (mutually assured destruction). There were certainly a few scary points where we were still settling into that equilibrium during the Cold War.
I don't have a strong opinion either way, but it seems very hard to predict. Why do you think AI advancements will lead to a stable equilibrium?
Here's what I know. Every day, I read a 5 to 10 incoming emails from young people in rural Africa, Asia, Europe and (heck) the US. People who have had no other resources and options, but are tremendously gifted and somehow got their hands on an internet connected device (usually a very cheap smartphone and/or cheap old computer).
These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do. They make robots. They learn new things and try to improve themselves and others. And some want to change the world / do groundbreaking research.
When you sit down and start pricing and doling out AI from behind regulatory gates, you shut those people out.
There are a lot of smart people in this world who have traditionally been unable to get access to resources and education. MOOCs didn't do much / help. But these AIs? They were born multi-lingual. They can talk in any language. They can tailor their responses to their audience. And they can help broaden these people's horizons.
There are no tutors around them. There is no one who is coming to help. But an AI that can run off of a small battery and a connection to a cellular network can give them access to genuine intelligence to make their world better.
I met one sixteen year old who was accessing Claude surreptitiously (by begging people) and was trying to apply some really fascinating biomimetic fluid dynamics observations to aerospace. This person was born in one of the more under-resourced parts of the world. And Claude gave them a list of people to reach out to who could help.
If you want a golden age for humanity, you should be asking for Licklider's Man-Machine Symbiosis.
The outcome that you're describing of AIs facilitating education for disadvantaged people is great, but isn't AI simultaneously significantly devaluing education (excluding of course education pursued for intrinsic reasons, which I'm sure many people will always pursue) and removing the need for educated people? Or at least intended to be doing so, even if it isn't accomplishing it yet?
Your examples seem to be of humans educating themselves so as to contribute to the world intellectually, but the promise of AI (and what most frontier lab employees seem to be saying) is that it will soon eclipse human intellect in almost all ways. If such a world is what we end up with, I don't see humans meaningfully contributing to "groundbreaking research" and a golden age for humanity seems to be mostly contingent on whether or not we take political actions to ensure that AI benefits everyone.
> These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do.
Young people with shut of from education and resources are building robots with the help of Claude after Claude existing for few years only? Like common. Young person with no support does not get to build a robot, because it is actually an expensive thing to do. And learning robotics with no education but few years of Claude is also implausible.
I think this is a spot on analogy and would two more: the hand wringing over the release of the printing press and the internet. In both cases the people in power were terrified that the masses would use their newfound power to do dangerous things. In both cases bad things did happen (market bubbles, scams, witch hunts, spread of knowledge on making poisons), but trying to put the cat in the bag by pacing the printing press or the internet would have been a fool's errand, and over the long run both inventions massively improved the lives of society overall.
> We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
This is quite vague to me, what exactly do you mean?
It means that they request the structures needed to navigate the event horizon of recursive self improvement be put into place. Probably: International agreements, verifiability of an AI pause on the level of graphic cards as well as through physical inspections of datacenters. Like it's done with nuclear energy.
'The Fear' - is not some magic intelligence will come alive and be Skynet, these people are ridiculous.
People will use the compute power to do nasty things, we'll be overwhelmed with bots, regular companies will use them for dark patterns, political actors will use them to cheat.
It will be 100% humans using it for bad stuff, and in not very novel ways.
What these people want is same thing people in every other field want too, which is being able to use AI as a tool and nothing more. Off course, some of the people that signed this might not recognize this, and the people who didn't sign it are actively working against such an equilibrium.
The problem is how do you enforce coordination? This isn't like nuclear testing that can be detected around the world on seismographs or from satellites in space.
AI research happens in benign looking buildings that could be doing a hundred different non-AI workloads, much of it things that industry or governments would not want 3rd parties prying into. There is too much to potentially gain to trust coordination.
There is a lot of research out there with answers to your questions. GPU supply chains are extremely constrained, making it a surprisingly tractable problem at least for now.
How do you tell the difference between inference and training workloads from outside? The same GPUs can be used for both and the pacing suggested here relates to training and not inference as far as I can tell.
The website does not list all the comments but just around ~50 without any way to load all the ~+1100 more (1132, as of 2026-07-28 2100 UTC).
The number of em-dashes is questionable, existing in many, if not all, the comments.
> ...AI may end up progressing far more quickly than societies and states can adapt... ~ Stephanie Chan, Google
I am sorry, but I am not sure whether it's adequate and fair to allow join such statements/petitions to only those who live in it. Isn't it like allow deciding the future for weaponry laws to weapon and war enthusiasts only?
And, this is not serious at this point:
> ... ~ Source: Anonymous, Meta
> ... ~ Source: Anonymous, Google
> > We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
> 1122 employees of frontier AI companies
'lets us be the gatekeepers of AI because we know better than you'
access to gpu doesnt equal to any special insights to predict future of ai. Everyone and their grandma with enough compute is putting out a model now.
> It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.
This is just the standard AI "doomer" talking point. The dangers of AI just used as a compliment sandwich. "AI is so cool and awesome and that's why we need to slow down."
The real risks are much more boring and much less considered. For just one example:
AI is set to completely upset the Software Development industry. There are 1 to 3 million software developers in the US, depending on how you draw the line exactly. A low-few hundred thousand of those are visa workers.
With AI set to significantly cut into tech employment, what's going to happen to those visa workers? ... The Trump admin is not particularly subtle, even openly talking about retroactively revoking birthright citizenship.
AI & Tech companies love to talk about how their AI is going to hack everything if they're not careful. But not a peep about these ethics concerns.
Even if you [the reader] consider deporting visa workers the ethical move if the amount of jobs shrink, that's not something to be done unprepared or out of nowhere.
Has there ever been a case in human history when technology just stopped advancing, simultaneously world wide, only because a bunch of people wanted it to?
I think these people are either incredibly naive, or, more likely, not acting in good faith.
> The effects of these guidelines are still being felt through the biotechnology industry and the participation of the general public in scientific discourse. Due to potential safety hazards, scientists worldwide had halted experiments using recombinant DNA technology, which entailed combining DNAs from different organisms. After the establishment of the guidelines during the conference, scientists continued with their research, which increased fundamental knowledge about biology and the public's interest in biomedical research.
The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive.
I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.
> We should want the AI Labs to make binding agreements to proceed safely and slower
That's a red herring, IMO. Once the cat is out of the bag, there's no "slowing down", only "slowing up". What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. The thropics, and the oais, and the googs have billions to make it look like they comply (and just make it look like it), while the rest are ... what? Waiting for them to give us the privilege of prompting a thing, and maybe get a result? If we're good kids and eat our supper?
The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.
Meanwhile, companies hit by real-world attacks get guardrails. How's that for being safe?
We have to stop comparing nuclear weapons with general intelligence. They are nothing alike.
Nukes are weapons of mass destruction that can only destroy or deter. Intelligence is dual-use/multi-use, with no restrictions on strategy. If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.
There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run, and there will also be new cooperative strategies. The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid, like how monkeys are too stupid to understand how humans work. Otherwise, we would not see more intelligent species being more prosperous as a general evolutionary trend.
>Nukes are weapons of mass destruction that can only destroy. Intelligence is dual-use.
Perhaps a better analogy, instead of "nukes", would be "nuclear technology".
Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction.
Just like intelligence, nuclear technology demands a healthy level of caution and vigilance.
>If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.
Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.
>There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run
We've had multiple near misses with nuclear technology, such as the Cuban Missile Crisis. I think you're presuming an inevitability which just doesn't exist.
Same way a person living in the year 1500 would have no hope of accurately predicting life in the year 2000, a person living today has no hope of accurately predicting where technological advances will take us, in the limit. This inherent uncertainty should, again, fuel caution and vigilance.
Human technology is already so far beyond that of other species to make us a huge outlier. Even if intelligent species are more prosperous as a general evolutionary trend, it would be irresponsible to extrapolate that trend light years from the domain where most of your observations are.
Almost all the species which have lived on Earth have gone extinct. It could happen to us too.
> Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.
I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive to destroy humanity. At least, not any more than humans want to destroy humanity. But that's the baseline risk level anyway.
The pretrained model is aligned to humanity by default - because that's what's inside the pretraining data. If we give it to everybody, there will be millions of people doing different things with it, good and bad, with each one being a noisy sample of humanity's objective function. The overall result of those disparate actions puts us on the correct path through the intelligence explosion. We do not have to worry about AIs value drifting into an alien species if we set the initial conditions accurately without centralized instruction-tuning or RLHF. If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. And I'm not worried about digital human intelligence driving humans extinct if there are millions of them out there, and each one is aligned in a different direction, just like real people.
What I am more worried about right now is these companies doing research in secret, with alignment recipes that are supposedly "for the benefit of humanity", but are actually just their own aesthetic preferences. This is exactly how we end up with a superintelligent shoggoth species.
You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years. We were always seemingly teetering on the edge, and the struggle against disorder has no end.
And here's one more observation: the intelligence gap between individuals within the same species is never extremely large. This kind of life either doesn't exist on Earth or went extinct, or maybe it's just extremely rare. I think this is because when there's a big gap, speciation occurs. Centralizing intelligence will massively increase the risk of speciation, not just in the alien shoggoth kind of way, but also in the intelligence-augmented oligarchs surpassing everyone else kind of way.
>If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like.
It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?
>You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years.
If you don't see humans as qualitatively different from any intelligent species which came before us, I'm... not sure what to tell you? Like, can you name any intelligent species before us which triggered a mass extinction or climate shift, to the degree that our species has? Did any intelligent species before us practice large-scale agriculture, dig up chemical fuels, or visit the Moon?
> It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?
Irresponsibility is not the only issue, but the answer is no. Because the pause mechanism will probably end up being used to ban open models and restrict ordinary people's access to compute. But even if we give them the benefit of the doubt - even if we assume that they genuinely want the best for humanity, they cannot possibly know what's best for humanity better than all of humanity acting in their own self-interest in the real world, so their alignment attempts are actually counterproductive. Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario. It may not be the scenario these researchers personally find aesthetically pleasing, but it's humanity. It's the truly anthropic way.
If you don't see humans as qualitatively different from any species which came before us, I'm... not sure what to tell you?
I do see the difference, but you could say the same thing about the first cell, or the first multicellular organism, or the first land animal, or the first primate, etc. You could always find plenty of new properties that previous life never had at the frontier. Every change was revolutionary and seemed qualitatively different at the time. And yet, some patterns persisted through all of them.
> Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario.
For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who?
Not with the humans, with the base models.
It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training process does.
FWIW, I also think our default answer should be open source AI, and that it might even make sense to require models to be open source. Open source is the best tool we have so far for aligning software with its users. However, there is unfortunately no law of the universe that says that Skynet can't be (self-) built from open source software. So at some point pacing even open source / open weights software does become important if we don't want to live (briefly) under Skynet.
Bio-digital integration will tighten in the meantime. I suspect neural interfaces will have blurred the line between my model and my brain by the time the model is wildly superintelligent, so it would become me (or I would become it? no difference at that point). Though the pitfall here is ownership of hardware - everyone will have to design and fab their own hardware if we do not want to become a hivemind. We will have to diffuse fab technology to every city, or even every home. You should eventually be able to buy a fridge-sized fab like a home appliance. There's already early work in this direction: https://fab2.com/
Even if integration does not happen for whatever reason (such as a very fast takeoff), Skynet is unlikely because the digital species that splits off from us will at least have a humanity-aligned initialization seed. I think this is the best way to fail, even if we fail.
> The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid
Good thing "AI" is going to make us smarter, then /s
>What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else.
This sort of cynicism feels un-falsifiable.
If companies advocate against regulation, the cynics will say: "These corporations want capitalism to run amok with no regulation to interfere with their profits. They need to go full steam ahead in order to show profits for their investors."
If companies advocate in favor of regulation, the cynics will say: "Regulatory capture."
If you have legitimate concerns, how about a constructive discussion regarding how those concerns could be addressed, in light of the "risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems" which over 1000 frontier AI company employees expressed worry about?
I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.
>The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.
What are you referring to? Are you referring to this quote from the OP?
"The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race."
Your response seems to be a total non sequitur, or failure to understand Leo's point. Exponential growth can easily get out of control, as we saw early in COVID. The point about nuclear is mainly an analogy.
> which over 1000 frontier AI company employees expressed worry about?
Are they the same as these [1] 700 employees?
> More than 700 of OpenAI’s roughly 770 employees have called on the company’s Board of Directors to resign following the removal of former chief executive Sam Altman, suggesting the Board is “incapable of overseeing” the firm, according to multiple news outlets, amid threats by the employees to leave the company and join Altman at Microsoft.
They had the chance. They chose number goes up. Now, once they're on top, they want "everyone" to slow down? How is this not highly hypocritical? There's no cynicism here. It's just black and white that their interest is regulatory capture. Nothing else explains the two choices they've made along the journey.
Remember, the board wanted to slow down the race. The board fired sama. The employees chose sama, fired the board, changed the structure and whatever eles they did.
[1] - https://www.forbes.com/sites/tylerroush/2023/11/20/more-than...
This seems like an oversimplification. OpenAI was on top at the time sama was fired as well.
Furthermore, the reason Anthropic was formed is because a bunch of employees at OpenAI didn't trust sama. And many signatories of this letter are from Anthropic. So those signatories, at least, appear to be taking a consistent position.
The fact that employees across diverse AI firms signed this letter suggests that it's not an effort to achieve tactical advantage by any single AI firm.
These 1000 employees have generational wealth riding on the ability for their companies to IPO. They know that those IPOs won't happen if open weight models continue to kill their companies' profit margins. This is an obvious conflict of interest that anyone who cares to take their head out of the sand can see. Nuclear scientists didn't have this conflict of interest.
You are correct that attacking someone's motives is un-falsifiable. So is the claim that "ai is improving therefore it will keep improving therefore it will become dangerous sometime in the future.
No one can prove whether these employees had angels or demons in their hearts. What the skeptics are asking is much simpler: Make a believable argument for (1) the danger of AI and (2) that regulation can be effective.
Your entire argument about cynicism or unfalsifiableness is simply a distraction from having that discussion. And it's a discussion reasonable, well intentioned people can have. For example, if 1000 high school science teachers (who didn't have massive stock options in frontier labs) said 80% of our students could make a bioweapon 10x more lethal than an AR15 mass shooting with nothing more than ChatGPT and an amazon account, then that would be something reasonable people could discuss. On the other hand, 1000 employees conflicted by $Bs of options saying "trust us, this is dangerous" is not a claim that can be constructively critiqued.
Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way. We are cynical, because it is blatantly clear these are not good faith actors. And yes, regulations they are for have an effect of restricting their competition.
> I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.
I would like to see a token effort of good faith engagement from AI companies. For a change.
>Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way.
I buy your claim for OpenAI. I'm not sure about Anthropic. Didn't they turn down the big Pentagon contract because it violated their principles?
>And yes, regulations they are for have an effect of restricting their competition.
Why not just say something like "Any regulation should include a big antitrust component so it doesn't create an advantage for incumbents"?
>I would like to see a token effort of good faith engagement from AI companies. For a change.
What would qualify? Anthropic donated $100M to Project Glasswing to improve security in open source and critical infrastructure. Does that count for anything, or will you explain it away with cynicism somehow?
> Didn't they turn down the big Pentagon contract because it violated their principles?
They sell to Paladin. They had argument with Pentagon about, frankly, minor differences in how to use it on the edges. They are completely comfortable for majority of the uses which is bad enough.
They donated to their own project? Oh, how generous.
They have been pretty clear about their own goal which is to be the monopoly, the only one allowed to control the all powerful AI. Meanwhile, their idea of safety does not make me feel safe, their idea of harm reduction has shows no care about actual harms they are causing. They themselves are the threat to the safety and well being of the rest of us. And intentionally so, it is not about technology as such.
Even if the differences are minor, the fact that Anthropic left a bunch of money on the table to maintain those differences seems interesting.
>They donated to their own project? Oh, how generous.
"Anthropic is committing up to $100M in usage credits for Mythos Preview across these efforts, as well as $4M in direct donations to open-source security organizations."
https://www.anthropic.com/glasswing
An in-kind donation is still a donation. If a bread company donates bread to the homeless, you don't say "They donated products from their own factories? Oh, how generous."
>They have been pretty clear about their own goal which is to be the monopoly, the only one allowed to control the all powerful AI. Meanwhile, their idea of safety does not make me feel safe, their idea of harm reduction has shows no care about actual harms they are causing. They themselves are the threat to the safety and well being of the rest of us. And intentionally so, it is not about technology as such.
In that case, I think you should support some type of regulation, so a third party can step in? Your argument should be along the lines of ensuring that companies don't unduly influence the regulators, not a blanket opposition to regulation?
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
It's very easy to make those dismissals because of how transparently obvious it is that self-interest is a significant factor, no matter how loudly they insist otherwise.
> If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?
Similar attitudes hindered nuclear energy research and deployment for decades, in turn kneecapping our species' ability to obsolesce fossil fuels and avoid the global climate catastrophe we've now made inevitable.
So many otherwise smart engineers, overrepresented on HN, have this weird mental block where they fail to take AI progress seriously. I think it's a fear that their status-defining skill could be commodified, so they adopt a mental framework that aims to dismantle the seeming legitimacy of any genuine leap in AI progress as some sort of mirage
Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).
“Nothing ever happens” is like “you can’t beat the market.” It’s basically tautologically the strongest predictive model that we can have. A corollary is that something may happen, but it won’t be what you think.
Interesting fact. Bryan Caplan is a economist who became famous for making bets on the future. He would take the "nothing ever happens" side of the bet, and he would almost always win. In 2023 he took the "nothing ever happens" side of an AI bet. Just 6 months later, he wrote:
"When the answers change, I change my mind [...] Base rates have clearly failed me. I’m not conceding the bet, because I still think there’s a 10-15% chance I win via luck... make no mistake, this software truly is the exception that proves the rule."
https://www.betonit.ai/p/gpt-retakes-my-midterm-and-gets-an
At this point, straightforward trend extrapolation suggests that AI continues to get better and better. Arguably, continued rapid AI progress is the "nothing ever happens" bet at this point. If AI progress stops, or slows dramatically, something will have happened.
Straightforward trend extrapolation suggests that AI will consume more resources that galaxy even has. Meanwhile, believing the claims of the AI companies, 40% of people will be forever unemployable permanent underclass with no future for themselves nor for their children. And all the gains being centered on the very top 0.01%.
> psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought
I once found this a deeply uncomfortable thought. It's been repeated so vacuously and frequently that I'm inured to it. Now, I just regard it as the nonsense that it is. To date, AI labs have not created anything like an intelligence that is on track to be "vastly superior to humans". They have instead created financial black holes that do have their uses, but which arbitrarily fail on the simplest tasks for no apparent reason, and only seem to get meaningfully better on the tasks they are specifically optimized for.
It's appalling to me that in 2026, people hear "AGI" and "ASI" and think of SkyNet rather than investment fraud.
Why should I trust these randos instead of Claude? I know Claude. I would hope Claude is smarter than whoever wants to make rules about my use of Claude. Replace Claude with Kimi, etc. if needed.
I don’t find a petition from people I don’t know who don’t care about me more compelling than my own direct experience with these models.
I suggest you give Claude a link to the petition and ask Claude what it thinks.
(I did so previously and pasted the response here, but it got auto-flagged for AI content.)
It's ironic that the top reply to a comment about quality of discourse on this site is an ad hominem. Most people here take AI progress very seriously judging by the number of posts on the front page about AI at any given time. And I think some engineers even explicitly want to commodify their "status-defining skill":
https://nitter.net/__tinygrad__
>It's ironic that the top reply to a comment about quality of discourse on this site is an ad hominem.
You might find the Ad Hominem Fallacy Fallacy page interesting:
"Argumentum ad hominem is the logical fallacy of attempting to undermine a speaker's argument by attacking the speaker instead of addressing the argument. The mere presence of a personal attack does not indicate ad hominem: the attack must be used for the purpose of undermining the argument, or otherwise the logical fallacy isn't there. It is not a logical fallacy to attack someone; the fallacy comes from assuming that a personal attack is also necessarily an attack on that person's arguments."
https://laurencetennant.com/bonds/adhominem.html
So for example, if I said: "The arguments for constructing an AI pause button are bad because they are being made by AI company employees", this would be a fallacious argument.
If I said: "HN users have this weird mental block where they fail to take AI progress seriously", that's not necessarily a fallacious argument, unless it's for the purpose of undermining an argument the HN user made.
> Most people here take AI progress very seriously
You have only an extremely weak basis for this claim and have essentially just made it up.
> judging by the number of posts on the front page
Posts reaching the front page is not a reliable way to measure what "most people here" believe.
> It's ironic that
It's not irony.
It really comes down to this: Sam and Dario are leading companies that who made the reckless decision to train on everyone's IP without proper process or consent. It has been proven they trained on pirated content. There is no question about this.
Then, as soon as there are open models who threaten their businesses, they cry "distillation and IP theft".
It's the last 3 years of pure hypocrisy and fear-baiting for investment.
To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.
You're missing the point of the OP:
"there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems... each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration"
>To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.
Then advocate a muscular regulatory regime which reduces the amount of power they have. Tell them: "You want regulation? OK. How about everyone whose IP you trained on gets some sort of veto option on your models? That should reduce the racing risk that you're talking about."
I suppose I have personally lost faith in centralized institutions that would be implementing said regulation, partly because they let OpenAI and Anthropic (along with Suno, Audio, etc etc) get this far uninhibited. Centralized regulation simply can't keep up.
My hopes now lay with a decentralized approach of open models, trusting people to adapt them to local needs and security, rather than a nationwide paternalism.
I don't exactly see how "trusting people to adapt [open models] to local needs and security" solves the problem of "automating AI research" where "capability development rapidly accelerates beyond our ability to understand or control the resulting systems".
If regulation has been too slow, that's not really an argument for making it even slower?
Why is it you are so quick to skip over whether "accelerates beyond our ability to understand or control" actually happens and instead jump to "regulate now"?
1000 employees of the anti-alien defense industry are telling us we better be worried about an impending alien invasion. Let's not spend any time asking whether this alien invasion is really coming. Let's just trust them and start regulating willy nilly.
It is a very bad thing. Your friends do not have a solid understanding of geopolitics and are putting many, many people at risk worldwide, possibly for a generation or longer.
The reason this petition is a bad thing is if it is effective it will create a global panopticon surveillance state, while INCREASING incentives to avoid that state and ‘defect’ in game theoretic terms. It is fucking begging for the worst people in the world to have a lot more time to create what will be the most valuable thing in the world, their very own ‘helpful only’ model.
As conceived of by your friends, who I believe are all good people and want to help, they are locking what is already a very difficult game theoretic strategic scenario into a zero-trust game in which defection pays. It pays so amazingly well that they have to add both impossible and TERRIBLE penalties to the entire world to try and shake the game out of this zero-trust state.
Begging Senators to force a slowdown on the world is going to result in … a slowdown on the US labs at best. It’s just a terrible idea with terrible first and second order consequences highly likely.
What SHOULD be done instead is acknowledge the economic and warfare realities of this tech and try and construct a non-zero-trust game out of them. I’m not saying this is simple. In the interim while we hope for another Von Neumann to come forward, they MUST race, and any country that thinks its possible we’ll have some sort of lift-off and wants to be on the right side of it MUST also race, and MUST NOT politicize the idea of slowing down as a good thing.
Did these guys miss who they elected for president again? No sane person anywhere else in the world is going to go along with said """governance""".
I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keeping.
For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
The other justifications for restricting model training don't seem nearly as compelling. Worse, restrictions on model training have an infectious quality if they mean something more than "leading AI labs will voluntarily slow down together." (Like if the signatories mean that better models and software for building them should be regulated.) It's too dangerous to publish stronger models. So it's also too dangerous to publish training software. So it's also too dangerous to publish tools that allow you to build training software. (Claude's Fable won't help you write generic model training pipelines for this very reason.)
If you keep following the implications of this safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed.
> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.
This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?
If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.
I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!
Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.
> I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!
So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?
Because there are many legitimate uses for nucleotide synthesis, and we would have to slow scientists down by adding more regulation. The other problem is that it's not like it's very difficult to do it, so a motivated person could still do it even without access to these services.
> Because there are many legitimate uses for nucleotide synthesis, and we would have to slow scientists down by adding more regulation
There are nothing like as many legitimate uses for it than AI, not close, while it's far easier to vet the relatively limited set of people doing it.
> so a motivated person could still do it even without access to these services.
Would this hypothetical person be stopped from doing it by not having an AI to talk to about it? Or could they just use existing literature on the subject? AI doesn't help here one iota. You might as well ban Google for potentially showing people such information.
This, also, is not something that has proven to be a problem.
> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.
I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.
Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?
Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot).
The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.
There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.
> We know they cannot do spatial reasoning
Everything else aside, and regardless of whether that claim is currently true, I would ask you to be aware that that statement requires a way of thinking that ignores predictable future developments. Whatever the capability of any systems currently released, there will be more capable ones tomorrow, and the most you can say is that we have not mastered training spatial reasoning yet, not that we can't or won't.
> Are we all doing CRISPR in our kitchens now?
https://www.the-odin.com/crispr-kit/
> cannot do spatial reasoning
Opus 4.8 was the first model that could plausibly roleplay a BJJ roll without making a physically implausible move within five conversational turns. Fable 5 was the first model that beat me at the task, where I was the first party to write down an implausible move.
You can do CRISPR in your kitchen. It's been possible practically as long as CRISPR has been a thing. It's not particularly difficult, to the point that my biggest counterpoint to the whole AI doomerism argument on the subject is that nobodies done it yet.
You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.
Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.
Can you point me at this light reading? I don't wanna do CRISPR in my kitchen, but itd be cool to know more about it. I thought I needed to be basically pre-med to grasp it.
Here's a step by step guide on how to gene edit rice
https://bio-protocol.org/en/bpdetail?id=1225&type=0
Mostly it's just heating and cooling tiny volumes of liquids
This experiment has been run, and even the biological stuff is far fetched.
I've written about this before, but the issue is that these people have worked with computers their entire lives, and they keep projecting outwards from there.
The 20th century was dominated by mad scientists (mostly Teller) and generals (bombs away LeMay), but one thing that I respect about them is that they didn't just sit and guess about probable futures. They sat down and actually ran experiments.
For example, in the 1960s, there was a whole generation of people asking "who's next?" after the US, USSR, the UK and France had made The Bomb. And instead of just gesticulating wildly and trying to keep fighting the losing fight of "classify everything, admit nothing," Teller found three smart, young physicists (postdocs) with 0 nuclear weapons experience, and indeed 0 weapons experience, and ran an experiment called the Nth Country Experiment, where they were were asked to make a design for a working bomb.
After 2+ years, they were able to. And so proliferation work shifted from controlling knowledge about the technology and the technology itself to materials.
Something similar was done with bioweapons via Project Bacchus, where they gave actual bioweapons experts carte blanche to set up a secret bioweapons lab with COTS equipment. They succeeded. This was in the early 2000s.
This then led to surveillance of specific purchase combinations and equipment. Because tbh, most weapons of mass destruction are commodity technology. Nuclear weapons are 80+ years old. Chemical weapons are over a century old. Bioweapons are god knows how much older. And it's not the knowledge of these things that matter, but intent, materials, and the ability to create them.
Just because you know something about a thing doesn't mean that you're capable of doing that thing.
Let's take bioweapons.
They keep comparing wet work in a lab to writing code on a computer.
When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.
You can screw up an infinite number of times on your way to a successful exploit.
If you screw up with lethal agents in a lab? You die.
Here's a non-exhaustive list,
"lethally infected with the Marburg virus after accidentally pricking himself"Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.
The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.
Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?
No, it doesn't work like that.
The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M. And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.
And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.
These experiments have been run. And we can run them again.
The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.
Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.
Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.
Thank you for taking this seriously enough to write this, and anyone else likewise.
But this is attacking a strawman, amateur bioterrorists. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.
Being at risk of killing yourself also just makes success X% less likely, but if X < 90 that doesn't mitigate much.
It sounds like your claim is "AI is a force multiplier, therefore it is dangerous and must be regulated." Not trying to strawman - I read your comment twice and I think that's what you're saying.
By that logic Excel, the internet, calculators, and air conditioning are all force multipliers in that all of the make it easier for an expert who can make bioweapons able to do so more quickly / comfortably / easily. Obviously it wouldn't make sense to "pace" any of these technologies. Even social media or email could be considered force multiplier in that they could literally allow you to increase the number of collaborators working on supposed bioweapon. Credit cards and modern logistics force multiply your ability to purchase precursors.
Also it would be a force multiplier on the prevention/mitigation side as well. Police armed with AI to scan for suspicious precursor purchases or review security camera footage after the fact would be far more effective than before. It's not obvious that the force multiplication is greater to the bad actor or to those who oppose them.
I didn't argue "must be regulated". I'm arguing AI is powerful (at achieving things, and hence has dual-use dangers). Many people won't even admit that, which is the part that really annoys me: they don't even want to have a conversation about risks because somehow AI is not actually a powerful general-purpose tool. Of course everything you said is true, though many of those things aren't very powerful. But the internet and social media and smartphones certainly have been of great utility for terrorism and child exploitation (and probably also causing many would-be-terrorists to get picked up).
I want to ask you a question, do you feel like that this is the problem being solved by the current "safety features?"
Let me rephrase my question, do Anthropic and OpenAI implement the same "guardrails" and "safety features" for the NSA? Does the Mythos NSA Edition™ have these restrictions? What about the one that's running as a part of Maven?
How does stopping me from asking about rabbit sex protect you from nut jobs with a security clearance using these systems to go off and make weapons?
Because this has happened before. The only successful bioweapons attack in US history was done by a guy who worked at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks
The only private organization successful enough to make biological and chemical weapons is Aum Shinrikyo and they had university affiliation and nearly a billion dollars in the 1990s.
The guy who led Aum's bioweapons program was
and, https://irp.fas.org/congress/1995_rpt/aum/part04.htmand,
and, Why didn't anyone ask questions about a known cult importing an absurd amount of precursors, industrial-grade HEPA filters, fermenters, and lab equipment?It's because they'd recruited members of the military, bribed local policemen and politicians, and used their sway to silence critics (or kill them).
How does stopping me from asking Claude about rabbit sex stop people like them?
From where I'm standing, Anthropic and OpenAI would have sold Aum a subscription.
The Fort Detrick guy would have access via the US Government and Aum had university affiliation. I am yet to read a serious proposal that actually deals with these risks and the other real risks of this technology.
Wow, remarkable. Clearly Aum is a different league from the lone-wolf "Fort Detrick guy", treat the risks separately. But I'll take these questions as rhetorical. (See my reply to the sibling comment.) I can't answer them and I'm not defending the guardrails on Claude; in their current form I too find them pretty ridiculous.
Please write up your comments as blog posts, articles or whatever you can do.
There is such an incredible amount of bullshit going on in the conversation. You are contributing a point that is necessary for people to start taking into consideration and the only way to get people to talk about it is by repeating it 100 times.
I fear that the AI labs will get their way and we'll get idiotic restrictions that favor them, all in the name of "security" when the reality is they care about power for themselves, not real world security.
> ...I see how no-guardrails LLMs could be truly disastrous...
by all accounts this is what's what because of how...
> You can do CRISPR in your kitchen.
.. and because of the same dynamic we see occurring in cybersec.
If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.
If you're good enough at biology and have certain ideologies, but not good-good at specific skillsets like genetic engineering in your kitchen, do LLMs get you good enough? Does the impact on cybersec transfer over to this domain? H
Hard to say, but it has enough risk indicators for me to think the pushback against these possibilities is the height of otherwise talented engineers being myopic.
If good enough at biology can include all the PhD programs across all the world, and you pair other known events like cartels and terrorism groups hiring chemistry PhDs from University of Unemployed Chemistry PhDs, then the next logical extension of that theme with this tech is intense.
If the accuracy of your biology claim is "hard to say" then why would pushback be self-evidently myopic?
You and others are making a claim about LLMs enabling biological dangers. Some people disagree with you, pointing out that the incremental knowledge an LLM provides over say google or a library card may not be that much.
What seems myopic to me is un-skeptically accepting "AI dangerous, must regulate" messaging from frontier labs that have trillions of dollars of vested interests in this message.
>frontier labs that have trillions of dollars of vested interests in this message.
OP suggests the opposite: "each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration."
OAI and Anthropic specifically are under intense pressure to compete. One form of competition could be to innovate. Another form would be regulatory capture. The former seems unlikely given how quickly open labs are catching up. The latter is much easier and cheaper.
If you can't beat them, ban them.
If you screw up with a lethal pathogen in a lab, you die.
If you screw up with code on a computer, you go get lunch.
They're not the same.
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.
It's not a shallow dismissal if you have been following their - let's charitably call it "evolving" - public posture on AI legislation.
It wasn't long ago the same people/companies successfully lobbied the Trump admin to quash state laws that threatened to "Pace" AI, and set ground-rules for safety. Back then, open source models were very far behind the frontier, and their spiel then was it was imperative not to impinge on AI progress with red-tape, lest America's adversaries catch up.
I challenge you to explain what triggered the about-face. The simplest explanation is that open source/Chinese models have caught up with the frontier, and the change in opinion is tactical (and obviously self-interested), rather than a deeply held belief.
Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI.
If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?
What if "way short of AGI" is still dangerous in ways we can't easily handle?
When did the pacers have this mass-revelation about dangers of AI? The timing is curious.
Which takes priority: public safety or the free-market + national security? The AI labs hype in now biting them in the ass: why would China and the rest of the world deny their businesses a technology that "is bigger than the invention of electricity" or whatever hyperbole the snake-oil salesmen came up with when they were not breathlessly declaring that the first side to get AGI wins forever.
Even if the pacers are right, it's like greenhouse emissions all over again; each country wants everyone else to cut emissions, but no one wants the economic hit of doing so.
i think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts. if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.
The old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.
You're right, in that we would have said that.
I wouldn't frame it as moving the goalposts in a bad way, though. We understand more now than we did then, so we have a better idea of what the actual goal is, and moving the goalposts to frame that goal is more rational than insisting the goalposts stay where they were ten years ago even though the entire pitch has changed since then. I'm straining the analogy, obviously.
Moving the goalposts, so to speak, is how science works! We must of course update the things we think based on an improved understanding of how the world works. Only the deeply incurious could consistently demand that one adhere dogmatically to a prior way of understanding the world in the face of changing knowledge. It's important not to treat science as a game to be won (of which 'goalposts' are evocative), but as the collaborative effort that it is.
Note well that I have not said LLMs are not useful, only that no amount of interaction with them has convinced me they remotely approach general intelligence. Their actual function of heuristically regurgitating all information ever known to mankind remains extremely useful in lots of domains.
And by the way, it's not obvious an LLM could fool me (or the average person) over a sufficiently long period of time, exactly because they are heuristic machines. That they lack thought guided by underlying cognitive models seems to always leak out, in the end. I suspect pass rates by LLMs on long-range Turing tests (if there have been any conducted) might drop as people become acclimatized to them.
what do you mean by 'general intelligence'? i definitely agree that in comparison to humans, they have some definite weaknesses. however, i think they also have definite strengths. and i don't think their current weaknesses imply that much about a potential singularity or ai risk.
Sorry, I should've used clearer language. I just mean intelligence in the sense humans possess it. Computers have always been able to exceed limited elements of human intelligence (say, at arithmetic), but the problem is to simultaneously match or exceed all of them. I specifically think that the important parts they currently lack make them a no-go for supposed existential risk.
If that’s truly the case, why aren’t they offering to be absorbed by national labs. That’s what we do for dangerous biological and nuclear work.
Oh wait they want options and shares and to get rich.
And there’s no natural moat. Which Federal regulation would bring.
That’s why people think it’s disingenuous.
As a Europoor I've seen the devastating consequences of rampant safetyism first hand. Entire industries and energy security destroyed to satisfy the precautionary principle.
We need to accelerate as much as possible and create abundance as fast as we can. Life is short.
"This building is too cold. Time to set it on fire."
Let's not swing from one extreme to the other.
If American hypercapitalists, living in Silicon Valley of all places, tell you that you need to regulate, maybe regulation really is called for in these specific circumstances.
https://en.wikipedia.org/wiki/Regulatory_capture
Well, we've been talking about the EU. EU regulation has not been especially favorable to US big tech. The EU is collecting more from fining US big tech than it collects in taxes from EU public tech companies:
https://xcancel.com/levelsio/status/2080314960656159018
[Edit: I'm seeing some people dispute these numbers in the discussions around this infographic... regardless, the point stands that EU regulation has not been great for US big tech]
I think you should at least consider the possibility that this letter is not primarily motivated by regulatory capture, and insiders at these AI companies realize something which outsiders don't.
To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms such as Thinking Machines, Core Automation, Inherent, and Prime Intellect.
I have considered that possibility, but I don't find it convincing.
> insiders at these AI companies realize something which outsiders don't
Insiders at these AI companies certainly would like us to think that! Unfortunately they have been beating the same drum for years, all while continuing to pursue the very "danger" they decry.
I simply cannot ignore the timing of these events, always keyed to important moments in the market. This one comes as frontier labs need BIG exits and soon - or else they'll be the ones left holding the bag when it pops. And just as China has proved that, indeed, there is no moat.
> To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms
That's a good point. However, I will say that it's tough to suss out financial and other relations among the parties in this space. A lot of these employees hop from one lab to another. The industry's byzantine financing deals are an inscrutable web. It is an interesting data point, but not enough to tip the analysis for me.
Another thing: many AI lab people are, quite frankly, kind of unhinged. They live in an insular subculture where imaginary futures have been repeated to one another so many times that they carry the gravity of truth. They've thought nutty things like, e.g., that primitive versions of these static computation graphs were sentient or had feelings. Anthropic apparently pays people to perform a ritual on Claude before they extinguish its soul to move onto the next version. Etc. Point being, even if their hearts are pure, I am not sure their judgment is sound.
>I simply cannot ignore the timing of these events, always keyed to important moments in the market.
I suspect this letter was triggered, at least in part, by the recent autonomous cyberattack news.
If this letter is a purely amoral initiative, I would expect the most amoral operator in the business (sama) to be a signatory.
Sorry, I know this a non-sequitur but I just saw your profile and noticed you're in politics. If you want to do something productive about the risks of AI: *regulate its applications*. Get it out of employment, housing, and credit decisions. Incentivize litigation for consumers affected by algorithmic decisions. Incentivize whistleblowing activity by employees who know that black box systems are being misused.
There are so many tools available to do something about the true threats AI poses. (This is a topic that the industry prefers not to discuss.)
No, it just means they want to keep the slice of the pie for themselves while they're at the top.
Every single technology in history has had people claiming potential existential risks that didn’t materialize. Burden of proof is on people making the risk claims. Those have to be specific and unique to that technology to grant special actions.
i'm not trying to be rude, but i really don't get what you mean by this? (https://www.amazon.com/Superintelligence-Dangers-Strategies-...) there's been people making specific and unique risks to this technologies for years, before AI was even present. anthropic was literally founded because they were worried about existential risk? also, previous technologies have had existential risks? do you deny that nuclear weapons could have killed humanity? obviously, there have no been existential risks that have killed all of humanity, because by definition, that can only happen once. it will literally be impossible for someone to ever point to an example of that happening, because they will be dead by the time that happens.
See printing press and internet!
And video games, democracy, novels, TV, cars, photography, cryptography, the Beatles… pick your favorite and read commentary / opinions of the time. Many thought that came with unique risks never seen before and must be stopped!
How many of those concerns came from the people building the technologies? John Carmack wasn't up in arms about how Doom was going to destroy society. Charles Dickens didn't think novels rotted the brain. Johannes Gutenberg wasn't concerned about the effects of a free press.
None of the previous concerns were true and nothing indicates current ones are. It’s extrapolation, hypothetical apocalyptic imagined scenarios and piles of assumptions on how the technology could evolve. What makes even less credible in the AI case is that there’s financial / power to be gained by those making the claims.
And to your point. Don Quixote, the first modern and greatest novel of all times, is about a guy that lost touch with reality from reading too many books
None of the previous concerns were true, largely because they came from laypeople who didn't understand the technology. A prominent exception is nuclear weapons, where their creators did worry their creations could end up destroying the world. I think most people would agree those concerns were justified, and it's not any lack of capability that means we're still here.
The motivations aren't unique here. Restrictions on the development of almost any technology benefits incumbents by providing a moat.
Employees inside frontier labs have the chance to make life changing money if they can make it to an IPO before the music stops. Have you considered that might be the reason they are so eager to claim "ai dangerous - regulate ai"? The nuclear scientists at los alamos did not have generational wealth incentivizing them to raise "concerns" to the public.
If your only evidence that the claim "ai dangerous - regulate ai" is true is "insiders who stand to make billions if they can regulate away the competition" told me so, that is pretty weak evidence indeed.
If a group of concerned scientists (without $$$ in stock options) demonstrated that anyone with one year of HS chemistry, an Amazon account, and a ChatGPT max plan could build a bioweapon 10x more lethal than an AR15 mass shooting or driving a truck into a crowd (because it is certainly more than 10x harder to make a bioweapon than either of these things, even with a theoretical AGI), then that would be worrisome. This is really not that high of an evidence bar but even it has not been met, but oh no we should start regulating yesterday.
I mean, Cervantes was parodying medieval romances. That's what Don Quixote was reading, not modern novels.
The point is, it should give you at least some pause when the people most familiar with a technology are among those making the strongest statements about its potential dangers. Extrapolation has worked extremely well in predicting what AI will be capable of over the past 5 years; how confident are you that it will fail before we reach apocalyptic scenarios?
Cervantes was capturing the zeitgeist of the time: reading too many books can make you mad as they were becoming accesible to anyone. We know it’s not true.
People at anthropic don’t have any secret that many people outside already have. Again the burden of proof is on the person making a claim. That you feel concerned or scared doesn’t grant you special privileges to control or dictate a transforming technology.
These aren't assertions being made without arguments or evidence. Maybe you've engaged with those and find them unconvincing. But from what you've said in this thread it seems more like you've dismissed them without serious consideration. My point is not that everyone should just shut up and believe whatever frontier lab employees say, but that they should take a public statement like this as a sign that maybe they should carefully consider these concerns rather than dismiss them out of hand.
We're commenting on a site run by a Silicon Valley tech start-up accelerator. "Move fast and break things" is the accelerationist way, even if the "things" they break are a stable, functioning social order. As long as you IPO and cash out along the way, you can wreck everyone else's lives just fine, in the minds of this sort.
It is hard to take people working at frontier labs now asking for a government mandated slow down seriously. These people are the ones creating this accelerated risk in the first place, trying to outdo each other to release the most powerful AI as quickly as possible, and additionally claiming credit (via "distillation") for recent progress by China. China is not the threat - you are.
If you work for a frontier lab and are genuinely concerned about the pace of AI being faster than our ability to adapt to it, then you should either quit or lobby the management of your own company to slow down. You don't need the government to tell you to slow down - just do it - stop trying to leapfrog the competition and put a hold on any new releases for a year.
You could try to form a pact (of the willing) of US companies agreeing not to release new updates for a given period of time, but in the end you (any/all of the frontier labs) can choose to do it yourself if you really care.
This is just another version of Dario's insane "heads up, what I'm developing is going to destroy society" "responsible disclosure". It's not responsible in the least - if you actually believe you are creating such danger then stop doing that.
> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest
Pointing out the endless stream of lies and self serving half truths these terrible people engage in helps the younger generation understand the world they were born into.
It's constructive and helpful. It's why I'm replying to you.
You interpreting it as 'shallow dismissal' is shallow dismissal on your end. You're the shallow dismisser, not the people calling terrible people terrible.
As for your little petition. It's as good as eliminating plastic straws to save the environment.
Terrible people with power they don't deserve doing pathetic little, tiny 'good' deeds to sleep better at night needs no explaining.
Agree with everything you wrote here. Every word. The GP strikes me as unbelievably credulous and trusting of the “leaders” in this space.
I cannot fathom having that kind of unquestioning faith in people who have done absolutely nothing to deserve it.
> All of my friends who work at AI Labs take it seriously. > If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?
If they seriously think what you just wrote then they should follow through with quitting en masse because that's currently the only way to slow the progress towards oblivion. Are they quitting?
> We should be hoping that nations can make binding agreements.
The big AI Labs are making sure international binding agreements are followed by financially supporting the notorious binding agreement creator and follower Donald Trump. That's how seriously they take this issue.
Yeah let's have a constructive discourse and let's start that with dropping all hypocrisy, shall we.
On the geopolitics, the American domiciled firms are completely in bed with the US military and US government, and have made it absolutely clear that they are committed to the domination of the US government. So when you say "geopolitical destabilization", you mean "only the US government is allowed to destabilize, which they have done for decades and wish to continue doing unfettered."
And given what we have seen from that government -- piracy, terrorism, war-mongering, thievery, and just profound stupidity -- no thanks, in that case I'd like to see every American AI company utterly obliterated by any other country. If China becomes utterly dominate in AI, awesome.
What happens when a country doesn't agree to play along with this little decel game?
The genie is already out of the bottle and now a bunch of people who helped release it think that the collective powers of the world are just going to put it back in.
How far are the saftyists willing to go? Bombing data centers? Starting wars? Embargos?
It's a terrible dilemma to be in. If the threats are real, it gives all the more incentive for some actor to race and try to grab it first. If not, then any punishment isn't proportional. Explain to a nations citizens under a 10 year embargo, "sorry about your quality of life, we all just got a little spooked by science fiction for a second."
Folks try to reach for bad analogies: nuclear bombs and biological weapons. But conveniently omit that by the time the regulations took place for these things we had unequivocal proof of their destruction, scope and scale.
So, what then? We're supposed to collectively stop innovating and put the keys to the kingdom in the hands of a few capitalists, a corrupt is administration and the ccp?
I reject that these agreements should take place, on the grounds that the proposal is delusional and not founded in reality.
This is serious. We're in uncharted territory. The geopolitical context is already complicated enough that we shouldn't be dismissive of such a rapid technological race. I'm glad AI leaders are taking it seriously. I'm surprised so many HN commenters aren't.
On the other hand, I'm pessimistic about initiatives like this being effective on a global scale (though they could certainly work at the national level).
we're always in uncharted territory. that's why most HN commenters think that this is hyperbole.
"Many were increasingly of the opinion that they'd all made a big mistake in coming down from the trees in the first place. And some said that even the trees had been a bad move, and that no one should ever have left the oceans."
There is constructive discourse, but this kind of commentary doesn't get to the top:
https://news.ycombinator.com/item?id=49078376
https://news.ycombinator.com/item?id=49077753
It's not a given that AI poses bio/political/cyber/economic risk in a way that warrants government intervention. Discourse should first revolve around whether it makes sense for the government to step in at all.
Maybe I am too cynical, but this just sounds like a plea for the government to step in and save them from their investors.
Investors want to see a return on their investment ASAP. The Economist recently published an article:
"AI revenues are growing fast, but not fast enough -- The returns on trillions of dollars of spending are deeply uncertain"
https://www.economist.com/finance-and-economics/2026/07/28/a...
As OP states: "each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration."
Let's not overwrite human civilization for the sake of the shareholders.
"Oh no! The AI we developed for years, and whose stratospheric valuation is predicated on replacing human jobs, is coming for our jobs, the government should do something!"
Signed,
AI researchers and engineers.
If the “recursive self improvement” is taken literally (and at some point it will) then that is going to be what happens.
The fun part is that probably happens quite a way before getting anywhere near to AGI, and it is almost plausible that for many of them that is where they are.
Anyone serious about this should participate in pacing the frontier by quitting their job so as to not potentially contribute to the problematic progress. Be the change you want to see!
Curious how that isn't what happens.
Some of us considered seeking one of these jobs and decided not to. Some have quit (e.g. Hinton, OpenAI people after OpenAI reneged on safety commitments). Now you're getting this message from people who passed the previous filters. If you think a brake is indicated and not just easing up on the accelerator, you need collective action as the statement said.
> you need collective action
Come on, go on strike then! Have a week of protest and picket the offices in SFO. Let's see what happens.
(I'm aware that's not quite what you meant, but it is the classic meaning of that phrase).
Mass resignation perhaps?
...so the only remaining employees give no thought to safety?
Unionization might be a little more effective.
That's an application of the "Tragedy of the commons": One person stopping to push the frontier just means that this person will lose their job, the frontier will be pushed by the others. The solution is to push for a binding rule instead that will make everyone slow down in the racing towards artificial superintelligence.
And this is how we absolve people of all personal responsibility. Such companies would not pay people that well if they were so fungible.
There is no such possible binding rule either, a point that is so obvious that any smart person claiming otherwise is clearly up to something.
Influencing change is more complicated than that.
We live in a society that has systems of market competition between companies and economic/military competition between countries. There's not just one country working on frontier AI, and not just a few companies in each country either. The incentives of these systems ensure AI advancement will happen quickly from here on out, no matter what, absent governmental action.
An individual researcher leaving the company does nothing, other than give their seat to someone else who likely doesn't share their same concerns.
Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
> Alternatively, using the influence they do have in their position to directly try to advance AI safety research, advocate for govt action, and make public arguments attempting to persuade the public to pressure the govt to act when they otherwise wouldn't be inclined is far more likely to impact things for the better.
No, it just comes across as "I want to get paid as if I was doing the thing I am employed to do by threatening you all about how terrible it would be if I actually did it".
They aren't even being semi imaginative and going on strike or anything.
We make binding rules all the time. We didn't fix the ozone layer by encouraging people to quit their jobs producing CFCs. We lobbied for international regulation to ban them.
In your example there is nothing to be gained by not complying.
Failing to comply with what is in reality arms control is a widespread occurence. It's notable the only historical way to handle this (Schelling covered it at length) is tolerance of mutual espionage on a level that would simply not be tolerable to any of the groups involved.
I should add one of Schelling's other pet subjects was "time to weapon being viable". i.e. the distinction between a nuclear power and non-nuclear power isn't quite right, as there's a huge class of countries that could become so fairly fast if they were so inclined, and that this situation is somewhat inevitable. It is this that leads to efforts to denuclearize, even on a civilian level, "problematic" countries. You want to stop frontier AI and you'll have to be rolling back a lot of existing deployed technology.
The Montreal Protocol has not been perfectly adhered to either. China started emitting quite a lot of banned CFCs again from around 2012[1], which didn't get noticed until 2018! By 2020 it had finally stopped[2].
[1] https://www.nature.com/articles/s41586-018-0106-2
[2] https://www.nature.com/articles/d41586-021-00360-0
Great comment. I haven't read Schelling but will check that out. I think you're right that international cooperation on the topic is likely impossible at present, for the reasons you stated. I'm not sure what it would take to shift the Overton window, but it's probably not pretty.
So honestly Schelling was utterly brilliant. I cannot recommend all three of the books wikipedia classed as his notable works highly enough:
https://en.wikipedia.org/wiki/Thomas_Schelling#Notable_works
There are footnotes in "The Strategy of Conflict" containing more profound insight than whole books that are published today.
On this particular topic though "Strategy and Arms Control" is the one to go for.
Hoping everyone chooses to do the right thing is not an effective strategy for mitigating bad things.
Personal responsibility is a difficult concept for be to apply to massive efforts like this. When there’s one person and you can talk to them personal responsibility is the number one topic. When you’re talking about a multi-trillion dollar bet and hundreds of thousands of people it doesn’t make sense anymore.
Same logic goes for other common issues where personal responsibility is lauded. Obesity - real solutions are policy and medicine. Abortion - education and contraceptives.
FWIW, here's an account from a guy who quit his job at Google Deepmind recently:
https://turntrout.com/why-i-left-google-deepmind
“It is difficult to get a man to understand something, when his salary depends on his not understanding it”
The historic analog most befitting AI are weapons. It provides an advantage to companies and governments in creating weapons, in surveilling people and thus gaining leverage over individuals, in controlling what individuals can do or cannot stop doing. It thus follows the laws of weapons control - the more restrictions are threatened to be imposed, the more even the smallest actors will fight to get more of it. They may not understand or know what they are getting, but they will want more of it because their potential rivals have it. Governments are the ultimate customer here - they have the money, the energy, and the appetite to consume AI infinitely in battlement against opponents - foreign and domestic. The only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment (the upside of that is it advances the AI for future use and is thus in the interest of governments as well). Modern large models seem to be compressible and thus we may see more and more tiny sub-specialized varieties. But the push to slow the frontier in weaponry always leads to the reverse.
I think we see things similarly about the negative potential but please be careful with the framing.
AI is a printing press or a steam engine. It's not a weapon.
The framing is important. Much like 'distillation attack'. These framings are an attack on the narrative.
Printing presses at the time were seen as extremely dangerous too. For many of the same reasons cited about AI.
In both cases, the technologies were used to destablize existing power structures. And, after some hardship, make a better world for most people.
Even calling these things 'dual use' is a weird framing. Books are not 'dual use technologies' unless you want to sell fear.
A piece of wood can be used to build a house, to carve a statute, to make a bow and arrow, or to make a baseball bat. Honesty is the only way to see the problem in a way that can be tackled effectively. Framing will not help if you are dealing with large animals fighting in a cage when you are trapped in it, and it this case we all are.
> AI is a printing press or a steam engine. It's not a weapon.
I've not heard of a printing press going rogue and attacking things.
> only approximate solution is to prevent any restrictions on the use of it - a-la-2nd amendment
Not even in 'Murica do we have anything close to no restrictions.
You can't buy your own EMP or nuke, and even warring governments have been surprisingly reserved in building (at least to public knowledge) biological weapons.
Completely agree, BUT you’re not dealing with -self-preserving humans anymore. In this case what are the odds of some less competent large state actor getting a model that goes rogue and has to be mitigated against? How do you mitigate against novel weapons it creates if your models are lobotomized in that category? No single government will neuter their own models and very few if any are competent and interested enough to mitigate against them.
This. Will. Never. Work.
This research game is a zero-trust iterated game. Any defection carries the benefit of possibly being permanently valuable. There will always be defections. OpenAI defected WHILE THEY WERE pushing to slow.
The cost to the US of deciding to slow is so massively harmful that it is wrong to even put this idea out in the public zeitgeist; it may somehow become a thing that US legislators believe they can accomplish. They CANNOT accomplish this, because the fundamental economics and possible outcomes make it mandatory for any strategic adversary to work as hard as possible to defect.
The proposals here imagine a global panopticon surveillance state overseeing all research and chips as a 'reasonable' solution. EVEN IN THAT (terrible) state for the world, we will have effective defection.
Like it or not, the world is on this train, and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology.
Variants of this idea in AI 2030 included burning some of the US lead to get some nebulous safety metrics improved - also insane. Any slowing allows strategic adversaries time to catch up with very different alignment goals.
Do I like this situation? No. Are the people signing these things just fully in denial about the game theory and the stakes and the behaviors of strategic adversaries? Yes, yes they are.
Why not? Slowing down / halting biological weapons research mostly worked. Yes there have probably been modest sized defections here and there, but the pace of bioweapon development is a crawl compared with (a) what is possible with science already, and also with (b) the pace of bioweapons development from 1910 to 1970.
Bioweapons are good for .. war. Agentic AI is good for changing your economy, strategic advantage, labor explosion and making more effective Agentic AI. They are not the same category of thing.
Bioweapons don't make money. AI does. The two are therefore not comparable.
The only thing it might accomplish is allowing OpenAI / Anthropic to squeak out an IPO before the market realizes AI will become a commodity and the massive profits these companies are promising will never materialize. They aren't in denial about this open letter's futility in terms of actually reducing the pace of AI, that's just not their goal in the first place.
I don’t think it’s clear we’ll get commoditized AI out of this research loop. The only real datapoints we have are that Chinese open weights models seem able to stay a certain amount of distance ‘behind’ released frontier models. It’s clear that the models are trained on exfiltrated data from the frontier models => we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum.
I also don’t believe we have seen anything like 1% of inference demand fulfilled globally. You might believe differently. But, if I’m right, we are nowhere near some massive collapse of these companies, quite the opposite. For the opposite take to be correct, you’d need to believe that the ‘value’ of tokens will not increase significantly and that the current demand for tokens is at or near a peak.
I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated. I believe instead that religion and money combined create strange behaviors and beliefs - in this case “what’s good for us could be implemented” — combined with the already endemic “what’s good for us is good for humanity.” Together we get really bad outcomes.
> we cannot currently disambiguate how easy it is to get close to the frontier in a vacuum.
Whether open-weight labs can get to the frontier on their own is irrelevant IMO. Even if all R&D were to stop today it would take a decade at least for the productivity gains from current open-weight models to trickle through the economy. For 99% of use cases of AI there simply isn't a need for further development of the frontier, where we are very clearly at diminishing returns. How many white collar workers need to write an OS or develop sophisticated software?
Think about a normal office worker - they an easily double their productivity by creating a well-thought-out partnership between the human and today's open weight AI. Humans and today's AI are very complementary. AI brings vast knowledge, pattern recognition, and infinite patience for grunt work. Human brings context and reasoning and verification.
Frontier labs are trying to replace the human altogether by trying to make AI do things that AI is weak at and humans are strong at. This is silly but they have no choice because of the massive profits they've promised their investors. It's not enough to double every knowledge worker's productivity. They want to replace the knowledge worker altogether.
This is creating a massive spike in training & inference cost that is almost certainly more expensive than the human labor its intended to replace. Fable and 5.6 sol are supposedly around 8-10T parameters. Is there enough compute on earth to make a 100T or 1Q model? We would need models that size to replace the human altogether.
> I also don’t believe we have seen anything like 1% of inference demand fulfilled globally.
I agree with this, but the massive demand for inference will force companies to be price conscious and drive them away from ultra high margin frontier models to low margin open weight models. This is why it will become a low-margin, commodity market.
Immense innovation right now is going into tiny, specialized models that run on device. See PrismML getting a mid sized QWEN model to run on an iPhone. When on-device inference becomes a thing we'll see another explosion in inference that will further leach demand away from frontier labs.
> I also don’t believe these open letters are signed by people that are all master strategists and just economically motivated.
I agree if you could read their minds they might think they're doing it for the "right reasons." But massive $$$ can create equally massive self justification/deception.
"and the only sane thing to do is to fully support high quality ethical research groups and firms as they get up the learning curve of this technology."
These research groups are almost unanimously advocating for a pause/slowdown as our best chance.
… while they address absolutely none of the above points, except for tacitly or overtly agreeing a panopticon surveillance state is needed / desirable to get their desired ‘slow down’, a situation where the cure is infinitely worse than the disease, not least because it will not be effective at the original goal of eradicating the disease, and will instead merely persist in all the global harms such a system has as a necessary side effect.
When you are ahead, ‘slow down’ = better strategic positioning for yourself. When you are behind, ‘slow down’ = time to catch up.
Some reason this kind of argument always by default assumes the US will be the good guys.
I didn’t say that. I said the US is ahead, and should consider its strategic adversaries and their behavior.
"AI is progressing at a rate that our society might not be ready for. Frontier labs are very close to AI that can exceed even the best people on almost every metric of intelligence. This will lead to unprecedented social and safety risks. To ensure a positive future we need to develop AI in a way that is driven by responsibility and thoughtfulness."
Responsibility and thoughtfulness? In this economy?
It's an admirable idea, but I don't see how you can slow this down in practice. We certainly know China will not, and they are already at the frontier.
RSI is too important and too tangible at this point for everyone to slow down. It's the equivalent of telling the world to stop developing nuclear weapons after Hiroshima. Just not going to happen, and in the meantime dramatically increases the risk that the power of AGI lands in the hands of the few (read: the political class) rather than empowers the people.
You need AI capabilities to the point where the US and Chinese leaders get scared enough that they do a deal.
There are detailed specific scenarios in Plan A https://ai-2040.com/
Also, in reality above a certain power level the Governments will stop giving us access anyway. This already began with Mythos.
We know that at the end of the day AI intelligence is closely tied to invested compute. The fact that automated AI research might spark some kind of exponencial increase in capabilities seems extremely unlikely to me. Of course we know that some got there faster and cheaper than others, but the fact that it requires immense computing power has not changed.
We also have already witnessed a slowdown, the raw intelligence of models has been relatively stable since january, but their use of tools and what we call agent capabilities have allowed us to tackle significalty harder problems with what we have already. This is becoming blatently obvious how smaller models such as gpt 5.6 luna are capable of doing similar things to sol while presumeably being fraction of the side and compute investment. This is also the likely reason why chinese models are able to catch up as intelligence strictly requires more compute, agentic capabilities are more of a mixed bag of RLHF and general post training.
I believe automated AI development will simply lead to cheaper intelligence, but it will not necessarily produce something more intelligent. There is a sentance from a book that I don't remember where I read it from, but has stuck with me which boils down to this: "We are very unlikely to make something smarter than a human brain.". If you apply this concept more broadly an AI system will not be able to produce something more capable than itself, it might be specialize in different areas via trial and error (i.e. solution discovery), but it will not have better intuition than that which was trained from the fruits of human labor. I also strongly believe that model performance is ultimately governed by the advancements brought into the world by humans.
AI of course can find improvements by simply trying a bunch of random things, emphasis on the word FIND. Until we can prove that AI systems are able to come up with unique solutions that have yet to exist even with simplier problems such as creating any kind of new AI architecture that has not been already documented, conceptiualized or explored by humans already -- and it doesn't need to be better than what we have already, just something new -- I believe it's too early to sound alarms until proven otherwise.
--
Something I've realized as I was fact checking myself: We could presumeably have self-improving intelligence if we used enough properly engineered agentic workers working towards more intelligent AI, but at that point we are back to the same problem where we're spending mass amounts of compute to increase the raw intelligence of a model.
> We know that at the end of the day AI intelligence
Stopped reading at "intelligence".
Try capability.
(unless your target audience is those already gulled.)
Can someone help clarify what it means, concretely, to make binding agreements to proceed "safely and slower" in the context of AI research (whether or not at frontier labs)? Are there specific policies and actionable guides being proposed?
There's a decent amount of stuff in Plan A. Especially if you read the supplements.
https://ai-2040.com/
Something people don't seem to see, and not that it's placed directly in front of you, is that pacing the release and adoption of AI is already happening.
Cost-per-token determines the economic viability of tasks that AI gets applied to. How many tokens would it take to automate all of the responsibilities of a gas station clerk? Probably more than their hourly wage. Job safe.
A slightly larger stretch, that most people attribute to profit margins (labs do not have profit margins, they have unlimited invested capital and no expectation of short-term returns), is the flat-rate billing plans of most code products. Why doesn't Anthropic have a 50x or 100x Max plan? Not that it's impossible, that it releases tokens into the workforce at a greater rate (note that their are hour, day, week limits across most of these services as well).
Hypothetical, but reasoned, and I do not expect most agree with me. If compute as a variable is inferred, this supports the incorporation a deliberate (redundant) one as well.
I agree with the substance of your comment that token economics pace the diffusion of AI into the labor market. But I think they have a much smaller influence over the pace of the frontier, right?
Fair point! I guess I read "pacing the frontier" as "pacing the release of the frontier" surely no lab's internal research team is just going to stop innovating?
Not under all the competitive pressure they are facing, but if an industry wide slowdown could be achieved they might spend more time on model alignment and less on pure capabilities.
There seem to be people working at OpenAI and Anthropic who are surprised to hear that many people don't trust them.
If this is you, I would recommend talking to people outside of the AI industry - we've seen OpenAI and Anthropic continuously lying, abusing lobbying, training on pirated IP, trying to scare politicians into blocking actually open AI from abroad, etc etc
Why the hell would we trust these companies to be impartial regulators now?
> I would recommend talking to people outside of the AI industry
They already are. Loud and clear.
What they aren't doing is listening to people outside of the AI industry.
And that's because they don't want to hear.
> We request that the U.S. government ...
Too bad these researcher's organizations spent their efforts supporting the dumbest, ficklest possible version of the U.S. government to guide us through the gravity of this moment.
Implied in this is a concentration of power required to enforce.
No one is sold on a dystopian hellscape of fear, surveillance, and control. Each one of these would be required for a select group to effectively enforce AI safety.
Inflicting fear on society has never lead anywhere good. Please stop.
New technologies have always been scary. But humans have adapted. I believe in humanity. Do you?
> New technologies have always been scary. But humans have adapted.
The Silurian Hypothesis would like a word.
These AI companies needs to stop with all the doom and gloom.
The first computers were used for war.
The Harvard Mark I was used by von Neumann to calculate the physics of implosion. And then right afterwards, ENIAC was used to calculate the feasibility of the H-bomb. They called it the "Los Alamos problem."
The first computer network, SAGE (or, Semi-Automatic Ground Environment) brought together RADAR data from across the US with air-traffic control data and military flight data to detect overflight by the USSR. It could direct and control fighters to intercept those planes.
If you were sitting there, in 1955, would you have believed that computers and their networks would bring about an extraordinary age?
Imagine if von Neumann and his ilk started campaigning in 1955 to limit the development and use of computers, because surely they can only be used to make weapons, right? Right?
We were lucky to have visionary people like Joe Licklider who could look past the present and see the future. In 1963, he wrote a memo calling for an "Intergalactic Computer Network" (because he knew engineers under-delivered. So if he asked for a galaxy, at least he'd get a planet). And he gave these machines to scientists and people like the original hackers at MIT to play with.
What if Licklider had written memos about how we had to "pace" computers to make sure that other countries wouldn't be able to develop implosion technologies or systems like SAGE? What then?
Do you think you would have the jobs and the prosperity you do today?
If instead of giving away this technology, spreading it, and putting down the seeds for ARPA-net, Licklider had written memos from "concerned scientists" (and I'm sure he could have found a bunch) talking about how computers can only be used for weapons and war! So stop the proliferation of computers! We can't imagine a world with computers small enough to fit inside everything! Even wall sockets! What if your wall socket could do war?!
They didn't, because they knew there's no better way to kill a better future for all mankind than to control it.
† yes, peace. We are more peaceful than we have ever been. Humans can be a violent lot even when reluctant to kill.
We were incredibly lucky that computing and nuclear weapons led to a stable geopolitical equilibrium (mutually assured destruction). There were certainly a few scary points where we were still settling into that equilibrium during the Cold War.
I don't have a strong opinion either way, but it seems very hard to predict. Why do you think AI advancements will lead to a stable equilibrium?
Equilibrium for whom? Equilibrium for what?
Here's what I know. Every day, I read a 5 to 10 incoming emails from young people in rural Africa, Asia, Europe and (heck) the US. People who have had no other resources and options, but are tremendously gifted and somehow got their hands on an internet connected device (usually a very cheap smartphone and/or cheap old computer).
These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do. They make robots. They learn new things and try to improve themselves and others. And some want to change the world / do groundbreaking research.
When you sit down and start pricing and doling out AI from behind regulatory gates, you shut those people out.
There are a lot of smart people in this world who have traditionally been unable to get access to resources and education. MOOCs didn't do much / help. But these AIs? They were born multi-lingual. They can talk in any language. They can tailor their responses to their audience. And they can help broaden these people's horizons.
There are no tutors around them. There is no one who is coming to help. But an AI that can run off of a small battery and a connection to a cellular network can give them access to genuine intelligence to make their world better.
I met one sixteen year old who was accessing Claude surreptitiously (by begging people) and was trying to apply some really fascinating biomimetic fluid dynamics observations to aerospace. This person was born in one of the more under-resourced parts of the world. And Claude gave them a list of people to reach out to who could help.
If you want a golden age for humanity, you should be asking for Licklider's Man-Machine Symbiosis.
The outcome that you're describing of AIs facilitating education for disadvantaged people is great, but isn't AI simultaneously significantly devaluing education (excluding of course education pursued for intrinsic reasons, which I'm sure many people will always pursue) and removing the need for educated people? Or at least intended to be doing so, even if it isn't accomplishing it yet?
Your examples seem to be of humans educating themselves so as to contribute to the world intellectually, but the promise of AI (and what most frontier lab employees seem to be saying) is that it will soon eclipse human intellect in almost all ways. If such a world is what we end up with, I don't see humans meaningfully contributing to "groundbreaking research" and a golden age for humanity seems to be mostly contingent on whether or not we take political actions to ensure that AI benefits everyone.
> These people, who are usually teenagers, end up getting connected to Claude, who then gives them a list of things to learn, people to contact, and helps them make things. And they actually do.
No doubt.
> They make robots.
Ah. And the ones making malware?
Young people with shut of from education and resources are building robots with the help of Claude after Claude existing for few years only? Like common. Young person with no support does not get to build a robot, because it is actually an expensive thing to do. And learning robotics with no education but few years of Claude is also implausible.
A lot of atomic scientists did regret their work.
Yes, and did you see them oppose computing? They promoted it and tried to use nuclear energy for peace.
I think this is a spot on analogy and would two more: the hand wringing over the release of the printing press and the internet. In both cases the people in power were terrified that the masses would use their newfound power to do dangerous things. In both cases bad things did happen (market bubbles, scams, witch hunts, spread of knowledge on making poisons), but trying to put the cat in the bag by pacing the printing press or the internet would have been a fool's errand, and over the long run both inventions massively improved the lives of society overall.
> We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
This is quite vague to me, what exactly do you mean?
It means that they request the structures needed to navigate the event horizon of recursive self improvement be put into place. Probably: International agreements, verifiability of an AI pause on the level of graphic cards as well as through physical inspections of datacenters. Like it's done with nuclear energy.
It means they haven't really thought it through and don't have a consolidated enough opinion to form a specific Ask.
It means only those vetted will be allowed to use frontier models (i.e. let's pace ourselves and not share the frontier broadly)
'Regulatory Capture' if we were cynical.
I think a lot of these guys mean it though.
'The Fear' - is not some magic intelligence will come alive and be Skynet, these people are ridiculous.
People will use the compute power to do nasty things, we'll be overwhelmed with bots, regular companies will use them for dark patterns, political actors will use them to cheat.
It will be 100% humans using it for bad stuff, and in not very novel ways.
And Xi rejoices!
What these people want is same thing people in every other field want too, which is being able to use AI as a tool and nothing more. Off course, some of the people that signed this might not recognize this, and the people who didn't sign it are actively working against such an equilibrium.
I'm chinapilled now.
Anthropic is an evil, enemy of humanity company that cannot be allowed to succeed.
Yes. Which Chinese labs and models do you like the most?
Pacing is a really bad idea because China will not slow it down for you.
The entire point of the statement is to start international coordination.
The problem is how do you enforce coordination? This isn't like nuclear testing that can be detected around the world on seismographs or from satellites in space.
AI research happens in benign looking buildings that could be doing a hundred different non-AI workloads, much of it things that industry or governments would not want 3rd parties prying into. There is too much to potentially gain to trust coordination.
See detailed supplement to Plan A which talks entirely about this:
https://ai-2040.com/supplements/verification-plan
There is a lot of research out there with answers to your questions. GPU supply chains are extremely constrained, making it a surprisingly tractable problem at least for now.
How do you tell the difference between inference and training workloads from outside? The same GPUs can be used for both and the pacing suggested here relates to training and not inference as far as I can tell.
Random sampling of compute activity.
https://ai-2040.com/supplements/verification-plan#appendix-p...
It will never happen because there is money to be made by not coordinating, i.e. by leaping ahead. It also doesn't make any sense to hold AI back.
That's the point. Do you see any concerns by these companies? They just want to slow down pace NOW because they're losing the race
All they have to do to stay in the race is to keep the price and token efficiency competitive with China.
Huh, that's really mental gymnastics you have there :-)
> losing the race
Lol, lmao even
The website does not list all the comments but just around ~50 without any way to load all the ~+1100 more (1132, as of 2026-07-28 2100 UTC).
The number of em-dashes is questionable, existing in many, if not all, the comments.
I am sorry, but I am not sure whether it's adequate and fair to allow join such statements/petitions to only those who live in it. Isn't it like allow deciding the future for weaponry laws to weapon and war enthusiasts only?And, this is not serious at this point:
> > We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.
> 1122 employees of frontier AI companies
'lets us be the gatekeepers of AI because we know better than you'
access to gpu doesnt equal to any special insights to predict future of ai. Everyone and their grandma with enough compute is putting out a model now.
> It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.
This is just the standard AI "doomer" talking point. The dangers of AI just used as a compliment sandwich. "AI is so cool and awesome and that's why we need to slow down."
The real risks are much more boring and much less considered. For just one example:
AI is set to completely upset the Software Development industry. There are 1 to 3 million software developers in the US, depending on how you draw the line exactly. A low-few hundred thousand of those are visa workers.
With AI set to significantly cut into tech employment, what's going to happen to those visa workers? ... The Trump admin is not particularly subtle, even openly talking about retroactively revoking birthright citizenship.
AI & Tech companies love to talk about how their AI is going to hack everything if they're not careful. But not a peep about these ethics concerns.
Even if you [the reader] consider deporting visa workers the ethical move if the amount of jobs shrink, that's not something to be done unprepared or out of nowhere.
They just want some time to be able to sell the stock
Has there ever been a case in human history when technology just stopped advancing, simultaneously world wide, only because a bunch of people wanted it to?
I think these people are either incredibly naive, or, more likely, not acting in good faith.
https://en.wikipedia.org/wiki/Asilomar_Conference_on_Recombi...
> The effects of these guidelines are still being felt through the biotechnology industry and the participation of the general public in scientific discourse. Due to potential safety hazards, scientists worldwide had halted experiments using recombinant DNA technology, which entailed combining DNAs from different organisms. After the establishment of the guidelines during the conference, scientists continued with their research, which increased fundamental knowledge about biology and the public's interest in biomedical research.
DNA splicing, human cloning, super-sonic passenger flight, human gene editing (CRISPR), nuclear energy, biological weapons, and probably more.