The Belmont analogy is great, but the deeper point is even scarier: most of the industry is giving non-deterministic systems direct access to deterministic infrastructure (databases, shells, email, etc).
Historically we spent decades reducing automation privileges and adding layers of verification. Agents seem to be reversing that trend almost overnight.
> As long as the penalties for data breach are a slap on the wrist and buying everyone one year of credit monitoring, no one will.
And, of course, that one year is totally useless when one is subject to multiple breaches per year. Throw in the fact that so many breaches aren't even with a company that affected individuals have a direct relationship with, and it becomes virtually impossible to fix this.
At this point, I'd be in favor of making any company that handles personal data pay in advance for the monitoring, and get refunded when they prove that that OR THEIR PROVIDERS haven't had a data breach.
> I'd be in favor of making any company that handles personal data pay in advance
How about we start with some strict data privacy and handling laws? Make it so you straight up just can't collect & store personal information without proving that it's required and without it your business would not work (and no, data harvesting for advertising/marketing doesn't count).
Security is the problem, but it would be less of a problem if everyone wasn't trying to hoard as much data as possible from their customers for seemingly no reason at all. Take a scroll through the Play Store/App Store and look how many really simple apps request permissions for camera, microphone, location, local network, etc. for something like a metronome app that needs none of that.
There is a reason for hoarding data: it’s an asset on the balance sheet. So long as it is legal to liquidate data for cash, there will be incentives to collect and keep it.
The real riches are in starting a credit monitoring company. Vibe coded, of course, and if you have a data breach, then it's a perpetual motion machine.
The fact that the average joe can't start their own credit monitoring company as competition and the incumbents get away clean everytime they screw up says a lot about "capitalism" as we practice it
I think that's definitely true to a degree, but I think the think more companies are worried about is the reputational damage from the terrible press. Look at Solarwinds (not a data breach, but similar press around it). It erased hundreds of millions in shareholder value and the company was taken private at pennies on the dollar in the aftermath. There's real risk there.
For every Solarwinds, there are hundreds of breaches that never get more that a cursory reporting (if that). And Solarwinds is still in business (and some would call "taken private at pennies on the dollar" as a feature not a bug, but I digress), as are vastly more consequential examples (Equifax, anyone?).
Yes...reputational damage is a thing, but in my experience (sitting in the decision making meetings, as a participant, many, many times in my career) it's a second-tier player at the end of the day. This is especially true of data breaches...I cannot count the number of times (in the last decade particularly) where the decision point was "What reputation damage? Everyone and their mother has had a data breach. No one cares.". I don't think they're wrong.
This, like many issues of security and risk, is the consequence of the vast majority of the customers not caring. How many users dropped Facebook in 2019, or LinkedIn in 2021 (or 2012)? How many swore off Ticketmaster? Marriott? Adobe? eBay? And that's just ungodly massive breaches. So why would the average business give a steaming crap?
In my dark little heart of hearts I sometimes think "what would it take for the average person to actually care", and then I realize what that looks like, and I don't sleep well for a couple of nights. Cheers!
For people to care of would have to be like healthcare. The Change Healthcare breach cost 2B+ and led to a huge loss in market share. Or like AMCA, which went bankrupt after the breach (Labcorp's billing company). If you're a health tech company you can no longer insure your way out of the problem over you reach a certain size.
The reality is that we need data breaches to be painful but maybe not company ending events unless it really is sensitive data. As patio11 likes to say the right level of fraud is not zero. There's a middle ground where we can increase company liability or reduce the damage caused by a beach.
Goes to a lot of trouble to build a mental model / map / landscape of how agentic ops work. Worth the read if you're looking for one, reasonable people know the map is never the terrain.
https://arxiv.org/abs/2506.10077
followup paper coming soon which further demonstrates these contextuality results for a suite of models. there is no way to fundamentally impose on the training data or processing effective guardrails that can transcend this reality.
> Not only is this pure science fiction at this point, but injecting non-determinism into your defensive layer is terrifying and incredibly stupid. If you use an LLM to evaluate whether another LLM is doing something malicious, you now have two hallucination risks instead of one. You also risk a prompt-injection attack making it all the way to your security layer.
I've found fictional displays of "system compromise" kinda ridiculous in e.g. Halo. Now I know that Cortana throws AI slop input into AI slop infrastructure with thousands of subagents until she's in.
30 years ago, playing cyberpunk tabletop RPGs, my friends and I would laugh with each other at how silly the idea of major corporations hooking vital computer systems up to the internet would be.
The Belmont analogy is great, but the deeper point is even scarier: most of the industry is giving non-deterministic systems direct access to deterministic infrastructure (databases, shells, email, etc).
Historically we spent decades reducing automation privileges and adding layers of verification. Agents seem to be reversing that trend almost overnight.
As long as the penalties for data breach are a slap on the wrist and buying everyone one year of credit monitoring, no one will.
> As long as the penalties for data breach are a slap on the wrist and buying everyone one year of credit monitoring, no one will.
And, of course, that one year is totally useless when one is subject to multiple breaches per year. Throw in the fact that so many breaches aren't even with a company that affected individuals have a direct relationship with, and it becomes virtually impossible to fix this.
At this point, I'd be in favor of making any company that handles personal data pay in advance for the monitoring, and get refunded when they prove that that OR THEIR PROVIDERS haven't had a data breach.
> I'd be in favor of making any company that handles personal data pay in advance
How about we start with some strict data privacy and handling laws? Make it so you straight up just can't collect & store personal information without proving that it's required and without it your business would not work (and no, data harvesting for advertising/marketing doesn't count).
Security is the problem, but it would be less of a problem if everyone wasn't trying to hoard as much data as possible from their customers for seemingly no reason at all. Take a scroll through the Play Store/App Store and look how many really simple apps request permissions for camera, microphone, location, local network, etc. for something like a metronome app that needs none of that.
There is a reason for hoarding data: it’s an asset on the balance sheet. So long as it is legal to liquidate data for cash, there will be incentives to collect and keep it.
That is the point. Make it illegal, and not something that can be handwaved away by an EULA or TOS.
The real riches are in starting a credit monitoring company. Vibe coded, of course, and if you have a data breach, then it's a perpetual motion machine.
The fact that the average joe can't start their own credit monitoring company as competition and the incumbents get away clean everytime they screw up says a lot about "capitalism" as we practice it
I froze all my credit way back in 2016 or so and have never regretted it, not once. I wonder how effective it is, as my credit limit keeps going up.
I think that's definitely true to a degree, but I think the think more companies are worried about is the reputational damage from the terrible press. Look at Solarwinds (not a data breach, but similar press around it). It erased hundreds of millions in shareholder value and the company was taken private at pennies on the dollar in the aftermath. There's real risk there.
If only.
For every Solarwinds, there are hundreds of breaches that never get more that a cursory reporting (if that). And Solarwinds is still in business (and some would call "taken private at pennies on the dollar" as a feature not a bug, but I digress), as are vastly more consequential examples (Equifax, anyone?).
Yes...reputational damage is a thing, but in my experience (sitting in the decision making meetings, as a participant, many, many times in my career) it's a second-tier player at the end of the day. This is especially true of data breaches...I cannot count the number of times (in the last decade particularly) where the decision point was "What reputation damage? Everyone and their mother has had a data breach. No one cares.". I don't think they're wrong.
This, like many issues of security and risk, is the consequence of the vast majority of the customers not caring. How many users dropped Facebook in 2019, or LinkedIn in 2021 (or 2012)? How many swore off Ticketmaster? Marriott? Adobe? eBay? And that's just ungodly massive breaches. So why would the average business give a steaming crap?
In my dark little heart of hearts I sometimes think "what would it take for the average person to actually care", and then I realize what that looks like, and I don't sleep well for a couple of nights. Cheers!
For people to care of would have to be like healthcare. The Change Healthcare breach cost 2B+ and led to a huge loss in market share. Or like AMCA, which went bankrupt after the breach (Labcorp's billing company). If you're a health tech company you can no longer insure your way out of the problem over you reach a certain size.
The reality is that we need data breaches to be painful but maybe not company ending events unless it really is sensitive data. As patio11 likes to say the right level of fraud is not zero. There's a middle ground where we can increase company liability or reduce the damage caused by a beach.
Solarwinds YOY Revenue is up $100 million since then so even Solarwinds didn't take that big of a hit.
I think it's better to compare data breaches to data breaches, like when Adobe got breached. Or Oracle. Or Rockstar.
Nothing happened in the grand-scheme of things. Even after Oracle lied and pulled some shady tactics to downplay what happened.
A few years ago Crowdstrike took down the entire set of corporate computers and everyone still uses Falcon. There is simply no accountability anymore
Goes to a lot of trouble to build a mental model / map / landscape of how agentic ops work. Worth the read if you're looking for one, reasonable people know the map is never the terrain.
FYI I believe the idiom is, 'the map is never the territory'.
i do https://github.com/npc-worldwide/npcpy
https://arxiv.org/abs/2506.10077 followup paper coming soon which further demonstrates these contextuality results for a suite of models. there is no way to fundamentally impose on the training data or processing effective guardrails that can transcend this reality.
> Not only is this pure science fiction at this point, but injecting non-determinism into your defensive layer is terrifying and incredibly stupid. If you use an LLM to evaluate whether another LLM is doing something malicious, you now have two hallucination risks instead of one. You also risk a prompt-injection attack making it all the way to your security layer.
I've found fictional displays of "system compromise" kinda ridiculous in e.g. Halo. Now I know that Cortana throws AI slop input into AI slop infrastructure with thousands of subagents until she's in.
Anyone know how many data breaches occur on a monthly basis that would require credit monitoring?
I think the election of Trump was perfectly aligned with the rise of LLMs.
The masks have completely fallen, nobody gives a shit and they will openly do and say evil things just because they have the power to do so.
You know how in video games literally everything is super easy to hack?
Turns out all those games were just very forward-thinking.
30 years ago, playing cyberpunk tabletop RPGs, my friends and I would laugh with each other at how silly the idea of major corporations hooking vital computer systems up to the internet would be.
Convenience beats anything else.