As someone who spent over 20 years active duty, and spent a ton of my career in the IT, security, etc. side of the house:
Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
That said, the timing of the disclosure and the issue are rather concerning.
Regarding the highest value targets to hit with an attack like this, you would want to target Guam, Hawai'i, and other isolated overseas locations where this would have ripple effects in the local economy. Guam specifically would cause catastrophic supply shortages, since DeCA probably supplies around 50% of the groceries on that island (that's a WAG based on my time there).
Generally agree with your assessment, but in the case of Guam or other more remote installations if there were catastrophic issues we'd just airlift food in. Costly but certainly manageable.
Hawaii I'm not sure why that would be an issue unless the whole island was attacked or shut down. Even if the on-base shops were hacked you could just go shop at Wal-Mart or Costco or any number of other locations on the islands. If there was an extended issue then the commander could authorize meal stipends as they do for some units today and then you would just go buy food off-base. Ideal? No. Manageable? Very much.
I always wonder by folks with some "inside" knowledge like you will then come out to share more details. Why? I understand that there's no security through obscurity, but I don't think that the details to get your point across matters. Loose lips sink ships.
>Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
Are these materially different? Assuming that there wasn't a literal command to turn off all freezers, from an organizational, non criminal perspective, to the organization the damage will be the same, the root cause will be a bug (whether exploited by chance or malice), and the fix will be the same(fixing the bug).
A couple years ago I worked on a service that had to communicate with a Siemens S7-1500 PLC. Based on my experience with that project, none of what I’ve read recently about unsecured industrial PLCs is surprising.
I opened Siemens TIA Portal and PLCSIM for the first time and thought “wow, I didn’t think the Windows 95 GUI library was still supported.” None of the PLC contractors we had hired knew how to enable TLS on the thing (user/pass eg admin/admin was their usual). Anecdote: I once spent hours reading the docs and clicking around trying to get it to accept an SSL certificate signed by a real CA and it wouldn’t go, but it accepted one I self-signed in openssl.
In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
Most factories I know do not allow their PLC be accessed from the internet. They are usually on a separate Network. However, the "engineering" station (the computer running e.g. TIA Portal) sometimes is.
The PLC engineers I had contact with usually had an electrical engineering background. That's why they like PLCs in the first place with the ladder logic programming languages, grafcet and if they feel fancy a bit of structured text (assembly like) or structured control language (pascal like). They indeed did not know much about software security but a great deal about machine safety.
A real security nightmare are older OPC servers (OPC-DA) which is super reliant on DCOM. OPC is quite important to connect the PLCs to SCADA systems or 3rd party devices.
Isn't this the industry expectation in that kind of equipment? If it was signed by a real CA the cert. could expire and render the equipment unable to communicate.
My mind was blown when I realized that the way tftp works is that as the machine is booting it asks the network if anyone has some software for it to run.
> In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
Stuxnet was over a decade ago.
There should be a simple rule that everyone with the ability to understand things like PLCs should be able to grasp: your equipment does not touch the internet or external storage, period.
Those who can't grasp this concept should be shown the door with a recommendation that they find a less mentally-taxing line of work.
This is eerily suggestive of a vulnerability Hank Paulson hinted at in his 2014 book "Dealing with China"
"Every nonelectric cooler comes with 25 years of free real-time monitoring. On a visit to the company in the spring of 2012, I watched as technicians in Broad Air’s space-age control room checked on the performance of its units in locations as diverse as the Adolfo Suárez Madrid–Barajas Airport in Spain; Qualcomm headquarters in San Diego, California; and Fort Stewart, the U.S. Army base in Georgia [emphasis added].
"Zhang says that 80 percent of his clients are repeaters. “If you bring long-term benefits for your clients, they will choose you.”"
America's tech-sector has a similar problem, which--until recently--was tempered by the idea that it was a dependable and predictable ally to most of its customers.
Not just in the sense of secretive kill-switches, but "US government commands you to turn over this encryption key and you're not allowed to say you did so" stuff.
The author doesn't really claim it was a hack, just that it is a possibility. But they are charging down the path of the potential hack before asking the more obvious question: How many refrigerators exist in the military at all? And of those, how many are having problems?
Because a half dozen a day sounds plausible as standard maintenance issues, as the author acknowledges. If it were a hack, I'd expect something like 50% of them to have problems. But not knowing how many there are, I don't know how significant these incidents really are.
They are charging down that path because vulnerabilities that effect the refrigerators were disclosed the same day as 14 refrigerators failed in an absurd way. They all turned on the defrost cycle and heated the food.
The problem is the author should have put a few concise bullet points at the top. (14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.)
I really recommend skimming the article to the end.
(Unfortunately, the article really is so verbose it's a borderline rant.)
First let's acknowledge that this could very easily be a misconfiguration issue.
But, I'd be a lot more inclined to that idea if it wasn't for how they failed: they started a defrost cycle that turned the freezers into heaters, spoiling the food quicker. And the failure happened overnight, delaying discovery of the problem.
It could be just a compounding of bad luck. But an attacker with access to the specs for the freezers might be aware of how long they would stay cold after being simply shut off.
I was thinking timezone update - because of course an internet-connected freezer needs updated zone info so one can configure defrost schedules without the benefit of UTC.
Maybe the whole point of this was not to ruin some food, but to prove infiltration of a DoD network in a way that would leak broadly because it's not classified?
I think it is worth mentioning that at least a couple of these bases are pretty critical from a natsec standpoint. Fort Huachuca is a big IT and secure communications installation (United States Army Network Enterprise Technology Command, the United States Army Intelligence Center, Intelligence and Electronic Warfare Directorate); F.E. Warren AFB is one of the three AFBs that operate the strategic nuclear ICBM fleet. Not saying that any classified systems were potentially hacked/at risk in this situation
Probably not, but my closest bet would fall to Hanlon's razor:
I was curious if this was continued evidence of poor appropriations and upkeep or what... I do see "U.S. military commissary refrigeration maintenance, equipment replacement, and physical infrastructure are funded through the 5% commissary surcharge paid by customers at checkout rather than direct congressional appropriations."
So, perhaps the first place would be to follow the money - are these being repaired at the proper rate? Is this repair outsourced to third party vendors? (my guess). Is this gonna end up being the McDonald's Ice Cream machine all over again?
Really though, Hanlon's would be much easier to believe this is yet further ineptitude by those who run things (I am not going to claim malfeasance/malevolence, except a general sense of such across the board by this admin).
Since I'm not on the inside, anything I have to say would be speculative, just like the above, or the author themselves (I have no idea who it is, and perhaps they have a better beat on the ground with regards to this), but it just falls in line with "we're running out of missiles" and "sailors attempting to kill themselves".
We're so insistent on being #1, we can't admit we're in a society that is falling apart (and again, it may be the case that this IS a hack, but if I were to place my bets...)
Ineptitude, lowest cost players, etc "efficiency" indeed. You get what you pay for, and I guess 5% don't pay for a whole hell of a lot these days.
Howdy y’all, author here. Just discovered this thread after wondering why Hacker News was a linked views source to my silly little freezergate braindump.
Wanted to offer a few clarifications:
I’m not a cybersecurity expert; I’m an investigator (in a totally different field), and this was essentially me following a weird thought to see where it went. My background is in natsec so that’s where my mind goes. There have since been at least dozens of additional freezer outages reported in a similar pattern, but I'm refraining from calling/tracking down every individual weirdness based on a Facebook comment at this point since there are much larger outlets with journalists more proficient than I covering this by now. Another interesting thing - There are numerous freezers and fridges within base stores, not under the control of DeCA/DoD, and none of those appear to be impacted.
I completely agree that a bad update/configuration or other shared technical failure may be the much more boring answer. The interesting part to me is that potentially hundreds of varying systems can converge upstream into common monitoring/control infrastructure. Anyone on this forum probably understands that, however IoT was something that was a relatively new concept when I was in school, and my degrees were somewhat relevant. The average person is blissfully unaware how expansive (and how much work behind safeguarding) the IoT is.
Also, since it came up: Yup. Human written. I’m pretty firmly anti-AI as a writer and also just, like, societally. I'll be sure to add some sort of footnote detailing my ai usage at the bottom of future public facing work, because I too detest reading (or questioning if what I'm reading is) slop. Minimal LLM used for understanding technical concepts and what the fuck fridge norms are…The weird formatting, excessive bolding, neurotic parentheticals, and rant energy are, unfortunately, totally my own. Sorry guys.
This was my first ever public post and intended audience was ~ 10 friends forced to read my diatribe, not thousands of strangers very validly raising questions I am not smart enough to answer myself. Appreciate the discussion and will be further educating myself on some of the points a few of you have brought up.
Hi, I'm the poster. Sorry for the unexpected attention! I saw this on Bluesky and thought it was interesting enough to share here. If anything the style of your writing makes it stand out in a good way, we shouldn't always have polished/corporate-speak posts here.
Even if it isn't an attack, you demonstrated how it could be one. The vulnerability is likely there and worth mitigating. Excellent investigation and write-up!
I thought this was excellent. It pulled together a whole bunch of interesting evidence, was very careful not to make claims that weren't supported by that evidence, and every time I had a question it answered my question in the next paragraph. Hope you publish more!
I came to these comments coz I was curious about the AI usage here, and FWIW I also thought it smelled AI written, but I didn't think it was slop. (IMO not all AI output is slop and not all slop comes from AI).
My main question was "did Claude do the investigation by itself or just write up the article from someone's notes?"
Also though, I'm quite willing to believe this is human written and the human just happens to have a Claudey style. The actual prose isn't that Claudey it's just the structure of how it presents ideas. But, Claude had to get that structure from somewhere. It's not that surprising to see people with that style of communication.
The article posits that they may just have centralized monitoring, not control. Since this didn't affect every location, I would assume it was a shared default password or something
Because they’re prefabbed walk-in coolers or freezers (at minimum, they could be purpose built cold storage warehouses) with multiple condensers, evaporators, pumps, temperature sensors, and humidity sensors. The refrigeration equipment needs some sort of control system and direct digital control is the usual way to do that these days.
This is food storage for a commissary, aka a store. They don’t use residential refrigerators.
Never attribute to malice what can be explained by "military intelligence". If they're all controlled by remote monitoring from one location, then an engineer can run a loop to change a setting, and it can turn on the wrong setting.
There doesn't seem to be anything indicating an advanced attack. If you're a foreign adversary and you want to flex your muscles (in a way that would be a borderline act of war), you don't just flip one switch and giggle about spoiled food.
To summarize for people who TLDR: 14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.
Regardless if this was a hack or a bug, the bigger lesson is that overcomplicated systems fail in catastrophic ways. Why do military commissaries need remote-controlled freezers? It seems like a very fragile, and needless, way to run a freezer.
---
But, there are some options that the author didn't consider:
1: This could be a quickly applied patch that failed.
2: This could be a "script kiddie" hack from someone who isn't a government actor.
I'm less onboard with a state actor. Generally, when a state actor has hacked something, they don't want the victim to know. In this case, if it was a state actor, I would anticipate that they would make a single freezer fail in a way that they could verify using something like a hacked video camera or otherwise by watching public social media feeds. IMO: A state actor would only "make sense" if they knew the hole was closing soon and they don't care if they're discovered, perhaps because their operation is winding down.
Could also just be a bunch of IoT devices running on synchronized time source that have what amounts to a defrost cronjob. And a latent bug that due to everything being synced and on the same schedule failed in the same manner (eg crashed the controller immediately after turning the heating element on and either didn’t boot back up or booted and some shitty programming forgot to check the heating element status). Saving state across boots or having a race condition of some sort like this gets my vote - since it’s a pretty common failure mode junior programmers are susceptible to.
I’ve seen similar too many times in my career - synchronized clocks are great until someone deploys a cronjob that fires at exactly the same time across the fleet and it breaks a certain OS or firmware revision. Which then exposes another bug that exists globally.
I definitely still subscribe to don’t attribute to malice what you can attribute to incompetence mindset.
Certainly all speculation though, real root cause will be interesting if it ever becomes public.
Iran is actively looking for ways to attack back against the United States especially against military targets without actually escalating the situation.
I'm sure some script kiddie broke into a government network, hacked an industrial process, and forced a limited supply piece of equipment into a failure mode that takes some thought and is more unique as an attack vector. It's just like buying hacks for CS source right?
These aren’t your typical refrigerator or freezer, these facilities have walk-ins or purpose built cold storage with multiple evaporators and condensing units. A building automation system is pretty standard for most buildings above a certain size, and monitoring and controlling the refrigeration is usually a part of it.
Unfortunately, I would wager that all BAS software is full of flaws and holes, allowing access to it for the public internet seems like a bad idea. I need to be on my company VPN to access our locally hosted BAS front end (which I have authorized access for) which seems like the bare minimum security.
This would be a bigger deal for the commissary locations outside the US, though I see none are on the list. Many of the very junior enlisted make very little money (~2400USD/month), and the low pricing at the commissary helps quite a lot. In the US, you would typically have some affordable off-base options. Overseas, it depends. Many of the locations are remote, or in places where the local groceries are significantly more expensive.
Yeah I don't know why "hack" is more obvious than this. Central control pushes an update, it bugs out and cooks a dozen commissaries' frozen foods. Smart hack would be to do this randomly and fly under the radar.
I'm in the firmware bug camp too. Over/under on "the remote management server went down and a bug on all the freezers decided to put them back into some form of local control where its first action was to do a defrost cycle then put it back into offline service"?
Freezer's went into defrost melting all the frozen food and ice. If they just go down you have days to weeks to respond before everything unfreezes just based on the thermal mass and size of the freezer.
This is a much bigger deal than the freezer being down.
What's funny is that trump of all people is banning a lot of crap like this - internet connected chinese cars, internet connected solar panels, and other utility stuff.
But I fear the vested/wealthy interests involved in iot data mining, advertising, "relationship management" and plain "we own this"...
It (probably) prevents a comprehensive law supporting common sense.
There's a far simpler explanation than some outside actor (either state sponsored or otherwise) deciding that the best thing they can do is to muck around with freezers.
We know there's been a severe rot of operational capabilities in the military thanks for Hegseths purges and general stupidity. It's entirely possible and quite likely that over the course of his various drunken binges he decided to get rid of people who were in charge of operational control for stuff like freezers across military bases.
I would agree, but the freezers going into high heat defrost mode seems like an intentional action from someone, whether that be incompetence or malice on the side of DeCA, or malice from a third party. If they got rid of the people commanding the freezers what to do, I feel like they'd just stay on whatever mode they were already on, rather than suddenly command all the freezers to defrost
Oops, you're absolutely right to call me out for that. Starting the defrost cycle without emptying the freezer first COULD lead to spoilage. The load-bearing temperature is 0 degrees Celsius — above that point, and frozen food starts to go bad.
It'll take them two weeks and then they'll find the systems were hacked half a year ago and they had industrial robots write messages on a literal chalkboard in order to share progress.
As someone who spent over 20 years active duty, and spent a ton of my career in the IT, security, etc. side of the house:
Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
That said, the timing of the disclosure and the issue are rather concerning.
Regarding the highest value targets to hit with an attack like this, you would want to target Guam, Hawai'i, and other isolated overseas locations where this would have ripple effects in the local economy. Guam specifically would cause catastrophic supply shortages, since DeCA probably supplies around 50% of the groceries on that island (that's a WAG based on my time there).
Generally agree with your assessment, but in the case of Guam or other more remote installations if there were catastrophic issues we'd just airlift food in. Costly but certainly manageable.
Hawaii I'm not sure why that would be an issue unless the whole island was attacked or shut down. Even if the on-base shops were hacked you could just go shop at Wal-Mart or Costco or any number of other locations on the islands. If there was an extended issue then the commander could authorize meal stipends as they do for some units today and then you would just go buy food off-base. Ideal? No. Manageable? Very much.
I always wonder by folks with some "inside" knowledge like you will then come out to share more details. Why? I understand that there's no security through obscurity, but I don't think that the details to get your point across matters. Loose lips sink ships.
>Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
Are these materially different? Assuming that there wasn't a literal command to turn off all freezers, from an organizational, non criminal perspective, to the organization the damage will be the same, the root cause will be a bug (whether exploited by chance or malice), and the fix will be the same(fixing the bug).
A couple years ago I worked on a service that had to communicate with a Siemens S7-1500 PLC. Based on my experience with that project, none of what I’ve read recently about unsecured industrial PLCs is surprising.
I opened Siemens TIA Portal and PLCSIM for the first time and thought “wow, I didn’t think the Windows 95 GUI library was still supported.” None of the PLC contractors we had hired knew how to enable TLS on the thing (user/pass eg admin/admin was their usual). Anecdote: I once spent hours reading the docs and clicking around trying to get it to accept an SSL certificate signed by a real CA and it wouldn’t go, but it accepted one I self-signed in openssl.
In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
Most factories I know do not allow their PLC be accessed from the internet. They are usually on a separate Network. However, the "engineering" station (the computer running e.g. TIA Portal) sometimes is.
The PLC engineers I had contact with usually had an electrical engineering background. That's why they like PLCs in the first place with the ladder logic programming languages, grafcet and if they feel fancy a bit of structured text (assembly like) or structured control language (pascal like). They indeed did not know much about software security but a great deal about machine safety.
A real security nightmare are older OPC servers (OPC-DA) which is super reliant on DCOM. OPC is quite important to connect the PLCs to SCADA systems or 3rd party devices.
Isn't this the industry expectation in that kind of equipment? If it was signed by a real CA the cert. could expire and render the equipment unable to communicate.
My mind was blown when I realized that the way tftp works is that as the machine is booting it asks the network if anyone has some software for it to run.
> In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
Stuxnet was over a decade ago.
There should be a simple rule that everyone with the ability to understand things like PLCs should be able to grasp: your equipment does not touch the internet or external storage, period.
Those who can't grasp this concept should be shown the door with a recommendation that they find a less mentally-taxing line of work.
This is eerily suggestive of a vulnerability Hank Paulson hinted at in his 2014 book "Dealing with China"
"Every nonelectric cooler comes with 25 years of free real-time monitoring. On a visit to the company in the spring of 2012, I watched as technicians in Broad Air’s space-age control room checked on the performance of its units in locations as diverse as the Adolfo Suárez Madrid–Barajas Airport in Spain; Qualcomm headquarters in San Diego, California; and Fort Stewart, the U.S. Army base in Georgia [emphasis added].
"Zhang says that 80 percent of his clients are repeaters. “If you bring long-term benefits for your clients, they will choose you.”"
America's tech-sector has a similar problem, which--until recently--was tempered by the idea that it was a dependable and predictable ally to most of its customers.
Not just in the sense of secretive kill-switches, but "US government commands you to turn over this encryption key and you're not allowed to say you did so" stuff.
The author doesn't really claim it was a hack, just that it is a possibility. But they are charging down the path of the potential hack before asking the more obvious question: How many refrigerators exist in the military at all? And of those, how many are having problems?
Because a half dozen a day sounds plausible as standard maintenance issues, as the author acknowledges. If it were a hack, I'd expect something like 50% of them to have problems. But not knowing how many there are, I don't know how significant these incidents really are.
They are charging down that path because vulnerabilities that effect the refrigerators were disclosed the same day as 14 refrigerators failed in an absurd way. They all turned on the defrost cycle and heated the food.
The problem is the author should have put a few concise bullet points at the top. (14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.)
I really recommend skimming the article to the end.
(Unfortunately, the article really is so verbose it's a borderline rant.)
Obvious sabotage would be addressed promptly. Subtle sabotage could persist as a minor torment indefinitely.
The article has a post that says this happened across 14 bases at the same time.
First let's acknowledge that this could very easily be a misconfiguration issue.
But, I'd be a lot more inclined to that idea if it wasn't for how they failed: they started a defrost cycle that turned the freezers into heaters, spoiling the food quicker. And the failure happened overnight, delaying discovery of the problem.
It could be just a compounding of bad luck. But an attacker with access to the specs for the freezers might be aware of how long they would stay cold after being simply shut off.
I was thinking timezone update - because of course an internet-connected freezer needs updated zone info so one can configure defrost schedules without the benefit of UTC.
Maybe the whole point of this was not to ruin some food, but to prove infiltration of a DoD network in a way that would leak broadly because it's not classified?
I think it is worth mentioning that at least a couple of these bases are pretty critical from a natsec standpoint. Fort Huachuca is a big IT and secure communications installation (United States Army Network Enterprise Technology Command, the United States Army Intelligence Center, Intelligence and Electronic Warfare Directorate); F.E. Warren AFB is one of the three AFBs that operate the strategic nuclear ICBM fleet. Not saying that any classified systems were potentially hacked/at risk in this situation
Probably not, but my closest bet would fall to Hanlon's razor:
I was curious if this was continued evidence of poor appropriations and upkeep or what... I do see "U.S. military commissary refrigeration maintenance, equipment replacement, and physical infrastructure are funded through the 5% commissary surcharge paid by customers at checkout rather than direct congressional appropriations."
So, perhaps the first place would be to follow the money - are these being repaired at the proper rate? Is this repair outsourced to third party vendors? (my guess). Is this gonna end up being the McDonald's Ice Cream machine all over again?
Really though, Hanlon's would be much easier to believe this is yet further ineptitude by those who run things (I am not going to claim malfeasance/malevolence, except a general sense of such across the board by this admin).
Since I'm not on the inside, anything I have to say would be speculative, just like the above, or the author themselves (I have no idea who it is, and perhaps they have a better beat on the ground with regards to this), but it just falls in line with "we're running out of missiles" and "sailors attempting to kill themselves".
We're so insistent on being #1, we can't admit we're in a society that is falling apart (and again, it may be the case that this IS a hack, but if I were to place my bets...)
Ineptitude, lowest cost players, etc "efficiency" indeed. You get what you pay for, and I guess 5% don't pay for a whole hell of a lot these days.
Howdy y’all, author here. Just discovered this thread after wondering why Hacker News was a linked views source to my silly little freezergate braindump.
Wanted to offer a few clarifications:
I’m not a cybersecurity expert; I’m an investigator (in a totally different field), and this was essentially me following a weird thought to see where it went. My background is in natsec so that’s where my mind goes. There have since been at least dozens of additional freezer outages reported in a similar pattern, but I'm refraining from calling/tracking down every individual weirdness based on a Facebook comment at this point since there are much larger outlets with journalists more proficient than I covering this by now. Another interesting thing - There are numerous freezers and fridges within base stores, not under the control of DeCA/DoD, and none of those appear to be impacted.
I completely agree that a bad update/configuration or other shared technical failure may be the much more boring answer. The interesting part to me is that potentially hundreds of varying systems can converge upstream into common monitoring/control infrastructure. Anyone on this forum probably understands that, however IoT was something that was a relatively new concept when I was in school, and my degrees were somewhat relevant. The average person is blissfully unaware how expansive (and how much work behind safeguarding) the IoT is.
Also, since it came up: Yup. Human written. I’m pretty firmly anti-AI as a writer and also just, like, societally. I'll be sure to add some sort of footnote detailing my ai usage at the bottom of future public facing work, because I too detest reading (or questioning if what I'm reading is) slop. Minimal LLM used for understanding technical concepts and what the fuck fridge norms are…The weird formatting, excessive bolding, neurotic parentheticals, and rant energy are, unfortunately, totally my own. Sorry guys.
This was my first ever public post and intended audience was ~ 10 friends forced to read my diatribe, not thousands of strangers very validly raising questions I am not smart enough to answer myself. Appreciate the discussion and will be further educating myself on some of the points a few of you have brought up.
Hi, I'm the poster. Sorry for the unexpected attention! I saw this on Bluesky and thought it was interesting enough to share here. If anything the style of your writing makes it stand out in a good way, we shouldn't always have polished/corporate-speak posts here.
Even if it isn't an attack, you demonstrated how it could be one. The vulnerability is likely there and worth mitigating. Excellent investigation and write-up!
I thought this was excellent. It pulled together a whole bunch of interesting evidence, was very careful not to make claims that weren't supported by that evidence, and every time I had a question it answered my question in the next paragraph. Hope you publish more!
I came to these comments coz I was curious about the AI usage here, and FWIW I also thought it smelled AI written, but I didn't think it was slop. (IMO not all AI output is slop and not all slop comes from AI).
My main question was "did Claude do the investigation by itself or just write up the article from someone's notes?"
Also though, I'm quite willing to believe this is human written and the human just happens to have a Claudey style. The actual prose isn't that Claudey it's just the structure of how it presents ideas. But, Claude had to get that structure from somewhere. It's not that surprising to see people with that style of communication.
My only question is, why would all refrigeration be under the remote control of DECA? That seems unnecessarily complicated.
The article posits that they may just have centralized monitoring, not control. Since this didn't affect every location, I would assume it was a shared default password or something
This _IS_ the U.S. Government.
Because they’re prefabbed walk-in coolers or freezers (at minimum, they could be purpose built cold storage warehouses) with multiple condensers, evaporators, pumps, temperature sensors, and humidity sensors. The refrigeration equipment needs some sort of control system and direct digital control is the usual way to do that these days.
This is food storage for a commissary, aka a store. They don’t use residential refrigerators.
Single source systems provider and integrator and a doom date?
Could be a hack or a design flaw. I await the root cause analysis.
Never attribute to malice what can be explained by "military intelligence". If they're all controlled by remote monitoring from one location, then an engineer can run a loop to change a setting, and it can turn on the wrong setting.
There doesn't seem to be anything indicating an advanced attack. If you're a foreign adversary and you want to flex your muscles (in a way that would be a borderline act of war), you don't just flip one switch and giggle about spoiled food.
To summarize for people who TLDR: 14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.
Regardless if this was a hack or a bug, the bigger lesson is that overcomplicated systems fail in catastrophic ways. Why do military commissaries need remote-controlled freezers? It seems like a very fragile, and needless, way to run a freezer.
---
But, there are some options that the author didn't consider:
1: This could be a quickly applied patch that failed.
2: This could be a "script kiddie" hack from someone who isn't a government actor.
I'm less onboard with a state actor. Generally, when a state actor has hacked something, they don't want the victim to know. In this case, if it was a state actor, I would anticipate that they would make a single freezer fail in a way that they could verify using something like a hacked video camera or otherwise by watching public social media feeds. IMO: A state actor would only "make sense" if they knew the hole was closing soon and they don't care if they're discovered, perhaps because their operation is winding down.
Could also just be a bunch of IoT devices running on synchronized time source that have what amounts to a defrost cronjob. And a latent bug that due to everything being synced and on the same schedule failed in the same manner (eg crashed the controller immediately after turning the heating element on and either didn’t boot back up or booted and some shitty programming forgot to check the heating element status). Saving state across boots or having a race condition of some sort like this gets my vote - since it’s a pretty common failure mode junior programmers are susceptible to.
I’ve seen similar too many times in my career - synchronized clocks are great until someone deploys a cronjob that fires at exactly the same time across the fleet and it breaks a certain OS or firmware revision. Which then exposes another bug that exists globally.
I definitely still subscribe to don’t attribute to malice what you can attribute to incompetence mindset.
Certainly all speculation though, real root cause will be interesting if it ever becomes public.
Iran is actively looking for ways to attack back against the United States especially against military targets without actually escalating the situation.
I'm sure some script kiddie broke into a government network, hacked an industrial process, and forced a limited supply piece of equipment into a failure mode that takes some thought and is more unique as an attack vector. It's just like buying hacks for CS source right?
>Generally, when a state actor has hacked something, they don't want the victim to know
Could be the Iranians, or someone aligned, conducting anti-morale operations. Could be the start of a series of small but annoying failures.
These aren’t your typical refrigerator or freezer, these facilities have walk-ins or purpose built cold storage with multiple evaporators and condensing units. A building automation system is pretty standard for most buildings above a certain size, and monitoring and controlling the refrigeration is usually a part of it.
Unfortunately, I would wager that all BAS software is full of flaws and holes, allowing access to it for the public internet seems like a bad idea. I need to be on my company VPN to access our locally hosted BAS front end (which I have authorized access for) which seems like the bare minimum security.
This would be a bigger deal for the commissary locations outside the US, though I see none are on the list. Many of the very junior enlisted make very little money (~2400USD/month), and the low pricing at the commissary helps quite a lot. In the US, you would typically have some affordable off-base options. Overseas, it depends. Many of the locations are remote, or in places where the local groceries are significantly more expensive.
I would suspect a firmware bug. Or a "Service Required" timer that was ignored.
Yeah I don't know why "hack" is more obvious than this. Central control pushes an update, it bugs out and cooks a dozen commissaries' frozen foods. Smart hack would be to do this randomly and fly under the radar.
I'm in the firmware bug camp too. Over/under on "the remote management server went down and a bug on all the freezers decided to put them back into some form of local control where its first action was to do a defrost cycle then put it back into offline service"?
That's... a lot of words to say "freezers are down", with very little actual substance.
Freezer's went into defrost melting all the frozen food and ice. If they just go down you have days to weeks to respond before everything unfreezes just based on the thermal mass and size of the freezer.
This is a much bigger deal than the freezer being down.
Very interesting article, very neurotically written. Definitely got grating by the end.
The bold text use make me think it was largely LLM-written. Maybe even LLM-researched.
Welcome to the internet of shitty unsupported and insecure crap! Are we really this dumb as a society?
As the saying goes, the S in IoT stands for security
Not exactly strong evidence presented here, but it wouldn't be a surprise either
What's funny is that trump of all people is banning a lot of crap like this - internet connected chinese cars, internet connected solar panels, and other utility stuff.
But I fear the vested/wealthy interests involved in iot data mining, advertising, "relationship management" and plain "we own this"...
It (probably) prevents a comprehensive law supporting common sense.
How many people do you know that have always on microphones in their home so that they buy things from amazon or google trivia answers?
Yes
it's not AI generated so it must be true
Would be hilarious if this was a runaway AI that someone was using to control their own IoT fridge.
> "I'm sorry I'm familiar with that function. Let me research enabling defrost for you."
The vulnerability research paper article mentions has a title that I could imagine an LLM take as an instruction - or a challenge.
Gilfoyle was here. Everything that has been mocked in the Silicon Valley Show has either already happened or will happen.
There's a far simpler explanation than some outside actor (either state sponsored or otherwise) deciding that the best thing they can do is to muck around with freezers.
We know there's been a severe rot of operational capabilities in the military thanks for Hegseths purges and general stupidity. It's entirely possible and quite likely that over the course of his various drunken binges he decided to get rid of people who were in charge of operational control for stuff like freezers across military bases.
I would agree, but the freezers going into high heat defrost mode seems like an intentional action from someone, whether that be incompetence or malice on the side of DeCA, or malice from a third party. If they got rid of the people commanding the freezers what to do, I feel like they'd just stay on whatever mode they were already on, rather than suddenly command all the freezers to defrost
To be fair, if you want to mess with your adversaries troop morale, screwing up dinner is pretty effective.
General Stupidity should probably be demoted for this.
This is exactly the sort of thing that a saboteur would want its targets to think.
“If sovereign and subject are in accord, put division between them.” —Sun Tzu, The Art of War
I'm waiting for the OpenAI report that their agents defrosted everything.
Oops, you're absolutely right to call me out for that. Starting the defrost cycle without emptying the freezer first COULD lead to spoilage. The load-bearing temperature is 0 degrees Celsius — above that point, and frozen food starts to go bad.
It'll take them two weeks and then they'll find the systems were hacked half a year ago and they had industrial robots write messages on a literal chalkboard in order to share progress.
Anthropic: “We opened the pantry door!”
"To be very clear: I do not have evidence that the Defense Commissary Agency was hacked."
Should be much closer to the top of the article. Otherwise this is just weird and potentially dangerously wrong research.
Can you spell out the danger this blog post represents?
what's with the pearl clutching?