A natural evolution of engineers losing touch with the customers and users.
I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.
I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.
I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.
Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform.
I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about.
At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or uses terminology interchangably, which confuses me. It sounds plausible, but it does not actually translate into a working system or shared understanding.
Then one day I did some pair programming with him and whenever there was an error or a resource missing, he would type it into the LLM, copy paste it out of it and then brute force error messages. He did not even wait a second to think or reconcile whats happening on the screen or what the exact requirement is. Never taking one step back and questioning any assumption.
Now that the timeline is shifting and everyone starts to be stressed, he continues to vibe code through me and it is so tiring, there is no higher level planning or architecture, its just a reactive type of trial and error to be faster.
It feels like these people are so used to talking to bots, that they treat you like an agent they can chat to or talk through monologs with.
It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...
So right now I'm considering quitting IT for a couple of years until people calm down, but I think its pretty futile
I don't think it's generalizable. The kind of person who copy and pastes from the AI is the kind who did the same from StackOverflow before. It's more compelling, and we probably see more of them because of it, but it's the same general thing.
The kind of person who insists on understanding things and working through the problem has always been rarer. It's not "humble", it's "inquisitive" and "persistent".
I'm seeing people who _used to be_ like that losing that understanding without realizing it's happening - they have a superficial idea of what the code is doing, enough to feel like they understand it, but the change is apparent when watching them handle something unexpected.
At my place, this is what they want. They want people to smash through things as fast as possible. They don’t want people to sit and craft a solution which takes in to account the whole. They are choosing tools which are low code, and use llm’s to produce what they need. as they say “this is the way things are going”.
You captured this phenomenon very well in this comment. Appreciate you sharing it because it’s hard to describe exactly what makes this sort of behavior so bizarre.
So quit pair programming. I never did, never will do that, nor worked at a place that remotely encouraged that. Each to their own, that's how it should be.
So how do you build competence in a world where AI is preached to be the most reasonable way to solve problems because it's supposed to be faster than humans?
It's the uncanny valley of AI. It's still not quite good enough that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.
I think LLMs have some of the same risks and benefits of stimulant drugs. They can make you more productive if used effectively as a tool, but they can also delude you into thinking you are better than you are and create a dependence such that you aren't just less productive without the LLM/drug, you fail to be productive at all because you don't know how to function without it.
I think there are many cases where it was the tech workers themselves who argued for isolation from the customer so that they may focus harder on whatever tasks. I used to be one of these workers. I argued very hard for it. I regret that today.
On the surface it seems rational, but it quickly turns into a system of perverse incentives because now the development team must maintain an illusion that they are constantly overwhelmed with tasks and could never hope to spare a microsecond to assist the customer. This misalignment is how you wind up building your own web frameworks and databases from scratch. It turns into a self serving monster that eventually dominates the entire business. From the perspective of the business, many of these development teams look like they're behind some modern day iron curtain.
If the solution is so simple, why claude did not found it? At this point we can assume, it is better than 90% of engineers (including me).
After three decades of outsourcing to lowest bidder, I do not buy that humans are somehow better!
> patient, old school troubleshooting
I usually see similar arguments around systems with major red flags (no docs, poor CI, decade ago no CVS...). And engineers with private stash of workarounds for job security!
Claude does not do anything special.
Or perhaps claude was misconfigured, it had no access to relevant part of system, and it tryied to work within its limitation. Often it means decompiling binaries in desperate loop...
The funny thing is that if you never understand the codebase then you will keep thinking Claude is doing a great work delivering all this incredible software, when all it has done is created unnecessary tech debt.
> A twenty line for loop. It overengineers most things.
Anecdote I like to tell.. I was working on a financial planning software, intentionally purely vibe coded as an experiment.
I eventually discovered AI had implemented seven duplicate copies of tax calculation functions. All of them different. All of them wrong. All of them giving different answers for same input.
Not even the most junior of newbie junior engineers would do something this crazy. But AI was happy to do it. It will solve the immediate problem, efficiently. Even if the most efficient solution is something ridiculous like this.
I am former java enterprise dev, so yes I often code this way. Unit testing, decomposition... Some projects CI refuse to merge commits with 20 line loop and duplicated code...
But that is not a point. Claude can code tight compact loops, it just needs to be instructed to do so! If it does "enterprise code", it means it had no instructions about code style.
If your documentation, spec, agent.md does not have proper guidance on coding style... yet another red flag!
So after 30 years of outsourcing to the bottom 10%, you think Claude is better than the bottom 90% even though it’s so stupid that it doesn’t even know it should ask for advice or more information when it’s stuck?
Because simplicity is hard and often the result of careful thought. Anybody can keep piling pile of shit on top of pile of shit which is why that sort of code is so common in our industry.
While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:
Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.
(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)
Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.
> At this point we can assume, it is better than 90% of engineers (including me).
Hard disagree. We absolutely cannot assume that. You can posit it, and we can have an informed debate about it. This is what irks me the most about LLM fans: they constantly try to reframe the debate to have their worldview as the agreed-upon starting point.
The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.
No worries, at some point your hit the wall with it and the reality will force you to look at the code. It won't be nice, but until that point delulu land is sustainable enough to fall forward
It's the uncanny valley of AI. It's still not quite good enough that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.
If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.
This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.
There is only so much need for taste and judgement, before even that has been incorporated into the models.
Like the fact that software "engineering" is mostly nothing like real engineering (and it’s further regressing now due to LLM coding!), the general lack of drilling is again one of the things that make software-related stuff look really naive and amateurish from the perspective of those dealing with the real world. Imagine if the military, police, fire service, and so on did not drill and rehearse incident response?
> If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator.
I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.
Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)
Out of context, but to address "AI will replace engineers".
Recently discussed something about economy/investing with a friend while at work at a slaughterhouse. I really didn't want him to get scammed buying crypto.
So, used ChatGPT to find some sources in Somali, a 3 videos with short description why it's worth watching. Intro into investing, intro about cryptocurrencies and about buying them. Had the text shortened down to 3 pretty short paragraphs, not more than twice this post.
He's a smart guy, but only went to primary Qur'an school. Doesn't read or such, mostly consumes internet in form of video/media. He couldn't read those 3 paragraphs, it was too long. Or rather, it wasn't just 3 paragraphs, it was a lot to read.
Maybe we're already dividing into murlocs and the surface dwellers?
This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.
I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past.
Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago.
This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.
The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started).
Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.
I've been thinking about this lately - is it like using 3rd party libs to achieve stuff faster? As much as I would lovr to hand craft the datetime logic in my app, I might as well use luxon and invest this time somewhere else. Only now with llms, you get virtually infinite 3rd party libs you can use, you create them on the fly. So if you have strong engineering values, I would say simply it boils down to "contracts over programs", you can still be in touch with the logic that glues it all together and treat some logic as a blackbox the same way we do with 3rd party libs?
It's not because with libraries you have a boundary somewhere and can decide to not care what's inside as long as the interface is stable and well designed. The problem of course you need to prioritise building well designed interfaces and decouple components from each other, and that's a skill most developers aren't good at.
I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system.
They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI.
You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.
The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not.
An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.
When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust.
One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time.
It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...
Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong.
Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.
If an RPA team ever gets direct access to a production database in your company, look for a new job.
> Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces.
For anyone else wondering what RPA means. Never heard that abbreviation before.
Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?
Coding via LLM is not similar to using an abstraction. Imagine a car. The controls like steering wheel, the pedals, the gear levers. Those are abstractions.
But using LLMs are like driving using a remote control that has probabilistic behavior. You just loss what it feels to be in a car and you fail to improve as a driver because of the erratic remote control.
Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me.
Depending on what your goals as the author are, you may or may not want that.
Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.
You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.
Prediction: this won’t happen. The abstraction will be good enough and people will need to know only as much as they need to know- things will stabilise at the equilibrium.
You rather generously assume engineers are in touch with their systems.
Even before layoffs many teams just maintained things org has long lost coherent knowledge of
After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn
Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.
Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM.
Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short.
There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.
I am exceedingly tired of poor metaphors that are popping up since AI has taken over writing.
No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.
The former CTO of a large MSP software company once told me on a call the reason their product had so many features removed with price increases was "you can't maintain a plane while it's in the air"
The immediate response was "We don't, your updates bring the on prem RMM down for hours at a time, the plane is grounded for maintenence regularly"
A natural evolution of engineers losing touch with the customers and users.
I'm noticing some of the concern play out regarding AI weakening the capabilities of software people.
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it after 3 days slamming it into Claude. The resolution is literally 1 line of code that could be arrived at in about 30 minutes of patient, old school troubleshooting.
I think what's happening is the AI system draws poorly aligned and led engineers into this ego inflation feedback loop where they are completely detached from reality because these tools can simulate a better one.
I'm seeing this happen in the security space right now. Someone on my team I was helping train and bring along is all of sudden regressing in their understanding of the issues we're working on, and instead focusing on AI tool outputs to do their job for them.
I can share a weird story:
Usually, I take my time to understand each keyword of the code I'm looking at, especially if it is new to me, like terraform.
I work in a team/with one architect, who only did the DevOps/Infra stuff for the past years and I had the expectation he knows what he is doing and talking about.
At around 2 weeks, I noticed how his knowledge has severe gaps and how he takes things at face value or uses terminology interchangably, which confuses me. It sounds plausible, but it does not actually translate into a working system or shared understanding.
Then one day I did some pair programming with him and whenever there was an error or a resource missing, he would type it into the LLM, copy paste it out of it and then brute force error messages. He did not even wait a second to think or reconcile whats happening on the screen or what the exact requirement is. Never taking one step back and questioning any assumption.
Now that the timeline is shifting and everyone starts to be stressed, he continues to vibe code through me and it is so tiring, there is no higher level planning or architecture, its just a reactive type of trial and error to be faster. It feels like these people are so used to talking to bots, that they treat you like an agent they can chat to or talk through monologs with.
It is quite shocking how people went from being humble (learn the basics or close the gaps in understanding) to full on authority on everything and berating people 24/7...
So right now I'm considering quitting IT for a couple of years until people calm down, but I think its pretty futile
I don't think it's generalizable. The kind of person who copy and pastes from the AI is the kind who did the same from StackOverflow before. It's more compelling, and we probably see more of them because of it, but it's the same general thing.
The kind of person who insists on understanding things and working through the problem has always been rarer. It's not "humble", it's "inquisitive" and "persistent".
I'm seeing people who _used to be_ like that losing that understanding without realizing it's happening - they have a superficial idea of what the code is doing, enough to feel like they understand it, but the change is apparent when watching them handle something unexpected.
Great comment.
At my place, this is what they want. They want people to smash through things as fast as possible. They don’t want people to sit and craft a solution which takes in to account the whole. They are choosing tools which are low code, and use llm’s to produce what they need. as they say “this is the way things are going”.
You captured this phenomenon very well in this comment. Appreciate you sharing it because it’s hard to describe exactly what makes this sort of behavior so bizarre.
>> he continues to vibe code through me
So quit pair programming. I never did, never will do that, nor worked at a place that remotely encouraged that. Each to their own, that's how it should be.
In my mind, this, not copyright or water use, is the best reason to boycott AI. It'll make you incompetent.
At least at my company the OKRs are quite clear and demand heavy AI utilization above all else
I gave the team an exact solution on a silver platter and they still failed to identify how to go about it
I think what's happening is ... poorly aligned and led engineers [in] this ego inflation feedback loop where they are completely detached from reality
A story about a team of humans with some very human problems.
Sincerely , I think you're blaming the AI incorrectly there. You just got incompetents on your payroll.
So how do you build competence in a world where AI is preached to be the most reasonable way to solve problems because it's supposed to be faster than humans?
incompetent people, surely?
It's the uncanny valley of AI. It's still not quite good enough that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
I it usually doesn't get me in this weird state of mind, but I once spent 6 months (all-in) building a thing that I, once finished, just left alone completely (on disk gathering dust). Weird experience. So I'd say AI physchosis is real.
I think LLMs have some of the same risks and benefits of stimulant drugs. They can make you more productive if used effectively as a tool, but they can also delude you into thinking you are better than you are and create a dependence such that you aren't just less productive without the LLM/drug, you fail to be productive at all because you don't know how to function without it.
That sounds somewhat applicable to many tools. Like Vim/Emacs, for example. Or computers and smart phones in general.
I don't think it's engineers, it's the rest of the org insulating the tech workers from every side of the business
I think there are many cases where it was the tech workers themselves who argued for isolation from the customer so that they may focus harder on whatever tasks. I used to be one of these workers. I argued very hard for it. I regret that today.
On the surface it seems rational, but it quickly turns into a system of perverse incentives because now the development team must maintain an illusion that they are constantly overwhelmed with tasks and could never hope to spare a microsecond to assist the customer. This misalignment is how you wind up building your own web frameworks and databases from scratch. It turns into a self serving monster that eventually dominates the entire business. From the perspective of the business, many of these development teams look like they're behind some modern day iron curtain.
If the solution is so simple, why claude did not found it? At this point we can assume, it is better than 90% of engineers (including me).
After three decades of outsourcing to lowest bidder, I do not buy that humans are somehow better!
> patient, old school troubleshooting
I usually see similar arguments around systems with major red flags (no docs, poor CI, decade ago no CVS...). And engineers with private stash of workarounds for job security!
Claude does not do anything special.
Or perhaps claude was misconfigured, it had no access to relevant part of system, and it tryied to work within its limitation. Often it means decompiling binaries in desperate loop...
Claude regular spits out six helper functions instead of... A twenty line for loop. It overengineers most things.
Overabstracting, deduplicating things that don't need to be. Building metaclasses because it saw a single orchestrator in the whole codebase.
If it is a better engineer than you... You need practice.
The funny thing is that if you never understand the codebase then you will keep thinking Claude is doing a great work delivering all this incredible software, when all it has done is created unnecessary tech debt.
Oh and how is it any different than most software engineers?
How many times I heard ORM are bad only to recreate the same shit?
How many times I heard ORM had bad performance and see 1+n stuff everywhere?
How many times I have seen tight coupling in the name of DRY?
> A twenty line for loop. It overengineers most things.
Anecdote I like to tell.. I was working on a financial planning software, intentionally purely vibe coded as an experiment.
I eventually discovered AI had implemented seven duplicate copies of tax calculation functions. All of them different. All of them wrong. All of them giving different answers for same input.
Not even the most junior of newbie junior engineers would do something this crazy. But AI was happy to do it. It will solve the immediate problem, efficiently. Even if the most efficient solution is something ridiculous like this.
I am former java enterprise dev, so yes I often code this way. Unit testing, decomposition... Some projects CI refuse to merge commits with 20 line loop and duplicated code...
But that is not a point. Claude can code tight compact loops, it just needs to be instructed to do so! If it does "enterprise code", it means it had no instructions about code style.
If your documentation, spec, agent.md does not have proper guidance on coding style... yet another red flag!
100%
It is my pet peeve with Claude and why I don't prefer it for most stuff
(also the comment spam - but that's a all of them in a way or another)
So after 30 years of outsourcing to the bottom 10%, you think Claude is better than the bottom 90% even though it’s so stupid that it doesn’t even know it should ask for advice or more information when it’s stuck?
It just follows instructions you give it.
Some asian devs will go for weeks without asking for help, all while giving amazing fake status reports. Loosing face etc...
Do you think those devs are in the top 10% of all devs like you said Claude is? Or is the bar suddenly much lower after all?
You're comparing scammers to incompetence
Because simplicity is hard and often the result of careful thought. Anybody can keep piling pile of shit on top of pile of shit which is why that sort of code is so common in our industry.
While I actually agree with you (though, outsourcing to lowest bidder would account for much of what you're seeing with humans), I just saw Bug Hunt Bench scores that gave me some pause:
https://x.com/PawelHuryn/status/2095982259761475945
https://bughunt.productcompass.pm/?preset=all
Claude Opus 4.8 ranks near last on this Bug Hunt benchmark, and missed 96% of the deliberately introduced bugs. If you're a developer who has been falling back to Opus 4.8 because of how Opus 5 talks, and Fable 5 being so expensive that it needs to be rationed... well, turns out Opus 4.8 can actually be quite poor for finding bugs.
(Which feels weird to me, because Opus 4.6 fixed a bug that myself and a group of humans had been hunting down for over a decade. Models are spiky.)
Also surprising to me: Luna Max performing better than Fable 5.1 High, at least on this benchmark. But Astra 6 & Fable 5.1 on Max both perform at the top as you would expect.
Still, basic debuging and trouble shooting is where LLM generally shine. Any model can bisect git history and isolate newly introduced bug.
If model can not automatically reproduce bug, while human manually can... you got a problem in CI.
> Luna Max performing better than Fable 5.1 High
Perhaps you are reading too many benchmarks.
> At this point we can assume, it is better than 90% of engineers (including me).
Hard disagree. We absolutely cannot assume that. You can posit it, and we can have an informed debate about it. This is what irks me the most about LLM fans: they constantly try to reframe the debate to have their worldview as the agreed-upon starting point.
That final 10% is the hard part. 90% is easy.
I find the use of AI like quicksand.
The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.
No worries, at some point your hit the wall with it and the reality will force you to look at the code. It won't be nice, but until that point delulu land is sustainable enough to fall forward
It's the uncanny valley of AI. It's still not quite good enough that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.
If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.
If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.
This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.
There is only so much need for taste and judgement, before even that has been incorporated into the models.
Like the fact that software "engineering" is mostly nothing like real engineering (and it’s further regressing now due to LLM coding!), the general lack of drilling is again one of the things that make software-related stuff look really naive and amateurish from the perspective of those dealing with the real world. Imagine if the military, police, fire service, and so on did not drill and rehearse incident response?
Netflix's chaos monkey was this, in a way.
> If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator.
I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.
Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)
Out of context, but to address "AI will replace engineers".
Recently discussed something about economy/investing with a friend while at work at a slaughterhouse. I really didn't want him to get scammed buying crypto. So, used ChatGPT to find some sources in Somali, a 3 videos with short description why it's worth watching. Intro into investing, intro about cryptocurrencies and about buying them. Had the text shortened down to 3 pretty short paragraphs, not more than twice this post.
He's a smart guy, but only went to primary Qur'an school. Doesn't read or such, mostly consumes internet in form of video/media. He couldn't read those 3 paragraphs, it was too long. Or rather, it wasn't just 3 paragraphs, it was a lot to read.
Maybe we're already dividing into murlocs and the surface dwellers?
This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.
For more in this vein, look up "Automation should be like Iron Man, not like Ultron". Sad to see so many people let go of their agency.
without Iron Man's plot armor, Ultron would have won
Code too.
I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past.
Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago.
This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.
Will AI remember ? Or rather how will we make AI remember ?
The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started).
Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.
I've been thinking about this lately - is it like using 3rd party libs to achieve stuff faster? As much as I would lovr to hand craft the datetime logic in my app, I might as well use luxon and invest this time somewhere else. Only now with llms, you get virtually infinite 3rd party libs you can use, you create them on the fly. So if you have strong engineering values, I would say simply it boils down to "contracts over programs", you can still be in touch with the logic that glues it all together and treat some logic as a blackbox the same way we do with 3rd party libs?
It's not because with libraries you have a boundary somewhere and can decide to not care what's inside as long as the interface is stable and well designed. The problem of course you need to prioritise building well designed interfaces and decouple components from each other, and that's a skill most developers aren't good at.
I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system.
They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI.
You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.
The article does not get the point. What aircraft companies did was separate training and work, and SRE/IT typically does not.
An AI can handle routine incidents and then present learning cases from that routine work for training, because the skill in SRE is not the mechanical log grepping, grafana dashboard browsing etc but forming the hypothesis. AI incident reports can create training cases that are a much better training for hypothesis forming and testing than the work itself can.
When someone else -whether AI agent or a human- solves the recurring minor problems for you, those problems become non-issue, get swept under the rug, just to accumulate more dust.
One day, those may become bigger as they are forgotten, causing havoc. The standard root-cause-analysis depending on systems having certain retention period, which may be expired at that time.
It is important to get real hold of one's systems from end-to-end aspect, which holds true for both AI and human operators...
Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong.
Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.
If an RPA team ever gets direct access to a production database in your company, look for a new job.
> Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces.
For anyone else wondering what RPA means. Never heard that abbreviation before.
That makes more sense than Rocket Propelled Automation that was my hallucination.
Isn't there anywhere to "go" from here? In the last decades, introducing new high level abstractions on top of existing paradigms naturally had everyone move up the ladder and work at the next higher level, why should this be different these days? Do we think AI will reach the top of the abstraction ceiling, so there's no where to go from here?
>introducing new high level abstractions...
Coding via LLM is not similar to using an abstraction. Imagine a car. The controls like steering wheel, the pedals, the gear levers. Those are abstractions.
But using LLMs are like driving using a remote control that has probabilistic behavior. You just loss what it feels to be in a car and you fail to improve as a driver because of the erratic remote control.
That’s the big question for sure
Meta: The blinking cursor of the "logo" of the blog being sticky in the top left corner makes it impossible for me to read the text. It constantly fires interrupts at me.
Depending on what your goals as the author are, you may or may not want that.
Being able to scroll it out of view might be enough to achieve the aesthetics goal, and the goal of people actually listening to you.
If you have something like ublock origin use the block element functionality to target the blinking cursor. It's gone on my end :)
Not just you. Must be nice being fully neurotypical and „not seeing“ all of this kind of stuff.
I mean seeing this stuff pays the bills and does so quite well, so..
We have been running Agents on infrastructure and letting to create resources, scale up and down, security scans etc.
I agree with premise of thr blog. The question i have been asking internal does knowing your system really matter if you can recreate it in minutes.
We recently had a situation, where in with our internal platform and claude we recreated everything in minutes.
Management in the end cares about the outcome and not how the meat is made.
You’d hope these AI incident responders have very constrained production tools to fix things. You’d hope the humans remain familiar with those tools and they are incredibly well documented.
Prediction: this won’t happen. The abstraction will be good enough and people will need to know only as much as they need to know- things will stabilise at the equilibrium.
You rather generously assume engineers are in touch with their systems.
Even before layoffs many teams just maintained things org has long lost coherent knowledge of
After layoffs and typical org knowledge churn - you can either rewrite it (but how? Product team responsible for original implement requirements is long gone too) or recoup (reverse document) some of that lost knowledge with AI and actually learn
Ironies of Automation is front and center in this article which is awesome. Many of the conversations on AI automation are describing or rediscovering the insights the paper covered.
Good article and I like the callouts to the aviation industry. For me what's missing is the author should also have touched on CRM and SRM.
Also, that paper "The Ironies of Automation" is one that everyone should read. It's fairly short.
There is a related problem in terms of these situations where the computer system is handing off to the human. It's called "the bumpy transfer of control." Very fascinating concept.
I am exceedingly tired of poor metaphors that are popping up since AI has taken over writing.
No, operating software is not like operating a plane. Not at all in fact. The people operating the software and resolving incidents are the same people who created the software in the first place, and continue to work on it day to day. Pilots have not and don't.
The former CTO of a large MSP software company once told me on a call the reason their product had so many features removed with price increases was "you can't maintain a plane while it's in the air"
The immediate response was "We don't, your updates bring the on prem RMM down for hours at a time, the plane is grounded for maintenence regularly"
That‘s the goal of the AI tech bro world: get us dependant on their tech, ruin our native/raw skills, lock us into their proprietary skills.
Reminds me of the move from on prem to cloud. Linux sysadmins were killed and replaced by aws focused devops.
"THIS IS NOT A DRILL"
It's just a picture of a drill! Thanks, Magritte.