Snake oil may be a bit strong, because snake oil never works (except maybe as placebo?) whereas anything with an LLM, even though stochastic, has a pretty high chance of working.

> ... you also realize that promised productivity gains are also snake oil because reading code and building a mental model is way harder than having a mental model and writing it into code.

Not really, though it depends on the code; reading code is a skill that gets easier with practice, like any other. This is common any time you're ever in a situation where you're reading much more code than writing it (e.g. any time you have to work with a large, sprawling codebase that has existed long before you touched it.)

What makes it even easier, though, is if you're armed with an existing mental model of the code, either gleaned through documentation, or past experience with the code, or poking your colleagues.

And you can do this with agents too! I usually already have a good mental model of the code before I prompt the AI. It requires decomposing the tasks a bit carefully, but because I have a good idea of what the code should look like, reviewing the generated code is a breeze. It's like reading a book I've read before. Or, much more rarely, there's something wrong and it jumps out at me right away, so I catch most issues early. Either way the speed up is significant.

Humans also drop any hard requirements you specify regularly, and similarly require review. Nevertheless we manage to increase reliability of human output through processes and reviews, and most of the methods we use for harnesses are taken from experience with how to reduce reliability issues in humans, who are notoriously difficult to ensure delivers reliably.

Everything you say is all possible, and in theory I agree with you.

However, I have been using spec-kit (which is basically this style of AI usage) for the last few months and it has been AMAZING in practice. I am building really great things and have not run into any of the issues you are talking about as hypotheticals. Could they eventually happen? Sure, maybe. I am still cautious.

But at some point once you have personally used it in practice for long enough, I can't just dismiss it as snake oil. I have been a computer programmer for over 30 years, and I feel like I have a good read on what works and what doesn't in practice.

I hope the only reason people are pretending these markdown suggestions are a "workflow" is fear that a more structured approach will be obsolete by the time it's polished. I can't imagine the pace of innovation with the underlying models will stay like this forever.

I hope to see harnesses that will demand instead of ask. Kill an agent that was asked to be in plan mode but did not play the prescribed planning game. Even if it's not perfect, it'd have to better than the current regime when combined with a human in the loop.

Don't let the perfect be the enemy of the good. Of course we know the AGENTS.md and skills aren't 100% effective. But no, it doesn't mean that they're 0% effective.

All this said, I quite like the mental model of documenting a simple process, and I suspect our future ai overlords will find it useful that I have a series of md files that outline my preferences and processes for certain tasks.

I am not however going to share any of this with work colleagues and make myself redundant.

I can see why this would seem to be “snake oil” logically. However, this approach does work in reality. Your comment just shows that you seem inexperienced with using generative AI.

I can understand skepticism to a degree, and even fundamentally believing that AI is bad for all sorts of reasons, but I am becoming more and more perplexed at the certainty behind statements like this one. How are you so certain that AI development is this doomed? It just hasn't matched my experience at all, and I wonder what your experience is that has driven you to this level of certainty about the certain doom of AI coding?

Is it just a philosophical belief that AI is morally bad? Or have you actually used AI to build things and feel confident that you have explored the space enough to come to such a strong conclusion?

I have been writing code every day for over 30 years, and have been doing it professionally for over 20. I have seen fads come and go, and I have seen real developments that have changed the way I do what I do numerous times. The more experience and the more projects I create with AI, the more certain I am that this is a lasting and fundamental change to how we produce software, and how we use computers generally. I have seen AI get better, and I have seen myself get more proficient at using it to get real work done, work that has already been tested with real world, production, workloads.

You can hate that it is happening, and hate the way working with AI feels, but that doesn't mean it is not providing real value for people and doing real work.

I’m a bit curious with these takes. Arguing in good faith - is the general assumption that people who use AI/agents/harnesses don’t ship features? We’ve been all in Claude Code since ~Septemberish, and have been able to successfully track the boost. Like the features that we ship that get used in production. Both from infrastructure side, and business logic implementations. Frontend and backend.

I don’t think people are wasting too much time. Although, I do agree most of these posts are just bs, including this one. But AI-development has been a thing across a lot of companies in the world.

Right, just like all the productivity lost when people stopped using paper ledgers to mess around with these so-called 'databases'

I work on projects where we measure the output. There's nothing "pseudo" about it.

i treat it like Minecraft automation - it's just for funsies and to pass the time haha

I don't think agentic workflows are there yet, but implementing skills to manually call and use while working side by side with an AI is definitely nice - our company is focused a lot on sandboxing right now and having safe skills

I don't think we've gotten feature development well yet, but the review skills + grafana skills they wrote have been pretty solid

I'm sure lots of people felt this way about steam power too.

Trick is to not burn too much time worrying about the perfect skills and this and that. See a lot of people filling skills with LLM junk, or overdoing rules that start confusing the LLM. Just try Vanilla, see something you don't like? Then you make a skill and funnel the LLM to use it for the style of task it's working on. E.g. database work is a mixed bag with LLMs, they tend to do work in totally different styles if you leave them unconstrained.

Agents are unbelievably useful at helping takeover and refactor messy codebases though. I just started taking over this monstrous nightmare of a codebase, truly ancient code the bulk of it written over 10+ years ago in PHP. With the use of Claude / Codex I was able to port over the vast majority of the existing legacy storefront and laid the groundwork for centralizing the 10-20k LOC mega-controller logic over to reusable repo/service patterns.

Just shit that would've taking years previously, is achievable in under a month.

I couldn't agree more, just because I know I already wasted months and pulled the plug :D

This will be another Microservices moment in our industry.

You haven’t made money from their use yet?

They will lie to themselves and deny it.

You’ll get downvoted for this hearsay!

Skills are often invoked imperatively by the user. In cases where they are intended to be used directly by the LLM, it would be included somewhere else in the context. E.g:

``` After implementing the feature, read the testing skill for instructions on how to test. ```

Well, to be fair, in e.g. Codex you can invoke a skill directly, with $my-skill, and this WILL lead to the skill being injected into the context. At that point, the LLM follows the skill as well as it follows any other part of the prompt, instructions, or context.

Same for MCPs and system instructions, there are a lot of people that just install everything without understanding it, cluttering their context, wasting >50k tokens for these tools they don't need and then complain that they need to pay >100$ per month because they reach their limits too fast.

I would love to know how many people are actually using superpowers.

I showed up on the agentic dev scene prior to superpowers, and I am getting concerned that >50% of my self-rolled processes are now covered by superpowers.

I no longer trust gh stars, can anyone chime in? Is superpowers now truly adopted?

If it is truly valuable, why hasn't Boris integrated the concepts yet?

This is like creating a React framework called ReactJS to compete with NextJS

Looks like a bunch of canned skills served through a plugin?

Does superpowers actually work? The main skill file doesn't inspire much confidence:

    "If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill."

Because we've been automating large parts of our former jobs for decades. Otherwise we'd all be trying to build things in the least efficient way possible to maximize how long the job takes, which IMO isn't a great idea.

Humans have been minimizing how much work is needed to get a certain level of output for as long as we can track. It's civilization. Should we go back to farming by hand with hoes, to maximize labor used? Go back to streetlights that are individually lit? The society that falls behind on automation becomes poorer, and eventually just dies, as even the people born there tend to choose to leave to higher productivity places. It happened to eastern europe, it happens to the Amish. To any poor society which gets emigration. Doing more with less has always been exciting.

It's likely the people that were not good developers that suddenly got accelerated "to the top" that seem the most for it. All of the good devs I know have been a bit more cautious on the uptake.

Because usually the people who lose their jobs are people who do not adapt to the market.

Right now it's not clear in which direction everything is involving and that's why people experiment with handing all their data to random agents, figuring out how to store and access context, re-use prompts and other attempts to harness this tech. Most of these will maybe be useless in a year as they might be deeply integrated into the next wave of models but staying on top of the development has always been part of the fun of working in this field.

I don't understand this thinking as a computer programmer. My whole life has been about getting a computer to do work so humans don't have to anymore. Every single piece of software written is supposed to take away work from someone.

Do you feel this way about every automation you create? I do know some old school sys admins who felt this way about a lot of infrastructure automation advancements, and didn't like that we were creating scripts and systems to do the work that used to be done by hand. My team created an automated patching system at a job that would automatically run patching across our 30,000 servers, taking systems in and out of production autonomously, allowing the entire process to be hands free. We used to have a team whose full time job was running that process manually. Did we take their jobs by automating it?

Sure, in a sense. But there was other work that needed to be done, and now they could do it.

The whole reason I like programming and computers and technology is precisely because it does things for us so we don't have to do it. My utopia is robots doing all the hard work so humans can do whatever we want. AI is bringing us one step closer to that, and I would rather focus on trying to figure out how we can make sure the whole world can benefit from robots taking our jobs (and not just the rich owners), rather than focus on trying to make sure we leave enough work for humans to stay busy doing shit they don't actually want to do.

Some people are playing the global optimization game; a world where anyone can have any (production grade) software they want.

people are now being encouraged to use ai notetaking features under the guise of productivity.

a worker is just the sum total of all work related context. to collate, verify and organize this context is just asking to be replaced.

Month 30 of software engineers not existing in 6 months

The reason they are long is because these skills are produced mostly by Claude Code and Opus and no sensible human will read these files, let alone build a mental model around them. There is just layers of assumptions that this works - when in reality it doesn't and it is wasteful.

Here is a fun experiment.

Ask any LLM to write something vaguely familiar. For example, ask it "write a fib". Since almost all LLMs are fine tuned on code, I find that all of them will respond with a fibonacci sequence algorithm even-though to a non-programmer "write a fib" means to write an unimportant lie.

So there is compression. You can express an outcome in just 3 vague tokens without going into details what exactly is a fibonacci sequence.

That should be enough to understand that the length of the prompt does not matter. What matters is the right words, frequency and order. You can write two page prompt or two sentence prompt and both can have the same outcome.

I quickly skimmed and it looks like at least a few of them are intended to be more like system prompts for a tightly scoped sub agent than a skill as such. I agree, I wouldn't want to use a lot of of these in a longer-running work session.

I have been successful with short and focused skills so far. I treat them as a reusable snippet of context, but small ones. For example a couple of paragraphs at most about how to use Python in my project and how to run unit tests. I also have several short "info" skills that don't actually provide the agent instructions, they merely contain useful contextual information that the agent can choose to pull in if needed.

Even having too many skills can be an issue because the list of skill names and their descriptions all end up in the context at some point.

I have written zero skills, so not sure how normal it is. I counted the words in couple of them and they seem to be around 2k range. So 5 skills would be around 10K. Even at a small LLM context of 128k, that's still around 10%. And for a 1M context window like the big ones, it barely registers.

> it would only take a couple of these to really fill the context alot.

Only skill front-matter (name, description, triggers etc) are loaded within context by default, so this isn't likely to happen without 1000s of skills.

I reviewed the line counts of my own project skill files, and the top 3 I have are:

    805 lines
    660 lines
    511 lines

Similarly, the agentic coding success stories are from orgs that had all of these things out of the gate.

All of these articles about setting up the perfect agent environments with skills, plugins, MCP servers, markdown files, etc. etc. reminds me so much of the culture around setting up the perfect "productivity stack". You need the perfect note-tacking app, ticketing app, calendar integrations, yada yada before you can really do anything meaningful. The reality is that you're going to get beat by someone with a few things written down on a piece of paper who is just getting stuff done.

The problem is it’s so rarely A/B tested, definitely not at scale. An engineer, who writes all these my-workflow-but-for-agents skills, proceeds to get the good outcome, while also seeing affirmations that the agent did follow the prescribed processes - that is considered a victory. In reality the outcome could’ve been just as good if they fed Claude a spec + acceptance criteria, or even a basic prompt for the simpler tasks.

This is how similarly we collectively approach Taylorism, isn't it? However, the world favors capitalism, of which Taylorism becomes a handy scaffolding.

> The best way to prompt an LLM is to describe the outcome you want, that's it. They are trained as task completers. A clear outcome is way better than a process.

This is not true for anything complex. They’re instruction followers, of which task completion is just one facet.

They’re also extremely eager to complete tasks without enough information, and do it wrongly. In the case of just describing task completion, despite your best efforts, there are always some oversights or things you didn’t even realize were underspecified.

So it helps a lot to add some process around it, eg “look up relevant project conventions and information. think through how to complete the task. ask me clarifying questions to resolve ambiguities. blah blah”. This type of prompt will also help with the new Opus 4.7 adaptive thinking to ensure it thinks through the task properly.

If there is anything we have learned in decades of Software engineering, it's "A clear outcome" is not easy to describe. In many cases, it's impossible unless people from 4 different domains collaborate. That's why process matters. It allows for software to be built is a "semi-standardized" way that can allow iterations to get us closed towards the expected outcome, that might emerge over time.

Yes, not everything I use LLMs for going to have the same level of ambiguity or complex requirements. Optimizing by choosing to skip over parts of the process is exactly Addy is talking in this article.

This seems like common sense but it does not work in practice.

Prompting is just the first part. To get the outcome, you need to have other systems to steer the agent as it get things wrong. Proper deterministic tests work. But there is also stuff that need to happen during the LLM execution like cyclic detection etc. All of this adds up.

You cannot just prompt an LLM an hope for a good outcome. It might work in small isolated scenarios but it just does not work consistently enough to call it reliable.

Without further guardrails enforced by the process or the harness, LLMs do not have sufficient capabilities to complete a task up to a certain standard.

I agree that many skills are overblown and unnecessary. But there's a lot of value in giving AI the right process. See how much more effective Claude can be for moderate or large changes when using the superpowers skill.

Sometimes people don't know what they want.

I prefer the start small and iterate approach to arrive at a result.

Then I ask it to summarize. Sometimes after that I ask it to generalize.

a skill is just reusuable/shareable context. It's just text, really. It's useful for things like documentation on how to use an API (this works better than MCP in my opinion), or a non consensus way of doing something. For example, you can use remotion to generate video. There are useful remotion skills that allow you to reliably generate specific types of videos. Captions of a certain style, for example.

That seems a bit reductive. Even with humans, there’s a range of interpretations and ways that something can be built or a task completed. Engineers remember stuff so you don’t have to keep repeating yourself. Skills are a way to describe your outcome without similar repetition.

A plugin is just a set of files, right? why wouldn't you be able to delete it later?

I wish this fucking meme of "post the prompt" would die. Very little work is one-shotted, very little has a singular "the prompt", most is iterated until it's close to the vision of what the author actually set out to write.

Wait, shit, are people using LLMs to name things now? I'm definitely out of a job then!