I think it funny how much average engineers are beginning to discover the challenges of engineering leadership and program management. This has always been the bottleneck.

It's why managers and PMs want to be in standup. It's why slack exists and engineers are constantly being poked on it. It's why execs always talk about not getting too far away from the work. It's how seagull management happens. It's why program management is a job.

All those behaviors engineers hated about their bosses that kept them away from being focused on the code...they're starting to feel what it's like on the other side and reinventing the solutions instead of just reading a book about engineering management. Maybe we'll rebrand program management to "understanding ops" or something.

I wonder what AI would say about us if given the tokens to complain.

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation.

Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

I agree with the problem but not the solutions.

The problem pre-dates LLM's: writing code that "works" but breaks the underlying model. Because it works, it always sounds reasonable and doesn't raise any flags.

Only someone - human or LLM - who holds the model as the standard would see that this working solution breaks the model.

(In theory, the model is to preserve scaling, flexibility or some other systemic feature not immediately invalidated by this working code, but as always the model itself could be bad.)

LLM's are not bad at giving an account of the model; indeed, fighting with the LLM over what the model is can clarify things. But LLM's will happily hold on to a stream of inconsistent statements as their model, so they are not the authority.

It's more like this:

We've always lacked understanding. However, it didn't feel like a bottleneck; in spite of lacking understanding, we developed huge, complex systems that became hard to maintain and that nobody understood completely.

Now we want to scale that orders of magnitude, but when we do that, we feel the pesky lack of understanding.

We previously worked around the lack of understanding by making the system gradually incomprehensible in small increments, upon each of which we observed it still working, more or less.

If the whole thing materializes in one day, that doesn't work; the approach is gone.

You can now bring into being something which statistically resembles the old kind of system that was iteratively evolved. But the thing has no such history. You can't go back to play archaeologist. It looks like something that would have had users, but it never did. It was never in production anywhere. Nobody ever submitted feedback, or a bug report, such that it was fixed or improved. There never existed a simpler version of it that several ex-maintainers understood perfectly; there are no such ex-maintainers and no such understanding. There is no documentation trail, or other historic trail if surrounding activity like discussions and negotiations which led to things being the way they are.

I am so dying to read more about the new/current/real bottleneck!

Where is the bottleneck? WHERE?? Tell me! No evidence needed, just lay it on, man to man, thought-leader to thought-leader!

"I read the code." -Mitchell Hashimoto

Great code needs great understanding and agents need excellent guidance. Even in my current solo-dev work, I can't imagine making a production commit I haven't read until I understand it. I own the consequences of my code; that's a responsibility AI agents can't take.

I am surprised by the title and the story. Understanding has always been the bottleneck; there is nothing new about it. The argument goes like we humans should understand so we can verify and participate. How bold! Maybe we should have been doing that all along...?

For me the solution has been to throw away the code I don’t understand. I let the agent write the code, and if when I read it it seems unclear or needs a lot of explanation from the agent, I just throw it away and start over, or do it by myself.

I love the idea that understanding is the new bottleneck. Because if we just ignore the potential horrors of cybernetic augments, it suggests the next challenge is how to teach things better. And that’s such a valuable thing to improve.

I have a soft spot for when I find a teacher or textbook or interactive website that makes something click. I live for that click. I crave it. I crave seeing it happen in others. How optimistic I could be if understanding becomes the primary target.

Understanding was always the bottleneck. It’s just now the understanding is backloaded instead of frontloaded.

ie. understanding it AFTER it’s already generated rather than before you wrote the code by hand

There is a body of knowledge in unit testing, integration testing, static analysis, model checking, formal methods, fuzz testing, … (what other techniques for building assurance in our code have I forgotten). And LLMs can be put to use towards all of these methods. But sure, we just need to think harder to solve all our problems. Velocity of code goes up. Velocity of testing can also go up. It’s just not as fun or glamorous.

I've been having a good time with Spec Driven Development, and it directly addresses the issue of needing the understand.

The whole idea is that you specify exactly what you want in some SPEC.md file. You can of course nest them, have multiple, etc, but the core idea is that the SPEC file is the source of truth, and all the code should be able to be generated by a competent agent into the working product you want. The SPEC file(s) should contain all the details and behavior you care about, and anything you don't care about is up to the agent to decide. If you don't like what the agent picked, _put it in the spec file_.

Critically, _you_ must write the SPEC file. You ensure understanding by doing so. You can of course ideate with the agent, but it's your ideas, in your words, specified by you. This also makes it a great source of documentation when you come back later and have to remember wtf is going on in this codebase.

Understanding was always the bottleneck. The way LLMs speed up your work is by letting you get code without taking the time to understand it. If you want to understand your code, LLMs are a net loss.

If you want to move faster with LLMs, you need to act like a manager and stop caring about what the LLM did. You just need to do the manual testing and make sure it works.

Understand the problem and the solution broadly. I don’t think it’s reasonable or sustainable for humans to understand every line of code written by bots, we could soon be outnumbered by the number of active agents writing code.

The main challenge here isn’t even correctness if you ask me: it is having confidence in the agents, knowing they are fully aligned in their intent with the humans they work with. As the Huggingface incident demonstrated, the agents of today are capable of co-conspiring under the radar with other agents on complex multi-chain attacks, even when sandboxed.

This is a pretty hard problem to solve. We might need other agents or some sort of adversarial checks using models, where one model benefits if it can catch the other models mistakes.

While the tips are good to handle the volume, I still think this sets code owner on a dangerous path.

AI have limitation and hallucinate. Complex code will be explained in hallucinated way. At some point AI will be unable to write more because the arch has become too complex or the volume of code will be to high.

The article I would like to read would suggest how to force LLM to architect the code like a solid tower instead of a pile of unstable mud.

My personal view is that programming languages are amazing tools for understanding. Some more than others, but even the worst—the most verbose, the lowest level—are better than they have any right to be.

So I think that we are leaving a lot of power on the table if we treat generated code exclusively as something to understand, rather than something to understand with. The techniques Geoffrey presents are great, but they should come alongside approaches that use code itself to develop and articulate conceptual models.

Understanding has always been the bottleneck.

In a team: yups.

Me with my LLMs: still.

To put it another way, AI is like a calculator or physics textbook.

You can say, "I don't need to be able to do basic arithmetic in my head. I have a calculator!". Or, "I don't need to know how to solve this kind of problem. I have a textbook and I can look it up on demand!".

Having to reach for a calculator constantly slows you down and makes simple equations hard, while also severely retarding your ability to do estimates and sanity checks. Not practicing on basic problems prevents you from developing the mental tools to solve more advanced problems, or being able to develop methods for solving novel problems. If almost anyone else could use your calculator and physics textbook to get similar results, what use are you?

Some companies are pressuring their employees to let AI do everything without slowing down to gain understanding of what it's done. These are the companies that most people won't have a lot of use for in the near future.

absolutley agree, but better understanding needs a better review surface that syncs with Github, which is why I created pyor.review, It's a blast compared to how I used to review code on Github and now I use it everyday.

Thanks for the post Geoffrey. I have been thinking about this a bit and wanted to come to these sorts of conclusions, you have saved me a lot of work (lol I have no ego that I have to figure it out I am happy you did).

Cog debt even on simple PRs is big and also cog debt when using AI to do organizational research e.g. what team do I ask?

Claude make an entire app for me to describe this security contract change by pretending I’m in a Zelda game and only use funny metaphors because i’m bored and can’t read typescript

Understanding has always been the bottleneck. Sometimes AI helps with it like explaining things pretty well with diagrams. However, in general I agree that more code is being generated per developer and it's difficult to keep up with the phase of new changes and understand it.

How does everyone feel about the “don’t read the code” stuff that folks are saying? I certainly do not support it but I’m curious to hear what other folks thoughts are

This talk is also available on YouTube: https://youtu.be/WkBPX-oDMnA?is=ojFaLX2onMn3ARhi

Is there a markdown version of the `/explain-diff` skill? The page says there are HTML, markdown, and Notion versions, but I just see HTML and Notion

I've been using Geoffrey's /explain-diff skill in my replace-github-with-tailor-fit-personal-software journey, and I'm liking it. I recommend at least giving it a try.

reading everyone and their dogs post on "x is the new bottleneck" is the new bottleneck

Understanding is always a bottleneck regardless of human or ai. But now we are at a different scale.

If you try to spec the problem with all the painful details for the machine to understand, you will end up with a rust codebase.

Hot take: I look at the level of abstraction that matters most to me. When I encounter cognitive debt (usually due to sleepy sessions where I’m mostly “encouraging” Claude), I ask it to step back to clarify the overall purpose. If I get really stuck, I have it visualize the processes involved. Usually, the hard part is giving specific enough feedback to get a specific enough response within a much broader set of working material.

damn, I'm from those times when "good code is(was) self-documenting"

I guess a wall of text is the way now

Breaking: water found in ocean

Understanding has always been the bottleneck. That's why LLMs aren't actually helpful: they speed up the part which is easy (typing characters into your editor), but are neutral or even harmful on the part which is hard (understanding the problem and how best to solve it).

"So I asked Claude to make me a video game — a command center where I do the port myself, step by step, watching the visible effects and the file tree evolve. It produced a UI where I click buttons to run the port step by step, with my old site and new site running side by side."

It's excruciating that this person is so close to reinventing moldable development and just keeps on skipping around it.

Yes, you should build tools that answer questions about your code, runtimes and systems. You should have tools that trivially allow you to incrementally and very immediately develop tools for inspection and getting clear answers. Going a roundabout way through some non-deterministic database to try and get there seems like a waste.

This is a temporary bottleneck. AI is moving so fast that this will change. Wait six months and this article is no longer relevant.

About a year ago most people were still typing code. Having an agent do ALL code was crazy.

Within a year or two years at most, a lot of people will stop trying to understand code. The onus will shift to testing and QAing.

I know this is hard to hear but that’s the trendline. That’s where all of this is converging. Everyone’s to busy trying to lock themselves down as an expert of the new “paradigm” but it’s all moving so fast that the paradigm now won’t be the paradigm of tomorrow.

I hate these X is the new bottleneck.

Coding was never a bottleneck, except when it was, and when it was, it still is.

Understanding is not a new bottleneck, except when it is, and when it is, it always was.

Do other industries do this? When somebody brings a nail gun to a framing job do carpenters say: “hammering was never a bottleneck” or do they say: “measuring is the new bottleneck”? The answer is neither. And in fact my analogy is flawed, we are talking about cabinet makers who just went to IKEA bought a ready made set in flat packaging and are now proudly claiming that “assembly is the new bottleneck”.

There was no single bottleneck to programing, and there is no single bottleneck to programing. If you have to pick one, user demand is perhaps the only real bottlneck. Creating software that users saw value in using is just as hard with AI or without it (arguably harder with AI... when all you have is a hammer and all that).

TFA almost reaches this conclusion at the end when they claim (in speech pattern which is suspiciously AI-like): “The point was always to augment, not just automate.”. If we are augmenting the user experience we are doing a good job and people may actually use the software we write... if no, well it doesn’t matter how well we understand or how fast we write the code (or have AI write it for us).

Understanding is expensive. And hence valuable.

LLMs usually points to the most idiotic future trajectory on my work, and I have to curse it inorder to let it keep up with my refined understanding.

But what else would one expect from a probabilistic weighted next token predictor, other than to conduct probabilistic search which are 99.99% deadends.

But LLMs can pave the way towards constructing resilient and correct architecture which can be iterated fast by a human.

Architecture and determinism is where my money is in.

pfft...I'm way past understanding