status_quo69
8 hours ago
No wonder most people at Anthropic believe our jobs are in jeopardy, they've already made a system wherein the machines are treating them like pets.
I really don't get the tendency here from Anthropic to describe what's most likely churn? A high pressure amount of changes/PRs a day on a TUI to wrap a while loop? I'm not going to be a reductionist here so I'm going to cut it off there, I'm positive that the complexity of handling multiple terminal emulators is something that needs some handling (even if doing their own rendering engine was a self made problem). Yes I've vibe coded my own harness from scratch without writing any of it, I still know how it works.
I'm also not sure that they addressed point 2 in any actual capacity? They say "the system verifies itself" which is cool and is something that automated testing has done for decades, but they eschew the mental model of how the system works as a whole and prefer instead to have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
N.B. Changed "1000 PRs" to "A high pressure amount" to be vague since the article is vague here on what CC undergoes.
jonahx
7 hours ago
> have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?
This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...
There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.
The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.
Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
patrickdavey
7 hours ago
"Tbf, it's worth pointing that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".
The real difference as I see it is that in the human maintained old style system, the human would take things slowly and ensure they understand any changes. Claude let loose will just say "yes, I know how to do this" and bork the system, because it doesn't have a concept of humbleness in the face of complexity!
daveguy
7 hours ago
Not only that, when you have a human designed and implemented system you know it's at least coherent enough that someone understood it at some point well enough to take responsibility for it. No such assurance with the llms.
dmd
4 hours ago
I'm guessing you've only ever worked at startups?
Large company codebases are very much not like that. I've worked on systems (e.g., banking) that have accreted still-running code for as much as 60 years. Nobody understands how it works. Nobody ever understood how the whole worked, and many of the parts were never actually intended to work together.
csallen
6 hours ago
But is this really a problem? If you have testing and you have version control via Git, you can roll back changes and investigate them.
I also don't even know if it's really true. Sure, maybe it's not coherent enough that a person understands it, but if an LLM wrote it, you can be ~99% certain (in my experience) that that LLM will understand it, and usually in less than the time it took to write it. And unlike humans, the LLMs don't die and take their knowledge with them, or change companies, or get ornery when you ask them to explain.
Also, in my experience, Claude doesn't really do what the parent is saying and dive head-first into complex problems and bork your code. It is fairly humble in the face of complexity. It constantly recommends breaking down larger problems into smaller ones that are more easy to grok.
But most importantly, anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful. Especially when they're writing code that's important. Unless you're an AI researcher pushing the bounds of what LLMs can do, which let's face it, the vast majority of programmers are not, it's almost always the case that you'll have your LLM writing intelligible code that you can understand and have it explain easily enough.
dieortin
5 hours ago
> can roll back changes and investigate them
Sure, you can do that, but irreparable harm might have been already caused. Sure, that’s unlikely when your product is an LLM wrapper. But in something like a database, it could be catastrophic.
csallen
3 hours ago
I mean it could be, sure. But I think people are generally speaking more careful with higher stakes code than they are with lower stakes code. I know a lot of people who are YOLO'ing it with LLMs, but who ask for more understanding, more reviews, more tests, etc., in trickier areas. Which is only natural. Just like people are more careful when carrying a baby than when carrying a pillow.
(Yes, there will be individual exceptions to this, just like with everything else in life.)
jdlshore
4 hours ago
> anyone who's a decently skilled software engineer and who's using LLMs to write code is going to be thoughtful.
Is this a definitional argument? As in, if you’re not thoughtful, you must not be a decently skilled engineer? Because I see no reason to believe that people are going to be thoughtful about LLM-generated code. Just the opposite. It’s incredibly tempting to LGTM it.
csallen
3 hours ago
> I see no reason to believe that people are going to be thoughtful about LLM-generated code.
I think that's true for low value code. But the more important and crucial and high-stakes code is, the more effort people put into it, generally. Even with LLMs.
pofferfish
3 hours ago
I would love to know what the Venn diagram of people who complain about people not understanding their code, to people who can't describe the boot process of their system is.
SR2Z
6 hours ago
I am not sure that "someone understood it well enough to take responsibility" means what you think it means. I have definitely come across old systems which have had addons bolted on to them in so many places that the overall behavior of the system is unpredictable and only mostly works.
pofferfish
3 hours ago
Someone understanding something and the codebase being coherent are different things. I can only think that you have been lucky with your career placements if you are making that statement.
What you are saying is leaning on 2 premises that seem to not hold water 1. Code written by people is always more readable and maintainable than LLM code 2. Code written by an LLM is always less readable and maintainable
Can you really make those statements in all intellectual honesty?
antihipocrat
6 hours ago
I'd like that to be true, but many software service startups were stitching together node packages and selling the result long before LLMs became useful.
cmrdporcupine
4 hours ago
> you know it's at least coherent enough that someone understood it
30 years working in this industry and I'd say that's the exception rather than the rule. When it comes to large systems.
monkpit
4 hours ago
Yes, [citation needed]
peab
7 hours ago
most large software companies are already too large for anyone to have a complete mental model of.
The idea that someone at say, Google, will be able to fit even half of the system architecture in their heads is not realistic. It also hinges on being able to find and understand the right slice of the system in any situation.
But we have all sorts of guardrails to make that easy - i.e logs, traces, etc
majormajor
34 minutes ago
One of the fun things that you currently need to babysit Claude, etc, on, is that while they've been tuned to hell now to do more testing and even logging and such, they don't currently do enough log-and-debug-oriented specific passes/revisions to distinguish meaningful logging for meaningful debugging from noise that won't necessarily tell you what you need to know.
And you can ask them to address this. And you can ask them to look for that in all your PRs. Etc etc. And that helps a lot.
But it's not something that can be brute-forced in the same way "this failed a test, let me fix it" is. So without a long-term-memory brain holding all the pain of past incidents and knowing and applying ahead-of-time judgement about what to build, it's a challenge to turn the general guidance into the specifics for any unique piece of software.
They're trying to get there of course. We shall see what's next.
("Human code turns into this too" is also a curious rejoinder from the "don't read the code anymore" crowd because there are entire disciplines and areas of study that have emerged and evolved from the problems of large ad-hoc multi-faceted systems being difficult and risky to deal with.)
tripleee
an hour ago
I don't think they're the same. AI code isn't just complex because it's huge.. it's complex because it makes no sense
If you have a large human-written system with a little thought put into it you should be able to find the specific pieces that handle the behavior you want and work on that, and if it's good then changing it should have a small blast radius and be covered with regression tests
In contrast, I've fully vibe coded a 15k line tool for myself with Opus and I have no idea how on earth it's architected after reading through it. It has "tests" but I don't trust them. It's just all over the place with no coherent design to it. It works, but it's making me more and more nervous to change it
jonahx
7 hours ago
Right, that's what I alluded to in my last sentence.
It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.
dasil003
6 hours ago
Having worked continuously this decade on one massively complex consumer facing app that interacts heavily with real-world scenarios, I can say that the problems of cohesion existed well before the agentic revolution. Tons of features built by myopic teams simply because they couldn't understand enough of the system (both online and offline, meat-space interactions) to make good design decisions. AI actually helps here by being able to perform deeper reading and analysis than any human can do. But then the risk is that it can also produce way more random non-cohesive features.
So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.
Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.
dimbletimbers
5 hours ago
> But.... That's not knowledge?
Thank you! This is a huge sticking point for me. There is no knowing the code without experiencing the code. If nothing else, you have to read the tests to know they verify what you think they do. Even then, you might know it does what you want, but can’t know it doesn’t do anything it shouldn’t.
There are no shortcuts to true understanding.
Terr_
8 hours ago
> wherein the machines are treating them like pets
I know some people will groan hearing the phrase again, but: "Reverse centaur." A centaur is a synergistic combination of two things, but now those things are in the wrong position: The human is the computer's tool.
Naturally, how "wrong" it is tends to depend on whether you ask the people who do all the work or the people who own all the computer-capital.
frereubu
7 hours ago
My favourite quote at the moment: "May not man himself become a sort of parasite upon the machines? An affectionate machine-tickling aphid?" Samuel Butler, Erewhon, 1872
giancarlostoro
3 hours ago
When you get to the point that you are doing 1000 PRs I have to seriously wonder what even are you building, I hear about this a lot from a number of companies, but their “too many commits / too many PRs” problem never seems to give anyone from the outside the feeling they are innovating at groundbreaking speeds.
hgh
7 hours ago
I think it's just a different focus on what matters.
There's clearly enormous (human) work that goes in to building and tuning their models.
The software they develop around their model is, relatively, trivial. It's not clear that enormous amounts of (human) energy need to go in to that code.
It seems largely a function of applying human energy towards the things that are marginally most important to generating you money and putting you in front of competitors.
xyzsparetimexyz
7 hours ago
>The software they develop around their model is, relatively, trivial
30,000 changes a day is not my idea of trivial.
californical
6 hours ago
I mean a harness around a model is pretty trivial, I’ve also built one (heavily relying on LLMs to build it).
With a few thousand lines of code over the course of a week or so of my limited evening energy, I have agentic loops, ide integration, and a limited selection of custom tools which are all very safe (no unintended changes), along with session management and awareness of git.
It works great, is customizable, and allows integration with any provider.
If they need 30k changes per day, it’s not due to the nature of the problem!
xyzsparetimexyz
6 hours ago
Oh, I mean I can't disagree there. But they've dug themselves into a hole where they're implementing custom ray tracing in the game engine that claude code runs on or whatever.
californical
6 hours ago
> implementing custom ray tracing in the game engine that claude code runs on
Haha, I don’t doubt it! Pretty damning overall for long-term maintainability of vibe-coded projects, even as the models are great at a lot of individual tasks
status_quo69
7 hours ago
I mean, sure we can assume that (although the Anthropic article from the other day say that claude mostly builds Claude) but this is the Claude Code team which, I again am assuming so I could be wrong, focuses on the TUI/CLI interface to those models, not necessarily the models themselves. That coupled with the fact that this is how a large portion of their clientele interact with their moat, it would be odd to me if they didn't apply significant work towards this thing to keep the hype or productivity train moving
markbao
7 hours ago
> I really don't get the tendency here from Anthropic to describe what's most likely churn? 1000 "PRs" a day on a TUI to wrap a while loop?
From the article, the 30,000 changes a month are in the monorepo, and Claude Code is in a separate repo. So it’s not the TUI that has 1,000 PRs a day.
I could easily see 1,000 PRs a day on the backend. Big tech was doing more than that pre-AI.
zeroonetwothree
2 hours ago
When I worked at Facebook (pre LLM) we were around 50-100k per day. I imagine it's much higher now (quantity, probably not quality).
status_quo69
7 hours ago
Fair enough, I did invent the 1000 a day because the article was vague, I'll edit my submission accordingly to also be vague
j45
2 hours ago
The Anthropic post the other day mentioned part of what helps Claude Code unlock some of this is the sheer ability to use more tokens than the rest to get something going and efficient.
tdhz77
4 hours ago
This is a good answer