Fowler is not really redefining "hallucination." He's using a form of irony that emphasises how fundamental "hallucinations" are to the operation of the system. One might also say "you can't get rid of collateral damage from bombs. Indeed, collateral damage is the feature, it's just some of that is what we want to blow up."

You're not meant to take it literally.

It's meant to contrast/correct the claim that

"LLMs produce either truth or they produce hallucinations."

Claims worded like that give the impression that if we can just reduce/eliminate the hallucinations, all that will remain will be the truth.

But that's not the case. What is the case is that all the output is the same thing, hallucination, and that some of those hallucinations just so happen to reflect reality (or expectations) so appears to embody truth.

It's like rolling a die and wanting one to come up, and when it does saying "the die knew what I wanted".

An infinite number of monkeys typing on an infinite number of typewriters will eventually produce the script for Hamlet.

That doesn't mean the monkeys know about Shakespeare, or Hamlet, or even words, or even what they are doing being chained to typewriters.

We've found a way to optimize the infinite typing monkeys to output something that passes for Hamlet much sooner than infinity.

I actually found that comment interesting. It's pointing towards something I've struggled with around LLMs. They are (currently) incapable of knowing if what they output is correct, so the idea that "it's all hallucinations" acknowledges that point and gives useful context for anyone using LLMs for software development.

I’ve never liked that this behaviour is described using the term “hallucination”.

If a human being talked confidently about something that they were just making up out of thin air by synthesizing based (consciously or unconsciously) on other information they know you wouldn’t call it “hallucination”: you’d call it “bullshit”.

And, honestly, “bullshit” is a much more helpful way of thinking about this behaviour because it somewhat nullifies the arguments people make against the use of LLMs due to this behaviour. Fundamentally, if you don’t want to work with LLMs because they sometimes “bullshit”, are you planning on no longer working with human beings as well?

It doesn’t hold up.

But, more than that, going back to your point: it’s much harder to redefine the term “bullshit” to mean something different to the common understanding.

All of that said, I don’t mind the piece and, honestly, the “I haven’t the foggiest” comment about the future of software development as a career is well made. I guess it’s just a somewhat useful collection of scattered thoughts on LLMs and, as such, an example of a piece where the “thoughts on” title fits well. I don’t think the author is trying to be particularly authoritative.

The point, though awkwardly stated, is that there is no difference between 'hallucination' output and any other output from these compressed databases, just like there is no such difference between two queries on a typical RDBMS.

It's a good point.

> You've just taken the label conventionally attached to a bad behavior, attached it to a broader category that includes all behavior, and used the power of equivocation to make something that sounds novel without saying anything new.

You mean like every evangelist says AI changes everything every 5 min...When in reality what they mean is neural nets statistical code generators are getting pretty good? because that is almost all AI there is at the moment?

Just to make the current AI sound bigger than it is?

If everything is a hallucination then nothing is a hallucination.

Thanks for listening to my Ted Talk.

> In my company I feel that we getting totally overrun with code that's 90% good, 10% broken and almost exactly what was needed.

This is painfully similar to what happens when a team grows from 3 developers to 10 developers. All of sudden, there's a vast pile of coding being written, you've never seen 75% of it, your architectural coherence is down, and you're relying a lot more on policy and CI.

Where LLM's differ is that you can't meaningfully mentor them, and you can't let them go after the 50th time they try turn off the type checker, or delete the unit tests to hide bugs.

Probably, the most effective way to use LLMs is to make the person driving the LLM 100% responsible for the consequences. Which would mean actually knowing the code that gets generated. But that's going to be complicated to ensure.

LLMs are amazing at producing boilerplate, which removes the incentive to get rid of it.

Boilerplate sucks to review. You just see a big mass of code and can't fully make sense of it when reviewing. Also, Github sucks for reviewing PRs with too many lines.

So junior/mid devs are just churning boilerplate-rich code and don't really learn.

The only outcome here is code quality is gonna go down very very fast.

Perlis, epigram 7:

7. It is easier to write an incorrect program than understand a correct one.

Link: http://cs.yale.edu/homes/perlis-alan/quotes.html

"but quality is definitely taking a hit now that no-one is able to keep up."

And its going to get worse! So please explain to me how in the net, you are going to be better off? You're not.

I think most people haven't taken a decent economics class and don't deeply understand the notion of trade offs and the fact there is no free lunch.

> What bothers me the most is that I much prefer to build stuff rather than fixing code I'm not intimately familiar with.

Me too. But I think there's a split here. Some people love the new fast and loose way and rave about how they're experiencing more joy coding than ever before.

But I tried it briefly on a side project, and hated the feeling of disconnect. I started over, doing everything manually but boosted by AI and it's deeply satisfying. There is just one section of AI written code that I don't entirely understand, a complex SQL query I was having trouble writing myself. But at least with an SQL query it's very easy to verify the code does exactly what you want with no possibility of side effects.

I'd argue that this awareness is a good thing; it means you're measuring, analyzing, etc all the code.

Best practices in software development for forever have been to verify everything; CI, code reviews, unit tests, linters, etc. I'd argue that with LLM generated code, a software developer's job and/or that of an organization as a whole has shifted even more towards reviewing and verification.

If quality is taking a hit you need to stop; how important is quality to you? How do you define quality in your organization? And what steps do you take to ensure and improve quality before merging LLM generated code? Remember that you're still the boss and there is no excuse for merging substandard code.

Imagine someones add 10 UTs carefully devised and someone notices they need 1 more during the PR.

Scenario B, you add 40 with an LLM, that look good on paper but only cover 6 of the original ones. Besides, who's going to pay careful attention to a PR with 40.

"Must be so thorough!".

As Fowler himself states, there's a need to learn to use these tools properly.

In any case poor work quality is a failure of tech leadership and culture, it's not AI's fault.

I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better.

https://jstrieb.github.io/posts/llm-thespians/

All models are wrong, some are merely useful - 1976/1933/earlier adage.

Intelligence in a way is the ability to filter out useless information. Be it, thoughts or sensory information

Yes, can't remember who said it but LLM's always hallucinate, it is just that they are 90 something percent right.

Finally, an opinion on here that’s reasonable and isn’t “AI is perfect” or “AI is useless”.

It works to confuse people and make them sound like they’re lying when they’re not, too. Gotta be careful with this.

> This works on people as well!

Only within certain conditions or thresholds that we're still figuring out. There are many cases where the more someone recalls and communicates their memory, the more details get corrupted.

> Cops do this when interrogating.

Sometimes that's not to "get sense of the variation" but to deliberately encourage a contradiction to pounce upon it. Ask me my own birthday enough times in enough ways and formats, and eventually I'll say something incorrect.

Care must also be taken to ensure that the questioner doesn't change the details, such as by encouraging (or sometimes forcing) the witness/suspect to imagine things which didn't happen.

Triple modular redundancy. I remember reading that's how Nasa space shuttles calculate things because a processor / memory might have been affected by space radiation https://llis.nasa.gov/lesson/18803

Who remembers that scene on Better Call Saul between Lalo, Saul, and Kim?

It feels like Tdd/llm connection is implied — “and also generate tests”. Thought it’s not cannonical tdd of course. I wonder if it’ll turn the tide towards tech that’s easier to test automatically, like maybe ssr instead of react.

I'd rather say that LLMs live in a world that consists entirely of stories, nothing but words and their combinations. Thy have no other reality. So they are good at generating more stories that would sit well with the stories they already know. But the stories are often imprecise, and sometimes contradictory, so they have to guess. Also, LLMs don't know how to count, but they know that two usually follows one, and three is usually said to be larger than two, so they can speak in a way that mostly does not contradict this knowledge. They can use tools to count, like a human who knows digits would use a calculator.

But much more than an arithmetic engine, the current crop of AI needs an epistemic engine, something that would help follow logic and avoid contradictions, to determine what is a well-established fact, and what is a shaky conjecture. Then we might start trusting the AI.

In that framing, you can look at an agent as simply a filter on those hallucinations.

I've prefered to riff off of the other quote:

"All (large language) model outputs are hallucinations, but some are useful."

Some astonishingly large proportion of them, actually. Hence the AI boom.

I find it a bit of a reductive way of looking at it personally

Isn’t that why people argue against calling them hallucinations?

It implies that some parts of the output aren’t hallucinations, when the reality is that none of it has any thought behind it.

I have a very similar (probably unoriginal) thought about some human mental illnesses.

So, we VALUE creativity, we claim that it helps us solve problems, improves our understanding of the universe, etc.

BUT people with some mental illnesses, their brain is so creative that they lose the understanding of where reality is and where their imagination/creativity takes over.

eg. Hearing voices? That's the brain conjuring up a voice - auditory and visual hallucinations are the easy example.

But it goes further, depression is where people's brains create scenarios where there is no hope, and there's no escape. Anxiety too, the brain is conjuring up fears of what's to come

Nah I don't agree with this characterization. The problem is, the majority of those hallucinations are true. What was said would make more sense if the majority of the responses were, in fact, false, but this is not the case.

What would such a workflow look like? I have Alfred but mainly just use the clipboard feature. I've tried to get into automation but struggled for inspiration. This one seems good!

Are you just opening a browser tab?

Have you considered that LLMs are also biased against new languages and libraries, so the code quality will be worse compared to something more established regardless of what you personally think/feel?

> written code is no longer a time sink

It still is, and should be. It’s highly unlikely that you provided all the required info to the agent at first try. The only way to fix that is to read and understand the code thoroughly and suspiciously, and reshaping it until we’re sure it reflects the requirements as we understand them.

> We are headed in a direction where written code is no longer a time sink.

Written code has never been a time sink. The actual time that software developers have spent actually writing code has always been a very low percentage of total time.

Figuring out what code to write is a bigger deal. LLMs can help with part of this. Figuring out what's wrong with written code, and figuring out how to change and fix the code, is also a big deal. LLMs can help with a smaller part of this.

> Juniors can onboard faster and more independently with LLMs,

Color me very, very skeptical of this. Juniors previously spent a lot more of their time writing code, and they don't have to do that anymore. On the other hand, that's how they became not-juniors; the feedback loop from writing code and seeing what happened as a result is the point. Skipping part of that breaks the loop. "What the computer wrote didn't work" or "what the computer wrote is too slow" or even to some extent "what the computer wrote was the wrong thing" is so much harder to learn from.

Juniors are screwed.

> LLMs have the ability to lighten cognitive loads and increase productivity,

I'm fascinated to find out where this is true and where it's false. I think it'll be very unevenly distributed. I've seen a lot of silver bullets fired and disintegrate mid-flight, and I'm very doubtful of the latest one in the form of LLMs. I'm guessing LLMs will ratchet forward part of the software world, will remove support for other parts that will fall back, and it'll take us way too long to recognize which part is which and how to build a new system atop the shifted foundation.

Written code is an auditable entity in regulated businesses, like banks. Fowler suggests we get more comfortable with uncertainty because other eng domains have developed good risk mgmt. Cart before horse. Other engineering domains strive to mitigate the irreducible uncertainty of the physical world. AI adds uncertainty to any system. And there are many applications where increased uncertainty will lead to an increase in human suffering.

Funnily enough, I left a similar comment just the other day: https://news.ycombinator.com/item?id=44944717

The conclusion I reached was different, though. We learnt how to do outsourcing "properly" pretty quickly after some initial high-profile failures, which is why it has only continued to grow into such a huge industry. This also involved local talent refocusing on higher-value tasks, which is why job losses were limited. Those same lessons and outcomes of outsourcing are very relevant to "bot-sourcing'.

However, I do feel concerned that AI is gaining skill-levels much faster than the rate at which people can upskill themselves.

> Prompting (overseas) humans [...] After over a decade of trying that, we learned that had... flaws.

I think this is a US-centric point of view, and seems (though I hope it's not!) slightly condescending to those of us not in the US.

Software engineering is more than what happens to US-based businesses and their leadership commanding hundreds or thousands of overseas humans. Offshoring in software is certainly a US concern (and to a lesser extent, other nations suffer it), but is NOT a universal problem of software engineering. Software engineering happens in multiple countries, and while the big money is in the US, that's not all there is to it.

Software engineering exists "natively" in countries other than the US, so any problems with it should probably (also) be framed without exclusive reference to the US.

I'm sure the blacksmiths and weavers will find solace in that take. Their time will return!

Predicting the future isn't about being correct tomorrow, rather it’s about selling something to someone today.

An insight I picked up along the way…

Reminds me of the classic Yogi Berra, “It's tough to make predictions, especially about the future”.

To me, it is more specific to say that many futurists don't take into account the social and economic network effects of changes taking place. Many just act as if the future will continue on completely unchallenged in this current state. But if you look at someone like Kurzweil, you can see the very narrow and specific focus of a prediction, which has proved to be more informative to me as a high bar of futurism.

Part of what got me into software was this: no matter how complex or impressive the operation, with enough time and determination, you could trace each step and learn how a tap on a joystick lead to the specific pixels on a screen changing.

There’s a beautiful invitation to learn and contribute baked into a world where each command is fully deterministic and spec-ed out.

Yes, there have always been poorly documented black boxes, but I thought the goal was to minimize those.

People don’t understand how much is going to be lost if that goal is abandoned.

As pertaining to software development, I agree. I've been hearing accounting (online and from coworkers) of using LLMs to do deterministic stuff. And yet, instead of at least prompting once to "write a script to do X," they just keep prompting "do X" over and over again. Seems incredibly wasteful. It feels like there is this thought of "We are not making progress if we aren't getting the LLM to do everything. Having it write a script we can review and tweak is anti-progress." No one has said that outright, but it's a gut feeling (and it wouldn't surprise me if people have said this out loud).

'potatolicious says we're going forwards: https://news.ycombinator.com/item?id=44978319

I would rather say it like this: Very good, very hardworking engineers spent years of their lives building the machine that raised us from the non-determinism of messy physical reality. The technology that brought us perfect, replicable, and reliable math from sending electrons through rocks has been deeply underappreciated in the "software revolution".

The engineers at TSMC, Intel, Global Foundries, Samsung, and others have done us an amazing service, and we are throwing all that hard work away.

It's uncertainty, not non-determinism. I don't design a rocket with no idea about the chain of events. I have inscrutable uncertainty everywhere, from manufacture to flight.

It was forward when Newton discovered the beautiful simple determinism of physics.

Was it going backwards when the probabilistic nature of quantum mechanics emerged?

Junior engineers get better and learn from their mistakes. An LLM will happily make then same mistake 10 times if you try asking it to do something similar in 3 months. In the long term, I don't think LLMs actually save you time considering the amount of extra time spent reviewing/verifying its code and fixing tech debt.

If an AI can’t write the code after two attempts, I’ve never had success trying ten times

(slightly off-topic)

Students shouldn't ever have become so laser-focused on a single career path anyway, and even worse than that is how colleges have become glorified trade schools in our minds. Students should focus on studying for their classes, getting their electives and extracurriculars in, getting into clubs... Then depending on which circles they end up in, they shape their career that way. The thought that getting a Computer Science major would guarantee students a spot in the tech industry was always ridiculous, because the industry was just never structured that way, it's always been hacker groups, study groups, open source, etc. bringing out the best minds

It has never been anywhere close to certain, we just had 20 years of wild, unsustainable growth that encouraged people to cover their eyes and pretend the ride would go on forever. 20 years of telling everyone under the age of 30 that of course they should learn to code and that CS was the new medical or legal degree. 20 years of smugly acting like we are the inevitable future when we are, in fact, subject to the same ups and downs as every other career.

The answers are even kind of contradictory. If the answer to "will we ever need software engineers again in the future?" is "we don't know" then the answer to "should I spend time and money to enter software engineering?" should be "no".

Exactly, there is a reason why we don't program in natural languages and use "programming languages" which use a subset of keywords to program. Natural language is too ambiguous and not of use in serious programming.

It did so because his training, weights and context and a billion of matrix calculations led him there.

How is it wrong?

Do you have any hard number and data to state that tab/auto complete are less popular than agentic coding?

> With enough specificity, LLMs will 100% return what you want.

By now I’m sure it won’t. Even if you provide the expected code verbatim, LLMs might go on a side quest to “improve” something.

Apologies for arguing from first principles, but for anything that spurs a lot of activity, the only alternative to a bubble is this: people ramp up investment and activity and enthusiasm only as much as the underlying thing can handle, then gradually taper off the increase and gently level off at the equilibrium "carrying capacity" of the new technology.

Does that sound like any human, ever, to you?

(The only time there isn't a bubble is when the thing just isn't that interesting to people and so there's never a big wave of uptake in the first place.)

I don't know enough about the early history of the airline industry but there was very definitely a long series of huge bubbles in the railroad industry.

The Intelligent Investor by Graham talks about investors putting so much money into airlines and air freight that it became impossible to make a return. I don't know if you would call that a bubble, maybe just over exuberance.

The AI bubble also isn't about the technology.

Just taking your first example, yes I believe you could characterize a lot of the investments in electrification as speculative bubbles even though there was underlying value that was borne out.

https://en.wikipedia.org/wiki/Public_Utility_Holding_Company...

I want to re-phrase your definition.

To me a bubble reflects a market disconnect from fundamentals - wherein prices go up steeply, with no help from the fundamentals (expected growth in base year cash flows and risk).

Indeed there is a subtle difference between a bubble and speculation of what could come of a technology. But the two are connected because the effects of technology are reflected by investors in asset prices.

Besides electricity (answered above), there were bubbles related to steel production in its early days intertwined with the railroad bubble that drove huge investments in steel production for railroad development; when the railroad companies crashed it brought down steel as well; ultimately both survived in a more subdued form (as with the dot coms).

avation: likewise there was the "Lindberg Boom" https://en.wikipedia.org/wiki/Lindbergh_Boom which led to overspeculation and the crash of many early aviation companies

As far as an LLM is concerned, there is no difference between "negative" hallucination and a positive one. It's all just tokens and embeddings to it.

Positive hallucinations are more likely to happen nowadays, thanks to all the effort going into these systems.

No, it’s just an attempt to pretend wrong outputs are some special case when really they aren’t. They aren’t imagine something that doesn’t exist they are just running the same process they do for everything else and it just didn’t work.

If you disagree then I would ask what exactly is the “specific behaviour” you’re talking about?

I'll take the bait.

What didn't he understand properly about inversion of control then.

These metaphors all suck. Well, ok, yours is funny. But anyway, LLMs are just very different from any human.

They are extremely shallow, even compared to a junior developer. But extremely broad, even compared to the most experienced developer. They type real fuckin fast compared to anyone on earth, but they need to be told what to do much more carefully than anyone on earth.

No, they're like a completely nontechnical marketing person who has a big library of papers on related subjects and who's been asked to generate a whitepaper by pulling phrases. What comes out will probably have proper grammar and seem perfectly reasonable to another person with no knowledge of the field, but if actually read by someone knowledgeable may be complete gibberish taken as a whole.

Individual sentences and paragraphs may mostly work, but it's an edifice built on sand out of poorly constructed bricks plus mortar with the wrong proportions (or entirely wrong ingredients).

LLM output is "truthy" - it looks like it might be true, and sometimes it will even be accurate (see also, "stopped clock") but depending on it is foolish because what's generating the output doesn't actually understand what it's putting out - it's just generating output that looks like the kind of thing you've requested.

Overconfident, but also easy to sway their opinion.

They follow directions for maybe an hour and then go off and fix random shit because they forgot what their main task was.

They'll tell you to your face how great your ideas were, and that you're absolutely right about something, then go implement it completely incorrectly.

They add comments to literally everything even when you ask them to stop. But they also ignore said comments sometimes.

Maybe they are kinda like us lol.

> who drinks heavily on the job

Yes, that's why you should add "... and answer in the style of a drunkard" to every prompt. Makes it easier to not forget what you are dealing with.

My analogy is a cursed monkey paw with unlimited wishes. It's actually really really powerful but you have to be careful of leaving any possible ambiguity in your wishes.

Like a person that has memorized every single answer in the world to every single question ever asked - but is completely dead behind the eyes.

Experienced and knowledgeable and they also believe in the technical equivalent of flat-Earthism in many, many non-trivial corners.

And if you push back on that insanity, they'll smile and nod and agree with you and in the next sentence, go right back to pushing that nonsense.

> Overconfident, forgetful, sloppy, easily distracted

And constantly microdosing, sometimes a bit too much.

> and they don't get mad when you fire them!

No, typically __you__ are mad when you fire them ...

“You’re absolutely right!”

Yes. Funny

But it is nothing like a wetware colleague. It is a machine.

> Are we going to continue to use the 60 billion dollar data centers in Louisiana when the bubble burst

Why not?

Tracks in the middle of nowhere are useless.

Data centers in the middle of nowhere are useful, as long as energy, cooling and uplink are cost-effective.

(Not saying Louisiana is in the middle of nowhere!)

I feel exactly the same way.

Even this post by Martin Fowler shows he's an aging dinosaur stuck in denial.

> I’ve often heard, with decent reason, an LLM compared to a junior colleague. But I find LLMs are quite happy to say “all tests green”, yet when I run them, there are failures. If that was a junior engineer’s behavior, how long would it be before H.R. was involved?

I don't know what "LLM's" he's using but I just simply don't get hallucinations like that with cursor or claude code.

He ends with this: > LLMs create a huge increase in the attack surface of software systems. Simon Willison described the The Lethal Trifecta for AI agents: an agent that combines access to your private data, exposure to untrusted content, and a way to externally communicate (“exfiltration”). That “untrusted content” can come in all sorts of ways, ask it to read a web page, and an attacker can easily put instructions on the website in 1pt white-on-white font to trick the gullible LLM to obtain that private data.

Not sure why he is re-iterating well known prompt injection vulnerability, passing it off as a general weakness of LLM's that applies to all LLM use when that's not the reality.

60+ APIs? Odd way to word. Did you mean endpoints?

What's the impressive thing here? No one said AI can't do boilerplate. Especially if you're doing run of the mill crud and starting from a clean slate, in fact, that's probably one of its only useful usages in my opinion. You still need to understand your system when maintaining or adding features to it, do a code review, understand the architecture, ensure its architecture allows for pivots in design/functionality, etc.

Can you share the code with us? I'd love to see that

Okay, but LLMs will produce a thousand lines of code for Hello World.

Plus, there is no telling how many bugs you have in that code.