Yes, exactly, it’s a post-facto value judgment, not a precise term. If I understand the meaning of the word, “hallucination” is all the model does. If it happens to hallucinate something we think is objectively true, we just decide not to call that a “hallucination”. But there’s literally no functional difference between that case and the case of the model saying something that’s objectively false, or something whose objective truth is unknown or undefinable.

I haven’t read the paper yet, but if they resolve this definition usefully, that would be a good contribution.

I don’t know who/how the term was initially coined in this context, but I’m concerned that the things that make it inaccurate are also, perhaps counterintuitively, things that serve the interests of those who would overstate the capabilities of LLMs, and seek to cloud their true nature (along with inherent limitations) to investors and potential buyers. As you already pointed out, the term implies that the problems that are represented are temporary “bugs”, rather than symptoms of the underlying nature of the technology itself.

This is due to an entire field of AI /machine learning leaning into anthropomorphism shaping terminology reinforced by a narrative.

Confabulation is the term I’ve seen used a few times. I think it reflects what’s going on in LLMs better.

How different things would be if the phenomenon had been called "makin' stuff up" instead. Humans make stuff up all the time, and make up far more outrageous things than AIs make up. One has to ask whether humans are really intelligent /not entirely sarcasm.

It’s still a bug, even if it’s the only way the system can behave as currently designed. I agree that “hallucination” is a poor term for it, though. For medication, we call a bug a “side effect” even though it’s really just a chemical interaction which, given enough information, could be predicted.

Ultimately the computer only does what we tell it to do. That has always been the case and probably always will be, just as we are the result of our inputs.

As with most bugs, I think to solve hallucinations we will need to better understand the input and its interactions within the system.

I think there's a useful distinction between plausible-seeming text that is wrong in some subtle way, vs text that is completely fabricated to match a superficial output format, and the latter is what I wish people used "hallucination" to mean. A clear example of this is when you ask an LLM for some sources, with ISBNs, and it just makes up random titles and ISBNs that it knows full well do not correspond with reality. If you ask "Did you just make that up?" the LLM will respond with something like "Sorry, yes, I made that up, I actually just remembered I can't cite direct sources." I wonder if this is because RLHF teaches the LLM that humans in practice prefer properly formatted fake output over truthful refusals?

This comment should be pinned at the top of any LLM-related comment section.

How sure are you that humans don’t also “simply produce plausible-sounding text”?

I’m watching my toddler learning to speak and it’s remarkably like an early LLM that outputs gibberish with nuggets of sense.

Conversely I’ve seen middle-aged professionals write formal design documents that are markedly inferior to what GPT-4 can produce. This is at every level: overall structure, semantics, and syntax. Diagrams with arrows the wrong way, labels with typos, labels on the wrong object, mixed up icons, tables with made-up content, and on and on.

Aren’t we all doing this to some degree? Half way through a sentence do you known ahead of time exactly what you’re about to say three pages later? Or are you just “completing” the next word based on the context?

I agreed with you until your last sentence. Solving alignment is not a necessity for solving hallucinations even though solving hallucinations is a necessity for solving alignment.

Put another way, you can have a hypothetical model that doesn't have hallucinations and still has no alignment but you can't have alignment if you have hallucinations. Alignment is about skillful lying/refusing to answer questions and is a more complex task than simply telling no lies. (My personal opinion is that trying to solve alignment is a dystopian action and should not be attempted.)

Your argument makes several mistakes.

First, you have just punted the validation problem of what a Normal LLM Model ought to be doing. You rhetorically declared hallucinations to be part of the normal functioning (i.e., the word "Normal" is already a value judgement). But we don't even know that - we would need theoretical proof that ALL theoretical LLMs (or neural networks as a more general argument) cannot EVER attain a certain probabilistic distribution. This is a theoretical computer science problem and remains an open problem.

So the second mistake is your probabilistic reductionism. It is true that LLMs, neural nets, and human brains alike are based on probabilistic computations. But the reasonable definition of a Hallucination is stronger than that - it needs to capture the notion that the probabilistic errors are way too extreme compared to the space of possible correct answers. An example of this is that Humans and LLMs get Right Answers and Wrong Answers in qualitatively very different ways. A concrete example of that is that Humans can demonstrate correctly the sequence of a power set (an EXP-TIME problem), but LLMs theoretically cannot ever do so. Yet both Humans and LLMs are probabilistic, we are made of chemicals and atoms.

Thirdly, the authors' thesis is that mitigation is impossible. It is not some "lens" where mitigation is equal to alignment, in fact one should use their thesis to debunk the notion that Alignmnent is an attainable problem at all. It is formally unsolvable and should be rendered as a absurd as someone claiming prima facie that the Halting Problem is solvable.

Finally, the meta issue is that the AI field is full of people who know zip about theoretical computer science. The vast majority of CS graduates have had maybe 1-2 weeks on Turing machines; an actual year-long course at the sophomore-senior level on theoretical computer science is Optional and for mathematically mature students who wish to concentrate in it. So the problem arises is a matter of a language and conceptual gap between two subdisciplines, the AI community and the TCS community. So you see lots of people believing in very simplistic arguments for or against some AI issue without a strong theoretical grounding that while CS itself has, but is not by default taught to undergraduates.

The term "hallucinate" may not be the best. However, if the context is a device generating a plausible sounding but non-existent scientific reference for a given claim it makes, some words more specific than "producing false information..." seem needed.

These thing make assertions, some true, some false. Since they emulate human behavior, they say things that would tend to convince people of their assertions. So "anthropomorphism" is a complicated question.

True. Actually, researchers know about it. In a sense, i feel "hallucination" is a way to keep the hype up, that we could fix it in the future, given enough data and compute and money. On the other hand, "generating false information by design" is too strong and negative. At least that's what I hear around in my research community.

"Hallucinations" just means that occasionally the LLM is wrong.

The same is true of people, and I still find people extremely helpful.

"So we built a blind-guessing machine, but how can we tweak it so that its blind guesses always happen to be good?"

If the model starts concocting nonexistent sources, like "articles from serious newspapers that just never existed", it is definitely a malfunction for me. AFAIK this is what happened in the Jonathan Turley case.

It’s also problematic to suggest that the problem is unsolvable. It’s unsolvable inside the LLM. It’s 100% solvable within the product.

Agreed. Put simply, everything produced by an LLM today is an hallucination. Errors are simply poor hallucinations.

Consequently we also shouldn't use intelligence or thinking when dealing with AI

Maybe with vanilla LLMs, but new LLM training paradigms include post-training with the explicit goal of avoiding over-confident answers to questions the LLM should not be confident about answering. So hallucination is a malfunction, just like any overconfident incorrect prediction by a model.

Cannot agree enough. Hallucination is a terrible name for this behavior.

Would second what you stated

It's an inaccuracy.

I prefer the clearer term "bullshit"

[flagged]

No, if I ask a human about something he doesn't know, the first thing he will think about is not a made up answer, it is "I don't know". It actually takes effort to make up a story, and without training we tend to be pretty bad at it. Some people do it naturally, but it is considered a disorder.

For LLMs, there is no concept of "not knowing", they will just write something that best matches their training data, and since there is not much "I don't know" in their training data, it is not a natural answer.

For example, I asked for a list of bars in a small city the LLM clearly didn't know much about, and gave me a nice list with names, addresses, phone numbers, etc... all hallucinated. Try to ask a normal human to give you a list of bars in a city he doesn't know well enough, and force him to answer something plausible, no "I don't know". Eventually, especially if he knows a lot about bars, you will get an answer, but it absolutely won't be his first thought, he will probably need to think hard about it.

> Isn’t hallucination just the result of speaking out loud the first possible answer to the question you’ve been asked?

No.

> In fact, if you observe your thinking…

There is no reason to believe that LLMs should be compared to human minds other than our bad and irrational tendency towards anthropomorphizing everything.

> So, to evaluate the intelligence of an LLM based on its first “gut reaction” to a prompt is probably misguided.

LLMs do not have guts and do not experience time. They are not some nervous kid randomly filling in a scantron before the clock runs out. They are the product of software developers abandoning the half-century+ long tradition of making computers output correct answers and chasing vibes instead

> So, to evaluate the intelligence of an LLM based on its first “gut reaction” to a prompt is probably misguided.

There's no intelligence to evaluate. They're not intelligent. There's no logic or cogitation in them.

> A human does not do this.

You obviously had never asked me anything. (Specialy tech questions while drinking a cup of cofee.) If I had a cent for every wrong answer, I'd be already a millionair.

Our brains also seem to tie our thoughts to observed reality in some way. The parts that do sensing and reasoning interact with the parts that handle memory. Different types of memory exist to handle trade offs. Memory of what makes sense also grows in strength compared to random things we observed.

The LLM’s don’t seem to be doing these things. Their design is weaker than the brain on mitigating hallucinations.

For brain-inspired research, I’d look at portions of the brain that seem to be abnormal in people with hallucinations. Then, models of how they work. Then, see if we can apply that to LLM’s.

My other idea was models of things like the hippocampus applied to NN’s. That’s already being done by a number of researchers, though.

Humans totally do this if their prefrontal cortex shuts down due to fight or flight response. See eg stage fright or giving bullshit answers in leetcode style interviews.

> In fact, if you observe your thinking, you might notice that your thought process often takes on different roles and personas.

I don't think it's possible to actually observe one's own thinking. A lot of the "eureka" moments one has in the shower, for example, were probably being thought about somewhere in your head but that process is completely hidden from your conscious mind.

I also like comparing a human thought experiment like Einstein (?) would do to forcing an llm to write code to answer a question. Yes you can make a good guess, but making many smaller obvious decisions that lead to an answer is a stronger process.

The US had a president for eight years who was re-elected on his ability to act on his “gut reaction”s.

Not saying this is ideal, just that it isn’t the showstopper you present it as. In fact, when people talk about “human values”, it might be worth reflecting on whether this a thing we’re supposed to be protecting or expunging?

"I'm not a textbook player, I'm a gut player.” —President George W. Bush.

https://www.heraldtribune.com/story/news/2003/01/12/going-to...

> Let me know if you need any further revisions!

Fun, you got me :)

Very well said!

> A human does not do this

Have you met humans?

> fundamentally ridden with "hallucinations" that will severely limit their practical usage

On the other hand, a LLM that got rid of "hallucinations" is basically just a thing that copy-paste at that point. The interesting properties from LLMs comes from the fact that it can kind of make things up but still make them believable.

C.S. Peirce, who is known for characterizing abductive reasoning and had a considerable on John Sowa’s old school AI work, had an interesting take on this. I can’t fully do it justice, but essentially he held that both matter and mind are real, but aren’t dual. Rather, there is a smooth and continuous transition between the two.

However, whatever the nature of mind and matter really is, we have convincing evidence of human beings creating meaning in symbols by a process Peirce called semiosis. We lack a properly formally described semiotic, although much interesting mathematical applied philosophy has been done in the space (and frankly a ton of bullshit in the academy calls itself semiotic too). Until we can do that, we will probably have great difficulty producing an automaton that can perform semiosis. So, for now, there certainly remains a qualitative difference between the capabilities of humans and LLMs.

What impresses me is frankly how bad it is.

I can't claim to have tried every model out there, but most models very quickly fail when asked to do something along the lines of "describe the interaction of 3 entities." They can usually handle 2 (up to the point where they inevitably start talking in circles - often repeating entire chunks verbatim in many models), but 3 seems utterly beyond them.

LLMs might have a role in the field of "burn money to generate usually-wrong ideas that are cheap enough to check in case there's actually a good one" though.

> Machine learning is about filling in the gaps based off of a prediction.

I think this is a generous interpretation of network-based ML. ML was designed to solve problems. We had lots of data, and we knew large amounts of data could derive functions (networks) as opposed to deliberate construction of algorithms with GOFAI.

But "intelligence" with ML as it stands now is not how humans think. Humans do not need millions of examples of cats to know what a cat is. They might need two or three, and they can permanently identify them later. Moreover, they don't need to see all sorts of "representative" cats. A human could see a single instance of black cat and identify all other types of house cats as cats correctly. (And they do: just observe children).

Intelligence is the ability to come up with a solution without previous knowledge. The more intelligent an entity is, the less data it needs. As we approach more intelligent systems, they will need less data to be effective, not more.

Yes, but it also makes a huge difference whether we are asking the model to interpolate or extrapolate.

Generally speaking, models perform much better on the former task, and have big problems with the latter.

I posit that when someone figures out those architectural changes, the result won't be called an LLM anymore, and the paper will be correct.

You can never completely solve the problem because it's mathematically undecideable, which you probably didn't need this preprint to intuit. That said, a better question is whether you can get good enough performance or not.

Honesty and accuracy builds trust.

And when you trust something it reduces the cognitive load because you don't have to build a mental model of the different ways it could be deceiving you and how to handle it.

Which is why for me at least when I use LLMs I find them useful but stressful.

LLMs are not fundamentally non-deterministic. It's trivial to do e.g. greedy sampling generation.

But but before that we need to achieve what we call "AGI" first.

Even before that we need to define it, first and the reality is no-one knows what "AGI" even is. Thus it could be anything.

The fact that Sam doesn't believe that AGI has been "achieved" yet even after GPT-3.5, ChatGPT, GPT-4 (multi-modal), and with o1 (Strawberry) suggests that what AGI really means is to capture the creation and work of billions, raise hundreds of billions of dollars and for everyone to be on their UBI based scheme, whilst they enrich themselves as the bubble continues.

Seems like the hallucinations are an excuse to say that AGI has not yet been achieved. So it time to raise billions more money on training, energy costs on inference all for it to continue to hallucinate.

Once all the value has been captured by OpenAI and the insiders cash out, THEN they would want the bubble to burst with 95% of AI startups disappearing (Except OpenAI).

LLMs amplify both intelligence and stupidity, which is why Terence Tao finds them about at the level of a mediocre graduate student (and getting better), and you can’t wait for them to die.

Technically, you are right. Donald Knuth still doesn't use e-mail, after all.

But for the global "we" entity, it is almost certain that it is not going to heed your call.

This is "anti-progress", and we must always pursue progress even if it leads us to a self-made reality-melting hellmouth. Onward to Wonderland, I say!

I want to use them. Lots of other people do, too. Feel free to get left behind.

As a game developer, I'm absolutely not going to risk my players' engagement with the game by putting character personalities, dialog, and (therefore) plot into the hands of an AI that's going to play out differently for each player. No way, no how. That's where all my game's value is - that's what pays my rent - and I will not be handing it off to a glorified RNG.

One big difference is that at least some people have a healthy sense for when they may be wrong. This sort of meta-cognitive introspection is currently not possible for an LLM. For instance, let's say I asked someone "do you know the first 10 elements of the periodic table of elements?" Most people would be able to accurately say "honestly I'm not sure what comes after Helium." But an LLM will just make up some bullshit, at least some of the time.

> I treat LLMs like a fallible being, the same way I treat humans.

The issue is LLMs are not marketed in this way. They're marketed as all knowing oracles to people that have been conditioned to just accept the first result Google gives them.

I treat them as always hallucinating and it just so happens that, by accident, they sometimes produce results that resemble intentional, considered or otherwise veritable to the human observer. The accident is sometimes of a high probability, but still an accident. Humans are similar, but we are the standard for what’s to be considered a hallucination, clinically speaking. For us, a hallucination is a cause of concern. For llms, it’s just one of all the possible results. Monkeys bashing on a typewriter. This difference is an essential one, imo.

"Bullshit" is even more appropriate.

Sorry but you're just wrong. There are issues but the paper is written well enough. The content (whether this is really a novel enough idea) is debateable because anyone could have told you that LLMs aren't going to develop the halting algorithm.

> want it to be an automation technology

They want it to be a wage reduction technology. Everything else you've noticed is a direct consequence of this, and only this, so the analysis doesn't actually need to be any deeper than that.

Human brains are also just picking tokens tho. A beautiful illusion of insight and thought in the chaos noise of information. But out of the chaos, the emergence of thought is real. It’s just not exclusive to humans.

I mean… even magicians (mentalists) can reliably hack humans into generating the next token they want you to generate.

If you ask a student to solve a problem while admitting when they don't an answer, they will stop at generating gook for an answer.

LLMs on the other hand regularly spew bogus with high confidence.

This gets at the heart of the problem. It doesn’t produce the wrong tokens. The tokens are right. It’s the data that was “wrong”. Or at least it was weighted “incorrectly” (according to the judge living outside the data with their own context they decide is true)

If you feed AI conspiracy theories and then it tells you Elvis is still alive, that’s an input problem not an algorithm problem.

Now, getting to an AI that doesn’t “hallucinate” is a little more complicated than simply filtering out “conspiracy theories” from data, but IMhO it’s not many orders of magnitudes away. Far from insurmountable in a couple Moores law cycles.

I think human brains operate on the same principal of divining next tokens. We’re just judging AI for not saying the tokens we like best even though we feed AI garbage in and don’t tell AI what it should even care about. “AI doesn’t live here” it wasn’t born “here.”

Someday soon we’ll probably give AI guard rails to respond considering the context of society (the programmers version of society), and it will probably hallucinate less than most humans.

Plus, why do we care about that degree? If we could make it so humans don't hallucinate too that would be great, but it ain't happening. Humans memory gets polluted the moment you feed them new information, as evidence by how much care we have to give when trying to extract information when it matters, like law enforcement.

People rag on LLMs constantly and i get it, but they then give humans way too much credit imo. The primary difference i feel like we see with LLMs vs Humans is complexity. No, i don't personally believe LLMs can scale to human "intelligence". However atm it feels like comparing a worm brain to a human intelligence and saying that's evidence that neurons can't reach human intelligence level.. despite the worm being a fraction of the underling complexity.

> there's always a nonzero chance of hallucination. There's also a non-zero chance that macroscale objects can do quantum tunnelling, but no one is arguing that we "need to live with this" fact.

True, but it is defeatist and goes against a good engineering/scientific mindset.

With this attitude we'd still be practicing alchemy.

Exactly.

LLMs will sometimes be inaccurate. So are humans. When LLMs are clearly better than humans for specific use cases, we don't need 100% perfection.

Autonomous cars will sometimes cause accidents. So do humans. When AVs are clearly safer than humans for specific driving scenarios, we don't need 100% perfection.

it will happen sooner or later*