I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do. They figure “it’s science, of course there’s math and theory behind it.”

AI research is almost as purely empirical as the gradient descent loops its practitioners use to optimize their models. “Why” anything at all works is barely an afterthought.

>But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.

We might not understand particular "emergent" capabilities, but the low level mechanism is not just understood, but a deterministic algorithm with a handful of basic componets, that are well understood themselves.

Please go on, else you risk sounding like the person you’re criticising. The structure of the neural network is somewhat opaque because it’s hard to understand as the individual weights can’t be usefully interrogated, and naturally, it comes from big datasets which a human brain can’t really absorb in toto. Your comment was interesting so I’d like more of it.

They don't "make decisions".

That's like saying "my d20 decided to roll a 17"

>> an LLM is completely opaque

And despite that, although they are not like that in practice as there are too many uncontrolled variables, with temperature at zero, for the same input they produce always the same reply.

We understand how networks compute decisions, though explaining every internal influence remains difficult.

thats no different from not understanding why a sufficiently complex and obfuscated binary of a program "makes decisions"

Isn’t it convenient that nobody understands? How could we possibly regulate something that isn’t understood? It’s like social media all over again. We can’t be responsible for someone else’s content; it’s not us so you can’t penalize us!!

I disagree, I think 'Hallucination' is a risk-shedding weasel-word. It's meant to shift blame away from the technology and its creator (multibillion dollar AI companies etc) in a way that doesn't hold those actors accountable or responsible for the outcomes.

In any other software it would be an error, regression, bug. And in a human process it would be at ~least something someone would call 'bullshit'.

I also agree with the parent, and I would also suggest "hallucination" is better than "error" which might imply an available deterministic correction. Hallucination makes it clear we're dealing with something different than an "error" or "bug".

Knowingly causing errors is not forgivable whereas hallucinations sounds esoteric and moves blame away from the people who are knowingly causing errors. It’s marketing speak.

>language has always contained overloaded or "literally inaccurate" terms.

"literally" is a great example of this, because it can also mean "not literally, but with emphasis".

OpenAI calls them 'mistakes'. But that's just a fig leaf.

Google does it too: "AI responses may include mistakes."

Mistakes have an air of innocence. But these are not mistakes, they are purposefully releasing stuff that they know is broken, they just don't know when it is broken...

It’s not an error or a hallucinations it works correctly every time, and statistically picks the next token for the sequence.

Retuning inf or crashing would be an error.

If you want to ascribe some kind of meaning to the tokens, then maybe the training data was insufficient to predict the token in the sequence you wanted, but it doesn’t predict the next “fact”, and it doesn’t “think” it predicts the next token.

Sure… but being wrong doesnt necessarily make it a hallucination:

>It was only just before the planned operation that officials dug deeper into the report put together by a special operations command analyst and found it had been generated with the help of artificial intelligence (AI) — and that a chatbot the analyst had used inaccurately identified the material the ship was carrying. CNN was not able to learn what the misidentified cargo was.

From the point of view of the system, this is an error. It is incorrect information.

The term "hallucination" feels much more like anthropomorphizing. The word hallucination implies an aberrant condition. A much better term would be "confabulation".

You don't trust things or individuals that confabulate.

Anthropomorphizing the tool led directly to this problem, where we nearly started a war with China.

I prefer “confabulation”. It seems truer to what is happening:

The LLM isn’t seeing something that’s not there, but deliberately making up _something_ so that it can return a response.

hallucination is common language for these models at this point which describes a particular type of error where the models make shit up.

it is noticeable that the form of this particular error holds a similar shape to what is casually described as hallucinations, in that there is a generated content that often appears to blend naturally into the rest of the output but is false.

the term hallucination often invokes a caution that this particular type of error may be influential and believable and is particularly dangerous

Every output an LLM creates is a hallucination.

The term “hallucination” is a projection of inappropriate expectations onto a program. We know that LLMs are not “truth machines,” but we really want them to be. So when they produce a result that happens not to match external reality - which, it should be noted, LLMs don’t generally have access to - we call it an hallucination.

“Bugs” are completely different. With bugs, we have a clear specification and we have a program that’s supposed to meet that specification. If it doesn’t, we say the program has bugs, and if it’s important enough we can change the program to eliminate the bugs.

You can try to apply similar logic to LLMs, but you’d be making a category error, and you’ll fail to get the results you want in general. It’s not the same thing at all.

If anything, the concept of an LLM hallucination is a bug in human understanding of LLMs.

Note that "bug" came from an actual moth in a computer: https://www.computerhistory.org/tdih/september/9/

note the airline industry has moved past firing pilots who make mistakes, since that turned out to be a recipe for more plane crashes, not less. Instead, they find out why the mistake happened, and fix it. In some cases, this involves firing the pilot. They do not do that by default.

What is it insanely useful for? (Besides convincing investors to sink more money into LLM-related companies? Because that is the one thing it does seem to truly be good at.)

LLMs generate text output that appears to be useful, but regularly is not. They're alleged to be a substantial boost to writing code, but that verdict seems to be in dispute. They can generate custom mediocre prose at scale, but that seems to be of ultimately limited utility (although it may be a godsend for propagandists).

We're coming up on the 4th anniversary of ChatGPT's release. And while I get that revolutionary technologies can take a while to mature, the Wright Brothers and Goddard weren't preaching imminent societal transformation by the end to the decade from the rooftops, either. (And that's before we get into the how they got there - getting to ignore laws and steal whatever they wanted might be insanely useful to a lot of people.)

Back then it was called “the truth” only later did it become something else. Perhaps an untruth.

both. they wanted to deceive people (lie). so they made up a reason (hallucinate).

Individual incidents aren't representative of accuracy or well-tuned truth seeking processes, that can only be assessed over time.

Pedantic clarification: The (original German) song itself didn't mention an actor in particular who had released the balloons, just that there were 99 balloons that flew to the horizon and jet fighters being scrambled in response. The epilogue tells of the consequent whole bunch of lasting destruction as the narrator talks about their patrols.

The English translation "99 Red Balloons" is considerably different as far as the details go.

We did sink many non-Chinese ships because a human thought the fish they were transporting were drugs.

That's a very limited view of why ww1 started.

That is a below high school level understanding of what caused WW1, for what it’s worth.

the assassination of archduke franz ferdinand was just the trigger but not the reason. the war would have started anyway.

Not sure what is there to laugh at when millions died in pretty horrible ways including americans, anyway that was classical war mongering where one of the parties got exactly what they wanted (prussians/germans). That they got more than they wanted and how it folded them is part of history.

This case, its bullshit machine bullshitting randomly in between specs of stolen wisdom. Nobody asked for that, nobody is in control. We all humans lose in all cases. Quite different scenarios if you asked me.

The user was absolutely right. There were over 100 children in that elementary school [0]. I should not have recommended the double-tap.

Checking to see if there are better targets...

Clauding...

[0] https://en.wikipedia.org/wiki/2026_Minab_school_attack#:~:te...

Just because they haven't done something in the past doesn't mean they won't in the future. China has been providing material support to Iran both in terms of intel and military hardward, so it's not outlandish to be monitoring and planning for it to continue happening into the future.

Responsible, no (as someone else pointed out). I don't know anything else.

AI technology was used to target Palestinian / Hamas targets (and often their families), ultimately there was a human behind the wheels that would do a thumbsup or down. They spent less than two seconds on many targets. Make of that what you will. https://en.wikipedia.org/wiki/AI-assisted_targeting_in_the_G...

> Wasn't AI responsible for the US bombing a girl's school?

No. The morons who relied on it were responsible.

I understand what you mean but the phrasing is unfortunate. The people and organizations who use AI are fully responsible for the outcomes of their use of AI.

Sorry your computer god is demonstrably useless. Even the boosters are running out of tokens apparently.

Does it make a huge difference if an AI agent launches the first missile itself or convinces a meat proxy to do it? End result will be the same.

Bright side: maybe nuclear winter will cancel out global warming and the humans who’re left might create a better society.

A lot of leadership is just making arbitrary decisions. Vendor A or B, it doesn't really matter. AI just lets that happen much faster. The consequences are the same though.

And all this time we've been worried about Skynet using some highly intelligent scheme to destroy humanity. LOL

For a long time, the movie “War Games” while entertaining, also seemed a bit absurd.

Doesn’t seem so absurd anymore…

OpenAI's Codex has a "ask user question" tool, which helpfully has a hard coded 60 second inactivity timer that passes a "user did not pick, just pick yourself" message to the model.

Well that was intentional. All the integrations with the US military didn't happen by accident.

Edit: There's no way you're going to get these AI systems and not have them integrated with the military. It's a consequence of releasing this stuff on the world.

OpenAI and Antropic are trying to create that reality, but it would be ideal if they failed.

I don't know how you can confidently say that without first knowing who the victims are. If AI causes a disaster that harms a member of the American billionaire class, someone will need to satisfy their bloodlust.

I'm excited for AI executives declaring private military action against each other.

There've already been numerous postmortems published about what happened with the school strike.

The school was located on a former military base. 10 years ago, that location was a legit military target. Nobody bothered to update the satellite imagery from 2013 when feeding it into whatever LLM was assisting in targeting. It saw an airstrip and missile base. The human reviewing the targeting saw an airstrip and missile base. When you go to take out a military target, you don't send one missile. You send a missile or two in first, and then another couple in a few minutes later.

Hanlon's Razor very much applies here. There's no need to posit that the children were IRGC members or that the attack was deliberate or even that the children were the targets. There's a very obvious explanation, which is that nobody bothered to check the date and assumed that if it was military base in 2013 it was still a military base today.

It would make sense, but minutes later? Did the IRGC brass break out the infamous FTL dirtbikes?

you are way to forgivining in how stupid the military is under current leadership.

This sounds like the same type of Israel apologism. "OK, well we kill a bunch of innocents, but they were related to all those evil people"

We know AI has been used to do target planning: https://www.washingtonpost.com/technology/2026/03/04/anthrop...

It is a quote from a future history book. And it may well be written like that, or some light paraphrase.

It is exactly what happened.

The law of war is an attempt to have belligerent nations voluntarily set some limits on how far they’ll go when in conflict, in the hopes that the number of non-combatant casualties is reduced. That said, the answer to your question is that a nation that tries to follow the law of war has procedures in place to catch errors like bad intel. That is what a court would adjudicate.

Well, on the civilian side that's exactly what's happening with openai hacking other companies, so I guess it stands to reason that the military wants to get in on that too.

> Does a horrific war crime like My Lai[0] get scrutinized and investigated

Of course scrutiny and investigations are better than nothing - but keep in mind in My Lai - all the charges were eventually dropped except for one guy (Calley), who in the end got 3 years of house arrest.

> gives the military a way to cover

it's not like they didn't just sweep crap under the rug before AI.

did Colin Powell go to jail for lying to the UN? of course not. did Colin Powell go to jail for smuggling anthrax into a UN meeting? of course not. he just blamed it on "being misled by bad intelligence". Did anyone go to jail for that bad intelligence? of course not.

But we had to pay through the nose for decades of this shit in Iraq and Afghanistan (hundreds of billions), all for absolutely nothing. Did anyone even explain why the fuck, or was held responsible? of course not. They don't need AI to just ignore shit.

AI does not actually cover anything. Just because OpenAI and Antropic act like "AI did it" is get out of jail card does not mean it is.

But USA wont prosecute own war crimes unless forced to, regardless of AI.

> AI gives the military a way to cover or deflect war crimes

Huh? Who is accepting “it was AI” as cover for war crimes?

> Does a horrific war crime like My Lai[0] get scrutinized and investigated in 2026 or do people just say "eh maybe AI gave them bad intel" and ignore it?

Of course it does. The girl’s school bombing and strikes on fishermen are scrutinized. Why wouldn’t any atrocity?

We do not suffer from a lack of scrutiny. We suffer from a lack of accountability. AI is only tangential.

It might not happen, but it's already passed the house and has support in the senate.

Maybe the victor will be whoever doesn't fall into the trap of believing that a next word predictor optimized through RLHF to sound convincing to the average human [1] is in any way intelligent, and restructuring their entire economy and military around doing whatever the magic oracle machine says.

If the entire "free world" goes all-in, the history books will likely be written by Iran, Russia and North Korea.

[1] possibly to the point of presenting a superstimulus that bypasses any facilities for critical thinking. This might be easier to solve than many other problems that are used for evaluating A"I" performance, and very likely doesn't require genuine intelligence, just brute-force search + evolution

AI's power and water requirements are at odds with what is required to address climate change, so it really isn't a given that it's either inevitable or a permanent addition to society.

I'm clearly no fan of this administration or US Military but I don't believe for a minute they purposely wanted to murder 250 school kids

if I remember the reporting correctly there were like 1000+ targets picked for simultaneous bombing by Tomahawks (at $4 Million a pop) and the division that reviews targets had been dismantled by this administration so the "AI" list was never double-checked

apathy vs malice

though the final reason doesn't matter to those kids or their families

Iran is going to end like Iraq and Afghanistan and Vietnam and Korea way before that, we just finally leave because we can't "win" and leave it worse than the horror it was in the first place

I don't even think the Dems can change it if they somehow win the Senate, it's going to be this nightmare through 2029

How often should we allow human provided targets to go unverified by a human before killing every innocent student inside the target?

That is, the question of whether targets should be verified is orthogonal to the question of whether humans or AIs supply targeting data with fewer errors.

That we do. So it'd be pretty cool if that was done responsibly too, and being able to answer my question very much matters for that.

If these things do outperform servicemen, or if there's not enough data to assess that, then selectively reporting this in headlines is not exactly helping anyone, quite the contrary.

Especially knowing that there's a significant anti-AI sentiment among people as-is, I really wouldn't put it behind news outlets to couple that with some conveniently missing context and take advantage of people for some cheap clicks. Kinda been the theme for a while now if you noticed.

Whether that then results in society making responsible decisions, and holding the correct people accountable...

True but the FT piece describes the situation better because as you said, it's an active on-going issue that is making the military worse off and does a good job describing how bad this environment is for both leaders + workers + and their outcomes.

Correction, it was 95% of the time in published works. My data may be antiquated. ymmv =3

https://www.kcl.ac.uk/news/artificial-intelligence-under-nuc...

https://www.youtube.com/watch?v=wL22URoMZjo

One could argue "AI" fanatical cults are already a societal problem, as people seek emotional support & therapeutic value from what is essentially a polished psychopathic entity.

Carl Sagan had predicted people losing understanding of their world could be a possible tragic consequence of irrational thought. Perhaps an allusion to the lotus-eaters from Homer's The Odyssey would be more accurate. =3

https://www.amazon.com/Demon-Haunted-World-Science-Candle-Da...