Beliefs that are true for regular software but false when applied to AI

237 pointsposted 8 hours ago
by beyarkay
(boydkane.com)

198 Comments

freetime2

5 hours ago

For a real world example of the challenges of harnessing LLMs, look at Apple. Over a year ago they had a big product launch focused on "Apple Intelligence" that was supposed to make heavy use of LLMs for agentic workflows. But all we've really gotten since then are a couple of minor tools for making emojis, summarizing notifications, and proof reading. And they even had to roll back the notification summaries for a while for being wildly "out of control". [1] And in this year's iPhone launch the AI marketing was toned down significantly.

I think Apple execs genuinely underestimated how difficult it would be to get LLMs to perform up to Apple's typical standards of polish and control.

[1] https://www.bbc.com/news/articles/cge93de21n0o

teeray

31 minutes ago

> minor tools for making emojis, summarizing notifications, and proof reading.

The notification / email summaries are so unbelievably useless too: it’s hardly more work to skim the notification / email that I do anyway.

SchemaLoad

9 minutes ago

Like most AI products it feels like they started with a solution first and went searching for the problems. Text messages being too long wasn't a real problem to begin with.

There are some good parts to Apple Intelligence though. I find the priority notifications feature works pretty well, and the photo cleanup tool works pretty well for small things like removing your finger from the corner of a photo, though it's not going to work on huge tasks like removing a whole person from a photo.

N_Lens

43 minutes ago

Apple’s typical standards of “polish and control” seem to be slipping drastically if MacOS Tahoe is anything to go by.

alfalfasprout

2 hours ago

> I think Apple execs genuinely underestimated how difficult it would be to get LLMs to perform up to Apple's typical standards of polish and control

Not only Apple, this is happening across the industry. Executives' expectations of what AI can deliver are massively inflated by Amodei et al. essentially promising human-level cognition with every release.

The reality is aside from coding assistants and chatbot interfaces (a la chatgpt) we've yet to see AI truly transform polished ecosystems like smartphones and OS' for a reason.

api

2 hours ago

Standard hype cycle. We are probably creating the top of the peak of inflated expectations.

__loam

4 hours ago

I'm happy they ate shit here because I like my mac not getting co-pilot bullshit forced into it, but apparently Apple had two separate teams competing against each other on this topic. Supposedly a lot of politics got in the way of delivering on a good product combined with the general difficulty of building LLM products.

Gigachad

2 hours ago

I do prefer that Apple is opting to have everything run on device so you aren’t being exposed to privacy risks or subscriptions. Even if it means their models won’t be as good as ones running on $30,000 GPUs.

gerdesj

an hour ago

On device.

If you have say 16GB of GPU RAM and around 64GB of RAM and a reasonable CPU then you can make decent use of LLMs. I'm not a Apple jockey but I think you normally have something like that available and so you will have a good time, provided you curb your expectations.

I'm not an expert but it seems that the jump from 16 to 32GB of GPU RAM is large in terms of what you can run and the sheer cost of the GPU!

If you have 32GB of local GPU RAM and gobs of RAM you can rub some pretty large models locally or lots of small ones for differing tasks.

I'm not too sure about your privacy/risk model but owning a modern phone is a really bad starter for 10! You have to decide what that means for you and that's your thing and your's alone.

alfalfasprout

2 hours ago

It also means that when the VC money runs dry, it's sustainable to run those models on-device vs. losing money running on those $$$$$ GPUs (or requiring consumers to opt for expensive subscriptions).

genghisjahn

2 hours ago

Apparently? From what? Where did this information come from that they had two competing teams?

alwa

an hour ago

I feel like I hear people referring to Wayne Ma’s reporting for The Information to that effect.

https://www.theinformation.com/articles/apple-fumbled-siris-...

> Distrust between the two groups got so bad that earlier this year one of Giannandrea’s deputies asked engineers to extensively document the development of a joint project so that if it failed, Federighi’s group couldn’t scapegoat the AI team.

> It didn’t help the relations between the groups when Federighi began amassing his own team of hundreds of machine-learning engineers that goes by the name Intelligent Systems and is run by one of Federighi’s top deputies, Sebastien Marineau-Mes.

AdieuToLogic

2 hours ago

I found this statement particularly relevant:

  While it’s possible to demonstrate the safety of an AI for 
  a specific test suite or a known threat, it’s impossible 
  for AI creators to definitively say their AI will never act 
  maliciously or dangerously for any prompt it could be given.
This possibility is compounded exponentially when MCP[0] is used.

0 - https://github.com/modelcontextprotocol

mrkmarron

an hour ago

That is a key item that we are working on addressing in the 2.0 milestone for Bosque[0]. The goal is to build a language and system model that allows us to reliably sandbox and support agents in constructing "Trustworthy-by-Construction AI Agents."

This addresses both accidental misbehaviors and targeted prompt (or other) attacks on agents -- for example ensuring that an agent will never misuse a payment API or return that an order succeeds without really placing it!

[0] -- https://bosquelanguage.github.io/2025/09/26/trustworthy-agen...

someothherguyy

2 minutes ago

why make a new language? are there no existing languages comprehensive enough for this?

worldsayshi

23 minutes ago

It sounds completely crazy that anyone would give an LLM access to a payment or order API without manual confirmation and "dumb" visualization. Does anyone actually do this?

nlawalker

7 hours ago

Where did "can't you just turn it off?" in the title come from? It doesn't appear anywhere in the actual title or the article, and I don't think it really aligns with its main assertions.

meonkeys

6 hours ago

It shows up at https://boydkane.com under the link "Why your boss isn't worried about advanced AI". Must be some kind of sub-heading, but not part of the actual article / blog post.

Presumably it's a phrase you might hear from a boss who sees AI as similar to (and as benign/known/deterministic as) most other software, per TFA

nlawalker

6 hours ago

Ah, thanks for that!

>Presumably it's a phrase you might hear from a boss who sees AI as similar to (and as benign/known/deterministic as) most other software, per TFA

Yeah I get that, but I think that given the content of the article, "can't you just fix the code?" or the like would have been a better fit.

AkelaA

2 hours ago

In my experience it’s usually the engineers that aren’t worried about AI, because they see the limitations clearly every time they use it. It’s pretty obvious that whole thing is severely overhyped and unreliable.

Your boss (or more likely, your bosses’ bosses’s boss) is the one deeply worried about it. Though mostly worried about being left behind by their competitors and how their company’s use of AI (or lack thereof) looks to shareholders.

Izkata

4 hours ago

It's a sci-fi thing, think of it along the lines of "What do you mean Skynet has gone rogue? Can't you just turn it off?"

(I think something along these lines was actually in the Terminator 3 movie, the one where Skynet goes live for the first time).

Agreed though, no relation to the actual post.

cantrevealname

3 hours ago

This sci-fi thing goes as far back as the 1983 movie WarGames, where they wanted to pull the plug on a rogue computer, but there was a reason you couldn’t do that:

McKittrick: General, the machine has locked us out. It's sending random numbers to the silos.

Pat Healy: Codes. To launch the missiles.

General Beringer: Just unplug the goddamn thing! Jesus Christ!

McKittrick: That won't work, General. It would interpret a shutdown as the destruction of NORAD. The computers in the silos would carry out their last instructions. They'd launch.

marssaxman

2 hours ago

Further than that, even - this trope appears in Colossus: The Forbin Project, released in 1970, where the rogue computer is buried underground with its own nuclear reactor, so it can't be powered off.

Gigachad

2 hours ago

In real life it won’t be that the computer prevents you from turning it off. It’ll be that the computer is guarded by cultists who think its god, and unstoppable market forces that require it to keep running.

cantrevealname

2 hours ago

When AI ends up running everything essential to survival and society, it’ll be preposterous to even suggest pulling the plug just because it does something bad.

Can you imagine the chaos of completely turning off GPS or Gmail today? Now imagine pulling the plug on something in the near future that controls all electric power distribution, banking communications, and Internet routing.

tehjoker

40 minutes ago

This is the case with capitalism today. I don't like where he took the philosophy, but Nick Land did have an insight that all the worst things we believe about AI (e.g. paperclip optimizing etc) are capitalism in a nutshell.

omnicognate

5 hours ago

It's a poor choice of phrase if the purpose is to illustrate a false equivalence. It applies to AI both as much (you can kill a process or stop a machine just the same regardless of whether it's running an LLM) and as little (you can't "turn off" Facebook any more than you can "turn off" ChatGPT) as it does to any other kind of software.

hackernewds

2 hours ago

The retitling now by HN appears more accurate

wmf

5 hours ago

Turning AI off comes up a lot in existential risk discussions so I was surprised the article isn't about that.

kazinator

7 hours ago

> AIs will get more reliable over time, like old software is more reliable than new software.

:)

Was that a humam Freudian slip, or artificial one?

Yes, old software is often more reliable than new.

kstrauser

6 hours ago

Holy survivorship bias, Batman.

If you think modern software is unreliable, let me introduce you to our friend, Rational Rose.

noir_lord

6 hours ago

Agreed.

Or debuggers that would take out the entire OS.

Or a bad driver crashing everything multiple times a week.

Or a misbehaving process not handing control back to the OS.

I grew up in the era of 8 and 16 bit micros and early PCs, they where hilariously less stable than modern machines while doing far less, there wasn’t some halcyon age of near perfect software, it’s always been a case of things been good enough to be good enough but at least operating systems did improve.

malfist

6 hours ago

Remember BSODs? Used to be a regular occurrence, now they're so infrequent they're gone from windows 11

wlesieutre

5 hours ago

And the "cooperative multitasking" in old operating systems where one program locking up meant the whole system was locked up

ponector

4 hours ago

I guess that is because you run it on old hardware. When I've bought my Asus ROG expensive laptop I had bsod almost daily. A year later with all updates I had bsod once in a month on the same device and windows installation.

krior

5 hours ago

Gone? I had two last year, lets not overstate things.

dylan604

an hour ago

Daily+ occurrences to two in a year pretty much rounds to zero. Kind of like we said measles were eradicated because there was <X per year cases.

rkomorn

4 hours ago

My anecdata is that my current PC is four years old, with the same OS install, and I can't even recall if I've seen one BSoD.

ClimaxGravely

5 hours ago

Still get them fairly regularly except now they come with a QR code.

spartanatreyu

3 hours ago

Depends, if you install games with anti-cheat they can often conflict and cause BSODs.

It's why I don't play the new trackmania.

dist-epoch

4 hours ago

Mostly because Microsoft shut down kernel access, wrote it's own generic drivers for "simple" devices (USBs, printers, sound cards, ...) and made "heavy" drivers submit to their WHQL quality control to be signed to run.

Podrod

4 hours ago

They're definitely not gone.

stiglitz

22 minutes ago

As a Windows driver developer: LOL

kazinator

5 hours ago

I remember Linux being remarkable reliable throughout its entire life in spite of being rabidly worked on.

Windows is only stabilizing because it's basically dead. All the activity is in the higher layers, where they are racking their brains on how to enshittify the experience, and extract value out of the remaining users.

yibg

34 minutes ago

Or just http without the s. We take it for granted now, but not even that long ago http was the standard.

Yoric

5 hours ago

I grew up in the same era and I recall crashes being less frequent.

There were plenty of other issues, including the fact that you had to adjust the right IRQ and DMA for your Sound Blaster manually, both physically and in each game, or that you needed to "optimize" memory usage, enable XMS or EMS or whatever it was at the time, or that you spent hours looking at the nice defrag/diskopt playing with your files, etc.

More generally, as you hint to, desktop operating systems were crap, but the software on top of it was much more comprehensively debugged. This was presumably a combination of two factors: you couldn't ship patches, so you had a strong incentive to debug it if you wanted to sell it, and software had way fewer features.

Come to think about it, early browsers kept crashing and taking down the entire OS, so maybe I'm looking at it with rosy glasses.

pezezin

2 hours ago

You are looking back with rosy glasses indeed.

Last year I assembled a retro PC (Pentium 2, Riva TNT 2 Ultra, Sound Blaster AWE64 Gold) running Windows 98 to remember my childhood, and it is more stable than what I remembered, but still way worse than modern systems. There are plenty of games that will refuse to work for whatever reason, or that will crash the whole OS, specially when existing, and require a hard reboot.

Oh and at least in the '90s you could already ship patches, we used to get them with the floppies and later CDs provided by magazines.

chipotle_coyote

3 hours ago

I know very little about Rational Rose, other than it always sounded like the stage name of a Vulcan stripper.

binarymax

5 hours ago

You know, I had spent a good amount of years not having even a single thought about rational rose, and now that’s all over.

kstrauser

4 hours ago

I do apologize. I couldn't bear this burden alone.

lossyalgo

2 hours ago

It could definitely be worse. I have the privilege of using it weekly :(

kstrauser

2 hours ago

What? How? I thought we stamped it out in the Purge of 2007.

cjbgkagh

5 hours ago

How much of that do you think would be attributable to IBM or Rational Software?

gridspy

2 hours ago

I think old in this sense is "released" rather than "beta" - it takes time to make any software reliable. Many of the examples here further prove that young software is unreliable.

Remember when debuggers were young?

Remember when OSes were young?

Remember when multi-tasking CPUs were young?

Etc...

sidewndr46

4 hours ago

Rational Rhapsody called and wants the crown back

kazinator

6 hours ago

At least that project was wise enough to use Lisp for storing its project files.

joomla199

7 hours ago

Neither, you’re reading it wrong. Think of it as codebases getting more reliable over time as they accumulate fixes and tests. (As opposed to, say, writing code in NodeJS versus C++)

giancarlostoro

7 hours ago

Age of Code does not automatically equal quality of code, ever. Good code is maintained by good developers. A lot of bad code is pushed out by management, and other situations, or just bad devs. This is a can of worms you're talking your way into.

LeifCarrotson

6 hours ago

You're using different words - the top comment only mentioned the reliability of the software, which is only tangentially related to the quality, goodness, or badness of the code used to write it.

Old software is typically more reliable, not because the developers were better or the software engineering targeted a higher reliability metric, but because it's been tested in the real world for years. Even more so if you consider a known bug to be "reliable" behavior: "Sure, it crashes when you enter an apostrophe in the name field, but everyone knows that, there's a sticky note taped to the receptionist's monitor so the new girl doesn't forget."

Maybe the new software has a more comprehensive automated testing framework - maybe it simply has tests, where the old software had none - but regardless of how accurate you make your mock objects, decades of end-to-end testing in the real world is hard to replace.

As an industrial controls engineer, when I walk up to a machine that's 30 years old but isn't working anymore, I'm looking for failed mechanical components. Some switch is worn out, a cable got crushed, a bearing is failing...it's not the code's fault. It's not even the CMOS battery failing and dropping memory this time, because we've had that problem 4 times already, we recognize it and have a procedure to prevent it happening again. The code didn't change spontaneously, it's solved the business problem for decades... Conversely, when I walk up to a newly commissioned machine that's only been on the floor for a month, the problem is probably something that hasn't ever been tried before and was missed in the test procedure.

freetime2

5 hours ago

Yup, I have worked on several legacy codebases, and a pretty common occurence is that a new team member will join and think they may have discovered a bug in the code. Sometimes they are even quite adamant that the code is complete garbage and could never have worked properly. Usually the conversation goes something like: "This code is heavily used in production, and hasn't been touched in 10 years. If it's broken, then why haven't we had any complaints from users?"

And more often than not the issue is a local configuration issue, bad test data, a misunderstanding of what the code is supposed to do, not being aware of some alternate execution path or other pre/post processing that is running, some known issue that we've decided not to fix for some reason, etc. (And of course sometimes we do actually discover a completely new bug, but it's rare).

To be clear, there are certainly code quality issues present that make modifications to the code costly and risky. But the code itself is quite reliable, as most bugs have been found and fixed over the years. And a lot of the messy bits in the code are actually important usability enhancements that get bolted on after the fact in response to real-world user feedback.

1313ed01

7 hours ago

Old code that has been maintained (bugfixed), but not messed with too much (i.e. major rewrites or new features) is almost certain to be better than most other code though?

DSMan195276

7 hours ago

"Bugfixes" doesn't mean the code actually got better, it just means someone attempted to fix a bug. I've seen plenty of people make code worse and more buggy by trying to fix a bug, and also plenty of old "maintained" code that still has tons of bugs because it started from the wrong foundation and everyone kept bolting on fixes around the bad part.

gridspy

an hour ago

One of frustrating truths about software is that it can be terrible and riddled with bugs but if you just keep patching enough bugs and use it the same way every time it eventually becomes reliable software ... as long as the user never does anything new and no-one pokes the source with a stick.

I much prefer the alternative where it's written in a manner where you can almost prove it's bug free by comprehensively unit testing the parts.

eptcyka

7 hours ago

I’ve read parts of macOS’ open source code that surely has been around for a while, maintained and absolute rubbish.

hatthew

6 hours ago

I think we all agree that the quality of the code itself goes down over time. I think the point that is being made is that the quality of the final product goes up over time.

E.g. you might fix a bug by adding a hacky workaround in the code; better product, worse code.

prasadjoglekar

7 hours ago

It actually might. Older code running in production is almost automatically regression tested with each new fix. It might not be pretty, but it's definitely more reliable for solving real problems.

shakna

6 hours ago

The list of bugs tagged regression at work certainly suggests it gets tested... But fixing those regressions...? That's a lot of dev time for things that don't really have time allocated for them.

kube-system

6 hours ago

The author didn't mean that an older commit date on a file makes code better.

The author is talking about the maturity of a project. Likewise, as AI technologies become more mature we will have more tools to use them in a safer and more reliable way.

giancarlostoro

2 hours ago

I've seen too many old projects that are not by any means better no matter how much they get updates because management define priorities. I'm not alone in saying I've been in a few projects where the backlog is rather large. When your development is driven by marketing people trying to pump up sales, all the "non critical" bugs begin to stack up.

kube-system

13 minutes ago

Absolutely. Which is why the author clearly meant "old code" as in mature. Not "old code" as in "created a long time ago".

izzydata

7 hours ago

Sounds more like survivorship bias. All the bad codebases were thrown out and only the good ones lasted a long time.

wvenable

6 hours ago

In my experience actively maintained but not heavily modified applications tend towards stability over time. It don't even matter if they are good or bad codebases -- even a bad code will become less buggy over time if someone is working on bug fixes.

New code is the source of new bugs. Whether that's an entirely new product, a new feature on an existing project, or refactoring.

james_marks

5 hours ago

I’ve always called this “Work Hardening”, as in, the software has been improved over time by real work being done with it.

jazzyjackson

3 hours ago

Ok, but metal that has been hardened is more prone to snapping once it loses its ductility

kazinator

6 hours ago

You mean think of it as opposite to what is written in the remark, and then find it funny?

Yes, I did that.

glitchc

6 hours ago

Perhaps better rephrased as "software that's been running for a (long) while is more reliable than software that only started running recently."

xutopia

7 hours ago

The most likely danger with AI is concentrated power, not that sentient AI will develop a dislike for us and use us as "batteries" like in the Matrix.

darth_avocado

7 hours ago

The reality is that the CEO/executive class already has developed a dislike for us and is trying to use us as “batteries” like in the Matrix.

vladms

6 hours ago

Do you know personally some CEO-s? I know a couple and they generally seem less empathic than the general population, so I don't think that like/dislike even applies.

On the other hand, trying to do something "new" is lots of headaches, so emotions are not always a plus. I could make a parallel to doctors: you don't want a doctor to start crying in a middle of an operation because he feels bad for you, but you can't let doctors doing everything that they want - there needs to be some checks on them.

darth_avocado

5 hours ago

I would say that the parallel is not at all accurate because the relationship between a doctor and a patient undergoing surgery is not the same as the one you and I have with CEOs. And a lot of good doctors have emotions and they use them to influence patient outcomes positively.

Ensorceled

4 minutes ago

Even then, a psychopathic doctor at least has their desired outcomes mostly aligned with the patients.

ljlolel

7 hours ago

CEOs (even most VCs) are labor too

toomuchtodo

7 hours ago

Labor competes for compensation, CEOs compete for status (above a certain enterprise size, admittedly). Show me a CEO willingly stepping down to be replaced by generative AI. Jamie Dimon will be so bold to say AI will bring about a 3 day week (because it grabs headlines [1]) but he isn't going to give up the status of running JPMC; it's all he has besides the wealth, which does not appear to be enough. The feeling of importance and exceptionalism is baked into the identity.

[1] https://fortune.com/article/jamie-dimon-jpmorgan-chase-ceo-a...

conception

6 hours ago

Spoiler there’s no reason we couldn’t work three days a week now. And 100 might be pushing it, but having life expectancy to 90 as well within our grass today as well. We have just decided not to do that.

Animats

6 hours ago

That's the market's job. Once AI CEOs start outperforming human CEOs, investment will flow to the winners. Give it 5-10 years.

(Has anyone tried an LLM on an in-basket test? [1] That's a basic test for managers.)

[1] https://en.wikipedia.org/wiki/In-basket_test

icedchai

5 hours ago

Almost everyone is "labor" to some extent. There is always a huge customer or major investor that you are beholden to. If you are independently wealthy then you are the exception.

darth_avocado

7 hours ago

Until shareholders treat them as such, they will remain in the ruling class

nancyminusone

7 hours ago

To me, the greatest threat is information pollution. Primary sources will be diluted so heavily in an ocean of generated trash that you might as well not even bother to look through any of it.

tobias3

5 hours ago

And it imitates all the unimportant bits perfectly (like spelling, grammar, word choice) while failing at the hard to verify important bits (truth, consistency, novelty)

Gigachad

2 hours ago

It’s already been happening but now it’s accelerated beyond belief. I saw a video about how WW1 reenactment photos end up getting reposted away from their original context and confused with original photos to the point it’s impossible to tell unless you can track it back to the source.

Now most of the photos online are just AI generated.

ben_w

5 hours ago

Concentrated power is kinda a pre-requisite for anything bad happening, so yes, it's more likely in exactly the same way that given this:

  Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
"Linda is a bank teller" is strictly more likely than "Linda is a bank teller and is active in the feminist movement" — all you have is P(a)>P(a&b), not what the probability of either statement is.

navane

4 hours ago

The power concentration is already massive, and a huge problem indeed. The ai is just a cherry on top. The ai is not the problem.

mrob

6 hours ago

Why does an AI need the ability to "dislike" to calculate that its goals are best accomplished without any living humans around to interfere? Superintelligence doesn't need emotions or consciousness to be dangerous.

Yoric

5 hours ago

It needs to optimize for something. Like/dislike is an anthropomorphization of the concept.

mrob

5 hours ago

It's an unhelpful one because it implies the danger is somehow the result of irrational or impulsive thought, and making the AI smarter will avoid it.

mmmore

4 hours ago

You can say that, and I might even agree, but many smart people disagree. Could you explain why you believe that? Have you read in detail the arguments of people who disagree with you?

preciousoo

7 hours ago

Seems like a self fulfilling prophecy

yoyohello13

7 hours ago

Definitely not ‘self’ fulfilling. There are plenty of people actively and vigorously working to fulfill that particular reality.

surgical_fire

6 hours ago

"AI will take over the world".

I hear that. Then I try to use AI for simple code task, writing unit tests for a class, very similar to other unit tests. If fails miserably. Forgets to add an annotation and enters in a death loop of bullshit code generation. Generates test classes that tests failed test classes that test failed test classes and so on. Fascinating to watch. I wonder how much CO2 it generated while frying some Nvidia GPU in an overpriced data center.

AI singularity may happen, but the Mother Brain will be a complete moron anyway.

alecbz

5 hours ago

Regularly trying to use LLMs to debug coding issues has convinced me that we're _nowhere_ close to the kind of AGI some are imagining is right around the corner.

ben_w

4 hours ago

Sure, but also the METR study showed the rate of change is t doubles every 7 months where t ~= «duration of human time needed to complete a task, such that SOTA AI can complete same with 50% success»: https://arxiv.org/pdf/2503.14499

I don't know how long that exponential will continue for, and I have my suspicions that it stops before week-long tasks, but that's the trend-line we're on.

Pulcinella

6 minutes ago

But will it actually get better or will it just get faster and more power efficient at failing to pair parentheses/braces/brackets/quotes?

surgical_fire

5 hours ago

At least Mother Brain will praise your prompt to generate yet another image in the style of Studio Ghibli as proof that your mind is a tour de force in creativity, and only a borderline genius would ask for such a thing.

worldsayshi

7 hours ago

> power resides where men believe it resides

And also where people believe that others believe it resides. Etc...

If we can find new ways to collectively renegotiate where we think power should reside we can break the cycle.

But we only have time to do this until people aren't a significant power factor anymore. But that's still quite some time away.

SkyBelow

6 hours ago

I agree.

Our best technology at current require teams of people to operate and entire legions to maintain. This leads to a sort of balance, one single person can never go too far down any path on their own unless they convince others to join/follow them. That doesn't make this a perfect guard, we've seen it go horribly wrong in the past, but, at least in theory, this provides a dampening factor. It requires a relatively large group to go far along any path, towards good or evil.

AI reduces this. How greatly it reduces this, if it reduces it to only a handful, to a single person, or even to 0 people (putting itself in charge), seems to not change the danger of this reduction.

fidotron

7 hours ago

I'm not so sure it will be that either, it would be having multiple AIs essentially at war with each other over access to GPUs/energy or whatever the materials are needed to grow if/when that happens. We will end up as pawns in this conflict.

ben_w

5 hours ago

Given that even fairly mediocre human intelligences can run countries into the ground and avoid being thrown out in the process, it's certainly possible for an AI to be in the intelligence range where it's smart enough to win vs humans but also dumb enough to turn us into pawns rather just go to space and blot out the sun with a Dyson swarm made from the planet Mercury.

But don't count on it.

I mean, apart from anything else, that's still a bad outcome.

alfalfasprout

2 hours ago

I mean, you can't really disprove either being an issue.

pcdevils

7 hours ago

For one thing, we'd make shit batteries.

antod

32 minutes ago

Sounds about right, most of us already are. But why would the AI need our shit? Surely it wants electricity?

noir_lord

6 hours ago

IIRC the original idea was that the machines used our brain capacity as a distributed array but then they decided batteries was easier to understand while been sillier, just burn the carbon they are feeding us, it’s more efficient.

CuriouslyC

an hour ago

If I could write the matrix reverted, Neo would discover that the last people put themselves in the pods because the world was so fucked up, and the machines had been caretakers that were trying to protect them from themselves. That revision would make the first movie perfect.

prometheus76

6 hours ago

They farm you for attention, not electricity. Attention (engagement time) is how they quantify "quality" so that it can be gamed with an algorithm.

drsupergud

7 hours ago

> bugs are usually caused by problems in the data used to train an AI

This also is a misunderstanding.

The LLM can be fine, the training and data can be fine, but because the LLMs we use are non-deterministic (at least in regard to their being intentional attempts at entropy to avoid always failing certain scenarios) current algorithms are inherently by-design not going to always answer every question correctly that it potentially could have if the values that fall within a range had been specific values for that scenario. You roll the dice on every answer.

coliveira

7 hours ago

This is not necessarily a problem. Any programming or mathematical question has several correct answers. The problem with LLMs is that they don't have a process to guarantee that a solution is correct. They will give a solution that seems correct under their heuristic reasoning, but they arrived at that result in a non-logical way. That's why LLMs generate so many bugs in software and in anything related to logical thinking.

vladms

6 hours ago

> Any programming or mathematical question has several correct answers.

Huh? If I need to sort the list of integer number of 3,1,2 in ascending order the only correct answer is 1,2,3. And there are multiple programming and mathematical questions with only one correct answer.

If you want to say "some programming and mathematical questions have several correct answers" that might hold.

Yoric

5 hours ago

"1, 2, 3" is a correct answer

"1 2 3" is another

"After sorting, we get `1, 2, 3`" yet another

etc.

At least, that's how I understood GP's comment.

naasking

5 hours ago

I think more charitably, they meant either that 1. There is often more than one way to arrive at any given answer, or 2. Many questions are ambiguous and so may have many different answers.

redblacktree

6 hours ago

What about multiple notational variations?

1, 2, 3

1,2,3

[1,2,3]

1 2 3

etc.

naasking

5 hours ago

> The problem with LLMs is that they don't have a process to guarantee that a solution is correct

Neither do we.

> They will give a solution that seems correct under their heuristic reasoning, but they arrived at that result in a non-logical way.

As do we, and so you can correctly reframe the issue as "there's a gap between the quality of AI heuristics and the quality of human heuristics". That the gap is still shrinking though.

tyg13

5 hours ago

I'll never doubt the ability of people like yourself to consistently mischaracterize human capabilities in order to make it seem like LLMs' flaws are just the same as (maybe even fewer than!) humans. There are still so many obvious errors (noticeable by just using Claude or ChatGPT to do some non-trivial task) that the average human would simply not make.

And no, just because you can imagine a human stupid enough to make the same mistake, doesn't mean that LLMs are somehow human in their flaws.

> the gap is still shrinking though

I can tell this human is fond of extrapolation. If the gap is getting smaller, surely soon it will be zero, right?

ben_w

4 hours ago

> doesn't mean that LLMs are somehow human in their flaws.

I don't believe anyone is suggesting that LLMs flaws are perfectly 1:1 aligned with human flaws, just that both do have flaws.

> If the gap is getting smaller, surely soon it will be zero, right?

The gap between y=x^2 and y=-x^2-1 gets closer for a bit, fails to ever become zero, then gets bigger.

The difference between any given human (or even all humans) and AI will never be zero: Some future AI that can only do what one or all of us can do, can be trivially glued to any of that other stuff where AI can already do better, like chess and go (and stuff simple computers can do better, like arithmetic).

naasking

4 hours ago

> I'll never doubt the ability of people like yourself to consistently mischaracterize human capabilities

Ditto for your mischaracterizations of LLMs.

> There are still so many obvious errors (noticeable by just using Claude or ChatGPT to do some non-trivial task) that the average human would simply not make.

Firstly, so what? LLMs also do things no human could do.

Secondly, they've learned from unimodal data sets which don't have the rich semantic content that humans are exposed to (not to mention born with due to evolution). Questions that cross modal boundaries are expected to be wrong.

> If the gap is getting smaller, surely soon it will be zero, right?

Quantify "soon".

hitarpetar

an hour ago

> That the gap is still shrinking though.

citation needed

mym1990

3 hours ago

Eh, proofs and logic have entered the room!

themanmaran

7 hours ago

> Because eventually we’ll iron out all the bugs so the AIs will get more reliable over time

Honestly this feels like a true statement to me. It's obviously a new technology, but so much of the "non-deterministic === unusable" HN sentiment seems to ignore the last two years where LLMs have become 10x as reliable as the initial models.

CobrastanJorji

7 hours ago

They have certainly gotten better, but it seems to me like the growth will be kind of logarithmic. I'd expect them to keep getting better quickly for a few more years and then kinda slow and eventually flatline as we reach the maximum for this sort of pattern matching kind of ML. And I expect that flat line will be well below the threshold needed for, say, a small software company to not require a programmer.

Legend2440

2 hours ago

Ahaha, of course nothing will ever be able to do my job!

criddell

7 hours ago

Right away my mind went to "well, are people more reliable than they used to be?" and I'm not sure they are.

Of course LLMs aren't people, but an AGI might behave like a person.

Yoric

5 hours ago

By the time a junior dev graduates to senior, I expect that they'll be more reliable. In fact, at the end of each project, I expect the junior dev to have grown more reliable.

LLMs don't learn from a project. At best, you learn how to better use the LLM.

They do have other benefits, of course, i.e. once you have trained one generation of Claude, you have as many instances as you need, something that isn't true with human beings. Whether that makes up for the lack of quality is an open question, which presumably depends on the projects.

tkgally

an hour ago

> LLMs don't learn from a project.

How long do you think that will remain true? I've bootstrapped some workflows with Claude Code where it writes a markdown file at the end of each session for its own reference in later sessions. It worked pretty well. I assume other people are developing similar memory systems that will be more useful and robust than anything I could hack together.

adastra22

6 hours ago

Older people are generally more reliable than younger people.

saulpw

3 hours ago

I'm not sure that's generally true. However, older people have a track record, and a reliable older person is likely to be more reliable than a younger person without such a track record.

artemisForge77

14 minutes ago

Even Apple found out that making AI work is harder than making emojis, maybe the hype train needs a reality check.

wrs

7 hours ago

My current method for trying to break through this misconception is informing people that nobody knows how AI works. Literally. Nobody knows. (Note that knowing how to make something is not the same as knowing how it works. Take humans as an obvious example.)

generic92034

5 hours ago

Nobody knows (full scope and on every level) how human brains work. Still bosses rely on their employees' brains all the time.

est

an hour ago

> In regular software, vulnerabilities are caused by mistakes in the lines of code that make up the software

> in modern AI systems, vulnerabilities or bugs are usually caused by problems in the data used to train an AI

In regular software, vulnerabilities are caused by lack of experience, therefor lack of proper training materials.

batch12

an hour ago

I think they're more caused by rushed deadlines, poor practices, and/or bad QA. Some folks just don't get it either and training doesn't help.

rivonVale3

19 minutes ago

Maybe AI isn't ready to take over the world yet, it still can't write a simple unit test without getting stuck in a loop.

CollinEMac

6 hours ago

> It’s entirely possible that some dangerous capability is hidden in ChatGPT, but nobody’s figured out the right prompt just yet.

This sounds a little dramatic. The capabilities of ChatGPT are known. It generates text and images. The qualities of the content of the generated text and images is not fully known.

kelvinjps10

6 hours ago

Think of the news about the kid who got recommended to suicide by ChatGPT, or chatgpt providing the user information on how to do illegal activities, these capabilities are the ones that the author it's referring to

kube-system

6 hours ago

And that sounds a little reductive. There's a lot that can be done with text and images. Some of the most influential people and organizations in the world wield their power with text and images.

luxuryballs

6 hours ago

Yeah, and to riff off the headline, if something dangerous is connected to and taking commands from ChatGPT then you better make sure there’s a way to turn it off.

alephnerd

6 hours ago

Also, there's a reason AI Red Teaming is now an ask that is getting line item funding from C-Suites.

Nasrudith

5 hours ago

Plus there is the 'monkeys with typewriters' problem with both danger and hypothetical good. In contrast, ChatGPT may technically reply to the right prompt with a universal cancer cure/vaccine. Psuedorandomly generating it wouldn't help as you wouldn't recognize it from all of the other queries of things we don't know of as true or false.

Likewise what to ask it for how to make some sort of horrific toxic chemical, nuclear bomb, or similar isn't much good if you cannot recognize it and dangerous capability depends heavily on what you have available to you. Any idiot can be dangerous with C4 and detonator or bleach and ammonia. Even if ChatGPT could give entirely accurate instructions on how to build an atomic bomb it wouldn't do much good because you wouldn't be able to source the tools and materials without setting off red flags.

smallnix

7 hours ago

> bad behaviour isn’t caused by any single bad piece of data, but by the combined effects of significant fractions of the dataset

Related opposing data point to this statement: https://news.ycombinator.com/item?id=45529587

buellerbueller

7 hours ago

"Signficiant fraction" does not imply (to this data scientist) a large fraction.

tptacek

7 hours ago

It would help if this piece was clearer about the context in which "AI bugs" reveal themselves. As an argument for why you shouldn't have LLMs making unsupervised real-time critical decisions, these points are all well taken. AI shouldn't be controlling the traffic lights in your town. We may never reach a point where it can. But among technologists, the major front on which these kinds of bugs are discussed is coding agents, and almost none of these points apply directly to coding agents: agent coding is (or should be) a supervised process.

mikkupikku

7 hours ago

I don't understand the "your boss" framing of this article, or more accurately, the title of this article. The article contents don't actually seem to have anything to do with management specifically. Is the reader is meant to believe that not being scared of AI is a characteristic of the managerial class? Is the unstated implication that there is some class warfare angle and anybody who isn't against AI is against laborers? Because what the article actually overtly argues, without any reading between the lines, is quite mundane.

freetime2

5 hours ago

> Is the unstated implication that there is some class warfare angle and anybody who isn't against AI is against laborers?

I didn't read it that way. I read "your boss" as basically meaning any non-technical person who may not understand the challenges of harnessing LLMs compared to traditional, (more) deterministic software development.

brookst

2 hours ago

The article doesn’t even mention prompting. Wha? Is it just talking about the ML foundations, not applications?

Animats

6 hours ago

Aim bosses at this article in The Economist.[1] If your boss doesn't read The Economist, you need to escalate to a level that does.

[1] https://www.economist.com/leaders/2025/09/25/how-to-stop-ais...

Traubenfuchs

6 hours ago

https://archive.is/R0RJB

Animats

6 hours ago

Management summary, from The Economist article:

"The worst effects of this flaw are reserved for those who create what is known as the “lethal trifecta”. If a company, eager to offer a powerful AI assistant to its employees, gives an LLM access to un-trusted data, the ability to read valuable secrets and the ability to communicate with the outside world at the same time, then trouble is sure to follow. And avoiding this is not just a matter for AI engineers. Ordinary users, too, need to learn how to use AI safely, because installing the wrong combination of apps can generate the trifecta accidentally."

andrewmutz

4 hours ago

Tremendous alpha right now in making scary posts about AI. Fear drives clicks. You don't even need to point to current problems, all you have to do is say we can't be sure they won't happen in the future.

nakamoto_damacy

3 hours ago

The said "don't use magic numbers" but LLMs are made almost entirely (by weight) of magic numbers...

skywhopper

5 hours ago

Not the point, but I’m confused by the Geoguessr screenshot. Under the reasoning for its decision, it mentions “traffic keeps to the left” but that is not apparent from the photo.

Then it says the shop sign looks like a “Latin alphabet business name rather than Spanish or Portuguese”. Uhhh… what? Spanish and Portuguese use the Latin alphabet.

marcosdumay

2 hours ago

It's an LLM.

It decided on the first line first (the place name), and then made the reasons on the rest of the text.

So the answer is more important on the justifications than the actual picture, and the reasoning that led it there doesn't enter the frame at all.

AlienRobot

2 hours ago

Am I correct to assume "modern AI system" means "neural network"?

avalys

4 hours ago

All the same criticisms are true about hiring humans. You don’t really know what they’re thinking, you don’t really know what their values and morals are, you can’t trust that they’ll never make a mistake, etc.

aloha2436

2 hours ago

I think you're misreading the article; the point here is not "LLMs are bad and can't replace humans," the point is that many non-technical people have the expectation that LLMs can replace humans _but still behave like regular software_ with regard to reliability and operability.

When a CEO sees their customer chatbot call a customer a slur, they don't see "oh my chatbot runs on a stochastic model of human language and OpenAI can't guarantee that it will behave in an acceptable way 100% of the time", they see "ChatGPT called my customer a slur, why did you program it to do that?"

tyg13

an hour ago

You can teach a human when they make a mistake. Can you do the same for an LLM?

excalibur

5 hours ago

> It’s entirely possible that some dangerous capability is hidden in ChatGPT, but nobody’s figured out the right prompt just yet.

Or they have, but chose to exploit or stockpile it rather than expose it.

bitwize

5 hours ago

Boss: You can just turn it off, can't you?

Me: Ask me later.

jongjong

6 hours ago

This article makes a solid case. The worst kinds of bugs in software are not the most obvious ones like syntax errors, they are the ones where the code appears to be working correctly, until some users do something slightly unusual after a few weeks of some code change being deployed and it breaks spectacularly but the bug only affects a small fraction of users so developers cannot reproduce the issue... And the cose change happened such time ago that the guilty code isn't even suspected.

fidotron

7 hours ago

But this is why using the AI in the production of (almost) deterministic systems makes so much sense, including saving on execution costs.

ISTR someone else round here observing how much more effective it is to ask these things to write short scripts that perform a task than doing the task themselves, and this is my experience as well.

If/when AI actually gets much better it will be the boss that has the problem. This is one of the things that baffles me about the managerial globalists - they don't seem to appreciate that a suitably advanced AI will point the finger at them for inefficiency much more so than at the plebs, for which it will have a use for quite a while.

hn_acc1

6 hours ago

A bunch of short scripts doesn't easily lead to a large-scale robust software platform.

I guess if managers get canned, it'll be just marketing types left?

pixl97

6 hours ago

>that baffles me about the managerial globalists

It's no different from those on HN that yell loudly that unions for programmers are the worst idea ever... "it will never be me" is all they can think, then they are protesting in the streets when it is them, but only after the hypocrisy of mocking those in the street protesting today.

hn_acc1

6 hours ago

Agreed. My dad was raised strongly fundamentalist, and in North America, that included (back then) strongly resisting unions. In hindsight, I've come to realize that my parent's weren't maybe even of average intelligence, and definitely of above-average gullibility.

Unionized software engineers would solve a lot of the "we always work 80 hour weeks for 2 months at the end of a release cycle" problems, the "you're too old, you're fired" issues, the "new hires seems to always make more than the 5/10+ year veterans", etc. Sure, you wouldn't have a few getting super rich, but it would also make it a lot easier for "unionized" action against companies like Meta, Google, Oracle, etc. Right now, the employers hold like 100x the power of the employees in tech. Just look at how much any kind of resistance to fascism has dwindled after FAANG had another round of layoffs..

fidotron

6 hours ago

Software "engineers" totally miss a key thing in other engineering professions as well, which is organizations to enforce some pretense of ethical standards to help push back against requests from product. Those orgs often look a lot like unions.

alganet

7 hours ago

> here are some example ideas that are perfectly true when applied to regular software

Hm, I'm listening, let's see.

> Software vulnerabilities are caused by mistakes in the code

That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on.

> Bugs in the code can be found by carefully analysing the code

Once again, not exactly true. Have you ever tried understanding concurrent code just by reading it? Some bugs in regular software hide in places that human minds cannot probe.

> Once a bug is fixed, it won’t come back again

Ok, I'm starting to feel this is a troll post. This guy can't be serious.

> If you give specifications beforehand, you can get software that meets those specifications

Have you read The Mythical Man-Month?

SalientBlue

7 hours ago

You should read the footnote marked [1] after "a note for technical folk" at the beginning of the article. He is very consciously making sweeping generalizations about how software works in order to make things intelligible to non-technical readers.

pavel_lishin

7 hours ago

But are those sweeping generalizations true?

> I’m also going to be making some sweeping statements about “how software works”, these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes.

I'd argue that this describes most software written since, uh, I hesitate to even commit to a decade here.

SalientBlue

6 hours ago

For the purposes of the article, which is to demonstrate how developing an LLM is completely different from developing traditional software, I'd say they are true enough. It's a CS 101 understanding of the software development lifecycle, which for non-technical readers is enough to get the point across. An accurate depiction of software development would only obscure the actual point for the lay reader.

hedora

6 hours ago

At least the 1950’s. That’s when stuff like asynchrony and interrupts were worked out. Dijkstra wrote at length about this in reference to writing code that could drive a teletype (which had fundamentally non-deterministic timings).

If you include analog computers, then there are some WWII targeting computers that definitely qualify (e.g., on aircraft carriers).

dkersten

7 hours ago

Sure, but:

> these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes

The claims the GP quoted DON’T mostly hold, they’re just plain wrong. At least the last two, anyway.

alganet

7 hours ago

Does that really matter?

He is trying to lax the general public perception around AIs shortcomings. He's giving AI a break, at the expense of regular developers.

This is wrong on two fronts:

First, because many people foresaw the AI shortcomings and warned about them. This "we can't fix a bug like in regular software" theatre hides the fact that we can design better benchmarks, or accountability frameworks. Again, lots of people foresaw this, and they were ignored.

Second, because it puts the strain on non-AI developers. It blamishes all the industry, putting together AI with non-AI in the same bucket, as if AI companies stumbled on this new thing and were not prepared for its problems, when the reality is that many people were anxious about the AI companies practices not being up to standard.

I think it's a disgraceful take, that only serves to sweep things under a carpet.

SalientBlue

6 hours ago

I don't think he's doing that at all. The article is pointing out to non-technical people how AI is different than traditional software. I'm not sure how you think it's giving AI a break, as it's pointing out that it is essentially impossible to reason about. And it's not at the expense of regular developers because it's showing how regular software development is different than this. It makes two buckets, and puts AI in one and non-AI in the other.

alganet

6 hours ago

He is. Maybe he's just running with the pack, but that doesn't matter either.

The fact is, we kind of know how to prevent problems in AI systems:

- Good benchmarks. People said several times that LLMs display erratic behavior that could be prevented. Instead of adjusting the benchmarks (which would slow down development), they ignored the issues.

- Accountability frameworks. Who is responsible when an AI fails? How the company responsible for the model is going to make up for it? That was a demand from the very beginning. There are no such accountability systems in place. It's a clown fiesta.

- Slowing down. If you have a buggy product, you don't scale it. First, you try to understand the problem. This was the opposite of what happened, and at the time, they lied that scaling would solve the issues (when in fact many people knew for a fact that scaling wouldn't solve shit).

Yes, it's kind of different. But it's a different we already know. Stop pushing this idea that this stuff is completely new.

SalientBlue

6 hours ago

>But it's a different we already know

'we' is the operative word here. 'We', meaning technical people who have followed this stuff for years. The target audience of this article are not part of this 'we' and this stuff IS completely new _for them_. The target audience are people who, when confronted with a problem with an LLM, think it is perfectly reasonable to just tell someone to 'look at the code' and 'fix the bug'. You are not the target audience and you are arguing something entirely different.

alganet

4 hours ago

Let's pretend I'm the audience, and imagine that in the past I said those things ("fix the bug" and "look at the code").

What should I say now? "AI works in mysterious ways"? Doesn't sound very useful.

Also, should I start parroting innacurate outdated generalizations about regular software?

The post doesn't teach anything useful for a beginner audience. It's bamboozling them. I am amazed that you used the audience perspective as a defense of some kind. It only made it worse.

Please, please, take a moment to digest my critique properly. Think about what you just said and what that implies. Re-read the thread if needed.

rester324

an hour ago

I thought this blog post was a parody. And to my surprise both the author and the audience takes it seriously. Weird

chasing0entropy

5 hours ago

70 years ago we were fascinated by the concept of converting analog to a perfect digital copy. In reality, that goal was a pipe drea!m and the closest we can ever get is a near identical facimile to which data fits... But it's still quite easy to determine digital from true analog with rudimentary means.

Human thought is analog. It is based on chemical reactions, time, and unpredictably (effectively) random physical characteristics. AI is an attempt to turn that which is purely digital into an rational analog thought equivalent.

No matter how much effort, money, power, and rare mineral eating TPUs will - ever - produce true analog data.

bcoates

5 hours ago

It's been closer to 100 years since we figured out information theory and discredited this idea (that continuous/analog processes have more, or different, information in them than discrete/digital ones)

largbae

5 hours ago

This is all true. But digital audio and video media has captured essentially all economic value outside of live performance. So it seems likely that we will find a "good enough" in this domain too.

chasing0entropy

2 hours ago

Interesting point with economic value extraction. The economy sacrificed accuracy and warmth of analog storage for convenience and security of digital storage. With economic incentive I am sure society will sacrifice accuracy and precision for the convenience of AI