> The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.

I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.

If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?

Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.

I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.

> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.

I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.

However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.

AI has an extremely high capital barrier to research and development. Getting that initial capital required promises of a world-changing technology. A few companies with lots of investment were able to develop the technology and roll their own models. The business case for this R&D was that providing the models to paying users would amortize the R&D costs. But, promising disruption was the first step to getting the money.

Then the Chinese came along and showed that you can take those model weights, repackage them as open weights, and undercut the business model. Suddenly Dario et al. start talking about how dangerous AI is and how it needs to be regulated right now, before someone's abliterated open-weight Chinese-provided business-undercutting model destroys the world. They want their walled gardens regulated into being the only game available.

Both stages of the game—raising capital and regulatory capture—require hyping up the disruptive aspect of AI.

> Had LLMs first been presented as a "better IntelliSense"... very few people would have been upset.

Presenting AI as something it isn't doesn't change the reality which is like the Wait But Why thing https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

That was written in 2015 before LLMs were much of a thing and has been fairly spot on so far. This is not just about LLMs any more than the industrial revolution was just about steam engines.

The thing I keep thinking about with this sort of behavior is how Zoom became big in video conferencing. They had good features, yes, and COVID obviously gave every player in that space a big opportunity, but Zoom distinguished themselves in great part by doing every underhanded shady thing they could to make their app 'frictionless', privacy-invading, data-grabbing, and generally preyed on end users' lack of awareness. They did not keep doing all of those things, as far as I know – they got called out a lot but I have steered clear of them since my initial experience with them so I don't know how much they cleaned up their act. But taking the most aggressive and least ethical approach certainly seemed to have helped them get to the top at a very competitive time, and once there they could probably afford to stop doing the most egregious things.

It seems obvious that this same dynamic is in play with the frontier AI companies now. They know if they win the race to the they'll be forgiven their sins, or anyway they'll be beyond the reach of effective punishment.

There are a huge number of people involved in AI outside of the major AI companies both in smaller AI companies and lots of independent developers. They aren't having much success bringing AI to the masses and this is because they don't care enough about money and they aren't selling their ideas aggressively enough. There is this odd tendency to undersell achievements because they weren't motivated by the Right Stuff.

The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.

> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations

Aside from the AI dungeon app, this was my first real exposure to LLMs wrt github's copilot extension. It sucked, and did a lot of stuff wrong, but they got exactly 5 bucks from me to try it.

It's sorta crazy to think how many years ago that was now.

> These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong

They certainly don't care when a model hallucinates or an agent does something it shouldn't be doing. It's your fault and they charge you for the tokens.

It’s more than that. Scaling LLMs had a fundamental problem - that it took $10b+ of investment to become useful. To get over this huge activation hump, you had to hype the thing as world changing. If you sold it as autocomplete you would never raise the money to do even autocomplete.

That’s why the AI heads all have extremely dramatic narratives. They are the end product of a selection tournament for raising massive amounts of money pre-product.

>Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.

You make it sound like this was a feasible option. This technology would exist regardless of how any one company decides to frame it. If OpenAI decided to go with a more conservative approach rather than just throwing ChatGPT out for the masses, then all it would take is for one person to realize they didn't take full advantage of this technology to just make their own ChatGPT.

>The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.

You can criticize the current tech leaders all you want, but what you're describing is a fundamental dynamic of capitalism. You must choose the most profitable option, otherwise you have chosen the wrong option. If the current tech billionaires all vanished just as ChatGPT was released, they'd be quickly and easily replaced with roughly equivalent individuals who would effectively operate the same exact way. That's true because they're operating based on the mostly deterministic policy of profit maximization. They don't really get a say in it; they must follow the path towards maximizing expected future returns, otherwise they'll get canned by the investors who are seeking maximized expected future returns.

None of this relieves them of their personal ethical responsibilities either. I'm just pointing out that your comment might as well be describing an alternate reality where everyone lives peacefully with each other and nobody goes without what society can afford to offer them. It's equally as unrealistic.

better intellisense is how it started for me at least.

The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.

If it weren’t OAI and Anthropic, it would be others.

If it weren’t US companies, it would be others.

It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).

Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.

I think this recent post about a 2017 AI safety risk review may give some solid historical context that many of the leaders do not want to be the next Oppenheimer and have taken steps overtime to prevent being in that role themselves personally and collaborating with others so no one is: https://news.ycombinator.com/item?id=50366387

The only leader I’m not aware of really taking a more active role in safety on the daily is Elon Musk, so I’d like to see evidence supporting what he is doing or not doing on that front if someone from SpaceX can share their experience. The evidence on the environmental front from SpaceX is very concerning and is disregarding the wellbeing of the world and local populations in my opinion.

If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money.

Can someone who thinks this about Altman explain to me why he has repeatedly declined to take any equity in OpenAI? I genuinely want to understand your theory of mind here. Do you think he's actually in it for the power, and for some reason thought taking equity would undermine his effort at accumulating power? It just makes no sense to me.

I enthusiastically agree. I've been on this ride since GPT 2 and 3, I did start out looking at the technology in a neutral or positive light. I did not think the stewards of this technology would be so irresponsible, its consequences would rival the Stalinist purges.

No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.

I’m sorry, but the AI workers are as much to blame.

Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?

That HN post was infamous and nearly took down this site.

An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.

We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.

Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.

However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.

I agree with your premise, but not your conclusion. The problem is not the scope or scale of things, but treating them in an absurdly stupid fashion. Why are power systems internet connected? It certainly enables some greater degree of power and flexibility, but it exposes them to potentially catastrophic outcomes in case of malfeasance. So what happened at some point is somebody convinced somebody else that the security was super duper ultra good, and that's simply not true.

And I'd go one step further to say that it probably can not be true. This has always been true and probably will always be true. All LLMs are doing is demonstrating that. And the LLMs of tomorrow will demonstrate that the most LLM hardened system of today is still just a vulnerable mess. So the conclusion is simple - things should not be connected to the internet unless the worst case scenario from hacking/exploiting is relatively low. If somebody wants to connect their coffee maker to the internet, for one can only imagine whatever reason, then the worst case scenario is a broken coffee pot - who cares, go for it. But connecting e.g. power, water, or other core infrastructure to the internet? No, never.

Just because something hasn't been proven to be a problem today does not mean it won't be a problem tomorrow. We shouldn't learned that yesterday, hopefully we can learn it today.

You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.

Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.

Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.

I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.

I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.

I’m old enough, and he definitely is, to remember when having something connected to a network was a considerable security consideration

Now supermarket fridges are openly accessible over the Internet

I’m not sure it’s AI that’s the issue, it’s just a scapegoat for decades of bad security choices that nation state bad actors could have always exploited

Now that’s a perceived threat, ‘AI will do bad things’ is a great excuse for awful national infrastructure and security management

To me this sounds a bit like "a zombie apocalypse wouldn't be such a bad thing, because natural evolution would lead to humans becoming faster runners" line of reasoning.

The one thing worse than security by obscurity is probably security by fiat.

I would posit that systems being hacked by AI were already vulnerable to the huge caches of secret zero days held by intelligence agencies and organized crime around the world. Which we know exist.

Insecure is insecure. These systems never were secure.

If you are stuck on censorship you are just way outside the window of what is actually of concern here...

I hear you and i find it devastating, too. On top of that, the negative effects on the planet for having this technology vomit this soul-crushingly pointless crap is just depressing.

AI exposes the Sturgeon's law (90% of everything is crap), and amplifies it much more, and anything good is drowned out. Most of human-generated popular music is slop. Most of human generated code is buggy, unmaintainable and full of debt. Social media and youtube have always been dominated by click-bait and mediocre time-wasting crap.

Before AI I was already being smothered by soul crushing pointless junk made by humans.

You’re placing blame on the wrong spot. It’s algorithms and social media incentive systems not AI. Our digital media incentives addictive minimum effort chum and slop. AI just makes it easier to make.

Ok, but Hinton is worried about what happens if the tech keeps getting better at the current pace.

Let's flip your last point: if the tech gets good enough to replace people at any task (including physical ones), do you think others will need you for much?

And then there's the systems that the corporations are making more powerful as fast as they can. Yes, they promise to do things like cure cancer, but they're also capable of scarier things like hacking banks. And they're progressing faster than we can understand them. Would it be so bad to delay the cancer cures by a couple years to ensure we still understand what the models are doing?

That's good. And I'm glad for you. But honestly, you're an outlier, or at the very least on the long tail of the distribution.

I would say: Like all things AI, very bad in the short term, but very good in the long term. Unfortunately, it's us who have to live through the short term.

I guess we could look at it as going through a world-wide Code Red...

> What's risk for people valued close to or over trillion dollars?

Mortality.

That's weird, they must have cut off the part where he said Sundar.

Yeah, seriously. AI has been the single biggest technological improvement to my life in the last 20 years. If I could choose between smartphones and AI, I'll take AI any time of the day. Smartphones (in combination with social media) had a worse impact on society than AI has had and are not worth their advantages to me.

Yeah, it's single-handedly one of the best inventions this field has ever had.

Can't agree more - some of us are just happily achieving our life goals in record time thanks to AI!

The people in wall-e were in AI heaven.

Having a machine write code for you isn't programming, any more than hiring devs to write software for your business was programming.

"NUH UH!"

Thank you, brave HN commenter. You've truly made the site a better place with your astute rebuttal.

I suppose, but precious metals and cryptocurriencies would still exist, and would probably be more immune to power struggles and hacking than stocks.

Not to say that they're unhackable or unstealable, but merely that they dont have a central weak point that if broken, destroys the whole system

You mean like OpenAI training their next model and not monitoring it?

Or some other government setting an AI to intentionally hack and crash the US?

>possession of the most critical infrastructures on earth.

I think you may underestimate how poorly we protect these things. They have all the same security flaws as our internet connected systems. They aren't written in some super special language, it's all C, C++, JS and other non-memory protected trash.

Said AI "experts" for the last 10 years.

i mean personally, i enjoy not having smallpox

Cooking food, surviving winter ..

>predicting the likely outcomes

And if all the likely outcomes are very bad, then what?

> It could help cure diseases, end boring work and give us more free time.

Seriously? It seems to be causing more problems, giving us more boring work, and less free time.... sounds like absolute bullshit.

You’re a cultist

I'm not sure of this. I remember back in the day finding people's blogs, reading old forum threads, participating in said forums - there was a ton of info out there and easily found. Now those kinds of sites are rare, and those that still exist aren't surfaced by search engines like they used to be.

AI has the benefit of having trained on all that material, it's a snapshot in a sense. But as people increasingly stop creating content like that and we turn to AI instead of asking for help on forums we stop growing the knowledgebase that AI draws from.

Knowledge was already extremely accessible in the age of the Internet, but it required time to read and learn. AI is making that same knowledge applicable without the prerequisite of learning.

Most people are not interesting in learning for learning's sake, they are motivated to learn something they can apply (see: kids complaining about how learning Math in school doesn't transfer to useful skills). Why learn about architecture and technical debt, if you can tell the machine to make it faster?

The internet pre-AI did far more to make knowledge accessible than AI has done. Its useful, but its not the critical thing.

Because it’s in Ukraine? Same reason we spell the capital Kyiv now.

Indeed, I spotted that too and thought it was interesting. Biblical almost.

So pumping CO2 into the atmosphere is a good thing and if someone does a backflip by recognising their mistakes that is a bad thing?

If the resulting issues are correct it doesn't matter whether the field left the guy behind or not.

He would still be right and the field practioners would be wrong.

I actually laughed out loud.

I do find capsules interesting though.

Clickbait, including the photos taken of him with various facial expressions. It all seems so trite and staged.

I would rather see the onus on the company owners to be responsible for what they sell. Serious penalties for seriously bad products.

Absolutely! And software engineer should be an hereditary title as well.

In East Asian countries, certification systems are still maintained, but in reality they're practically meaningless. If a certification system exists, people will take an interest in CS and work harder at it, or so you'd think—but in reality, the exam questions end up stuck in the old style. In other words, there's a big gap between the exams and the actual field.

Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.

The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.

But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good

>so far I’m happy we haven’t had another world war

The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.

I would serious debate that it's the threat of nuclear MAD that has prevented WW3.

Rather, after WW2 the idea of millions of people dying in a war became anathema -- countries would simply revolt against the idea.

Also, an estimated 20-40 million people have died in wars since WW2 (depending on how you do the counting), so it's not like wars have stopped.

The ghost of Douglas Adams has entered the chat

9 minutes. Was that enough time to read the article?

Why did he start a research direction that 50 years later "might turn out" to not be a good idea?

That's a relatively silly question, don't you think?