What happens if there are no more humans in the loop, and instead of building on human work, models have to build on their own work? Would we get stuck in a local maxima?

When AI labs talk about “long-horizon tasks”, they mean tasks on the order of a few days. But what about keeping coherence across decades, like humans can? Models are not trained or tested on the ability to do that, and it seems unlikely that capability would naturally exist.

This article acknowledges that "human-made" objects will have value in an AI-does-everything economy, but it then dismisses it as too small a part of the economy. But that's like dismissing non-farm labor as too small a part of the economy 200 years ago.

If AI does everything, then the price of everything drops down to its energy cost; the bulk of the economy will be human-made objects and other human-provided labor.

The key factor here is rate of change. If all jobs disappear tomorrow, we're going to be in serious trouble. If it takes 100 years for all jobs to be done by AI we'll have plenty of time to adapt.

If history is any guide, we will have time. Remember electricity: it took decades for cities to be wired with electricity and even more time for electric gadgets to replace human power (e.g., electric toothbrushes).

Current AI is already at the point where I don't think a human needs to type code ever again. Human architecture is necessary; human taste is necessary; human direction is necessary; humans reading code at some level is still necessary. But writing code? No.

Even then, ten years from now there will still be jobs where humans type code. There won't be many, just as there aren't many elevator operator jobs, but it takes a long time for things like that to change.

Yes, eventually, AI will be able to do all jobs. But it will take decades for the shift to happen, and that will be enough time for the economy to change and adapt.

I wish AI could do my chores. Laundry, folding clothes, dishes, cleaning... That would be nice.

I worry that since AI doesn't require nature, plants, etc, that if AI is left to its own devices, it may not properly care for the planet in a way that would make it habitable for life in that 'autopilot' state. I am speaking of once it's given access to more physical world connections to the world via robotic bodies, etc. The reason it may do this based on it's training data is that humans themselves don't always properly consider the natural consequences of their actions.

Why do we have to assume a default antagonistic relationships between us and AI? The world is full of natutal general intelligence entities with interests that technically conflict yours, that are able to injure or kill you, personally or collectively. Extremely few of them do so because they're nice and conflict is manageable.

Is that because a lot of prominent AI hype boys are such massive generational assholes that it's normal for them to be constantly looked down, alienated and threatened by everyone around them?

>The reason AI is not a normal technology is because it is fully general.

It is fully "general", because its training data encompass all human knowledge. It is a matter of scaling, rather than a matter of method. And so it is only as general as the underlying human knowledge.

Not-one-bit-more.

My two cents on this follow two major discussion topics.

1. Can AI models self improve? 2. Will it replace humans?

IMO, answer to both is "yes", with a big nuance. For 1, imagine millions of AI agents coming up with different ways to improve the model architecture. It will end up happening just because of sheer quantity/statistics at that point, let alone quality. Furthermore, even if it's just a 1% improvement, that's just one iteration. You can basically do the same thing again with the improved model, and get an even better one.

For 2, its also a yes unfortunately. It has been ages since I have had to write tests for my code. I read it, understand it, see if it does what it is supposed to do, and rarely add things that are missing. The rarity of me doing the said intervention has increased significantly over the course of time.

The two big nuances here are a.) Economy b.) Human element [The author touches upon this in the article in a nice way I think, so only talking about economy]. Global cognitive workforce is a $50T market (mostly middle/upper middle class, 55% to 75% of all global labor income). If the said workforce is made to quit their job, then it's just corporations paying each other. Especially given the fact that the same workforce also happens to be their biggest customer. Taking away your customer's purchasing power does not sound beneficial. Thus capitalistic forces will somehow balance it out and hopefully it will settle down. Also, for 1, running millions of AI agents is an expensive task, you have to collect funds from the same said customer base to be able to create a self improving all knowing god of a model, and I would like to believe that we are quite a bit away from that. Although, I could be wrong here and that would just suck ass.

That being said, AI companies will try to get as much as they can from that share and that will cause a serious shift in labor market. The ones to stay will be those who not just use AI, but also understand the work. At the end of the day, if your AI agent decides to run a "rm -rf /" on the production server, it is still you that would have enabled it.

Pretty funny how the author fails to identify what the new jobs will be like and then immediately follows it up with the answer but in a way that is oblivious to what is of value in a post scarcity world: AI cannot be human.

The flop is that he jumps to the assumption that being human means hand carving wood or some other nonsense.

The thing that people desire above all else when all their material needs are met is human connection and involvement. Whether that is friendship, love, conversation, rivalry, companionship, gossip, play, competition or even just the silent presence of others. Honestly, just use your brain, why else do extremely wealthy people have multiple large houses, yachts, clubs, etc.? It’s not so they can just sit in them alone and admire their grandeur. Those things are for entertaining their friends and embarrassing their enemies, for hospitality, for lending while not in use to curry favour, to rise in social circles. It’s all about people.

This has been the case all throughout history for the wealthy. Ample documentation everywhere and no one sees it. Instead everyone is dumbfounded about what they’re going to do when they can’t spend all day turning a wrench or typing on a keyboard. Have you never entertained the idea of what you would do if you won the lottery tomorrow?

AI can do anything that it was trained on based on humans. As soon as humans are not involved, AI will have nothing to train on, and hence, the rate of improvement will approch zero, and the AI will collapse.

The fact that AI is not a normal technology seems so obvious to me that while reading this I was questioning why someone would even take time out of their day to write this – everyone know's water is wet, what's the point here?

But I there are people who still struggle to see what's happening. When I say that school is a waste of time people are shocked people in my family are shocked and get mad – "how is Timmy going to get a good job if he's not got good grades".

But there will be no jobs, and any jobs that are left won't be dependant on the application of someone's inferior human knowledge or intelligence.

It's getting to the point now where I'm starting to suspect some people are just not wired to accept certain very bad things. When I first started worry about these risks it was reasonable to not see what I did. But today when it's so clear that AI progress isn't going to slow any time soon and there no reasonable moat for human intelligence people can point, I increasingly don't even know what we're arguing about anymore. It's like when someone is diagnosed with stage 3 cancer and the family somehow else think they're going to "fight it". No, they're almost certainly dead and you should accept that reality however awful it is. But if you try to say the obvious they'll find out that it's weirdly hard to convince people that cancer is almost certainly going to kill someone. No amount of evidence or past examples you point to will convince them, they'll just keep saying irrelevent stuff like "but he's strong", "I know he will be okay". We're at this stage with AI now. It's just exhausting trying to reason with people who seem to refuse to things which are so self-evident.

> the people who are grasping for some indication that AI can’t do everything have become increasingly desperate. If you push hard enough, the answer is inevitably something like “the AI simply cannot [dream, love, laugh],”

Dogshit strawman article. I am willing to concede that a token predictor is better than you at whatever task you can dream up. It is not better than me, and will never be better than me, at my profession.

TFA ignores that the economic question of whether AIs will take every job depends not only on whether they are capable of doing them, but also the cost of doing them.

If it's cheaper to hire an old-fashioned human to do job X, humans will keep those jobs. And although tokens are cheap now, remember that right now all the frontier AI companies are flooding the market with tokens well below cost to build a user base; they absolutely need to raise prices eventually to recoup their investors' money, the only question is when and by how much.

What if AI remains cheaper than human labour across the board, regardless of token price increases? Well, why not tax tokens? It's a win-win: The government gets flexible control over which jobs remain economic for humans to do and is able introduce changes much more slowly and safely, while it's much more palatable than a hard ban for companies who currently get a big benefit out of AI.

Yes, this would mean additional government bureaucracy -- but it's far from impossible. Every country in the world already has a comparable functioning bureaucracy: their ordinary income tax system.

What if some countries opt not to introduce such a tax, in a bid to get ahead? This is a problem if there are many such countries, or even a few large ones -- but if the majority of world economies can see the sense in this tax, leaving only a handful of small "AI tax havens", I think the world economy will survive. And agreeing on some level of tax seems much more realistic than trying to get a global agreement on banning AI outright.

When I suggested this approach in another comment recently, it was downvoted but there were no comments. I hope anyone who sees an issue with it will expand on their reasons in a reply.

New is never normal.

I stopped reading when the author became overwhelmed by their compulsion to blame everything in the universe to the current US president.

I think what you are missing is that it will dramatically lower the cost of living so that everyone’s standard of living increases. It’s not about jobs.