Bringing elitism into it for what benefit? Its more like were giving away the ability to drive our own destiny.

I can see humanity walking to its collective doom with a sense of satisfaction, like "at least/last it is happening to those other guys too".

So what? You can't see the difference between "the average person doesn't understand it" and "no one understands it"?

I am failing to square this techno-utopia with the rising hate and authoritarianism worldwide.

It seems far more likely that AI will be working in the service of a government to oppress change and violate any human rights that are standing in the way.

Keep what’s happening today in your mind as you consider the future.

> Those "demons" didn't eat our lunch?

Tell that to the Europeans who were so afraid of naming the bear that its original name is still lost to us :) The Luddites also had a lot to say about the mechanical demons that were eating their lunches.

> Cognition is in the process of taking the leap from glacially slow biology under blind evolution, to lightening fast (scales of software in days, hardware in months) engineered self-optimization.

Yes-but, time scales are relevant only relative to other time scales (or the Planck time, if we get there). If the whole universe were to start running at 1,000,000× speed, you wouldn't notice. Speed is important to us humans currently because of our individual perspective and associated limited lifespan, but not in a universal sense (modulo some big hurdles like star lifetime and heat death). There are already self-replicating organisms with much faster iteration cycles than humans or human technology, like bacteria.

> Within 10 years, the robotics side will catch up with the speed of the software side. Biological cognition, bodies and economic status are all getting revolutionized within the next 10-15 years.

Just like it was in the 1980s, or the 1800s? I don't claim that these things _won't_ happen in the next ten years, but looking at it empirically people thinking that they will coïncides with every new technology and is a poor predictor.

> If there is reason in the short term to be discouraged, it is due to ourselves. But pushing for more ethical systems, hard, right now, is a better and right response.

I don't disagree, but it's not like we haven't been trying. Attempts to codify ethics go back about as far as human civilization, and it's not obvious to me that there's any faster road, and certainly not one fast enough to outpace our current rate of technological development (before you even start talking about AGI!).

Colonizing the solar system for resources is something that's said by people who don't understand astronomy or space technology (like Elon Musk).

On average is much different then nobody understanding.

> There is so much argument that boils down to "this would be very bad for me, therefore it can't happen".

I would gently suggest that you may be assigning the most uncharitable interpretation to people's honest attempts at having an intellectual discussion around these topics.

Being obtuse comes effortlessly to me, I'm afraid. My comment pertains specifically to the kind mathematics that has no applications as a rebuttal to what I read as the author's main claim it will inevitably decline as a human endeavor. Saying that, I believe that AI has great potential to advance engineering.

Furthermore, any attempt to pretend otherwise is an appeal to the idea we are in literal fact haunted by our own internal demons that are inexplicable to science or mathematics.

One of the effects of this entire debate is it's exposed how many people claiming to be pure mechanistic rationalists actually never were. At least those of an openly religious inclination have been sticking to their position consistently.

> AIs will become better at explaining math before they become better at doing math

I am not so sure. I am talking about math with AI and I feel as a math teacher, it is terrible. It introduces new notation and concepts faster than I can ask for definitions. It doesn't provide many examples. Sometimes it realizes it made a mistake in a guess and the confusion compounds.

Maybe it's just my (in)experience - but I would welcome a practical advice on how it could explain math to me better.

> I still have some hope in the last one "mathematicians and the mathematically-inclined will still do math on their own". Maybe some limited communities of this sort will form.

I think it’s pretty likely. Contrary to all the negative things said about mathematicians in this thread, in my experience they’re mostly not profit-driven (for the ones that are tend to filter out of academia during grad school, it’s a high-effort, low-reward environment).

Tao does not seem particularly threatened by AIs doing math, and I trust his judgment.

> Taking arguments seriously doesn't necessitate agreement.

I agree. That’s what I’m saying, so I don’t think I understand what you mean. Mind clarifying the point you’re making?

To try to clarify, what I do mean that is that disagreement does necessitate at least some measure of taking arguments seriously. Some arguments maybe don’t deserve to be taken seriously, but you need to first be open to taking them seriously so you can decide.

In other words, if you go into an argument with the initial mindset of taking it seriously, by the end you’ll be able to decide if it should be taken seriously or not; but if your initial mindset is to not take it seriously, most likely you won’t be open to understand the nuance of what is being said. Thus, the former leads to productive discussion while the latter does not.