A metaphor I'm constantly drawn to is the transition from agrarian to urban societies following the Industrial Revolution. Somebody who somehow saw the Industrial Revolution coming from the perspective of somebody living in an agrarian society might have envisioned it leading to 'super farms.' And it did.

But the biggest change wasn't what it did to farming, but enabling people and societies to start doing much more than just farming, as well as enabling some great social change as well by simply economically obsoleting slave labor. And trying to imagine all of the implications of this, as well as much society might look like, from the perspective of somebody living in an agrarian society would probably have been simply impossible.

I think people keep ignoring this possibility for things that LLMs will change. There's a vast amount of the 'cognitive economy' that LLMs stand to be able to automate. And I think that will open up a vacuum in society for people to build on top of what LLMs will do (and already are doing). I don't know what that means exactly, but that's because we still live in that 'agrarian society' and trying to imagine what things will look like after the 'Industrial Revolution' is probably just impossible.

> If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail

But suppose some future holy grail AI can do much more than that.

Suppose it could find a cure for cancer, fix the climate, build fusion plants, Dyson spheres and so on.

But nobody can understand anymore how any of it works. We just ask and then trust the AI to deliver (as it always has).

Isn't it fun to imagine how life would look like in that scenario?

We would probably no longer care about code, engineering or even physics and mathematics among other things. We would probably mainly care about

>Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say.

I think you have a fundamental misunderstanding here, and it's not really explained because I think it seems self-evident from within the field. In short: writing code is a means to an end; doing mathematics research is not, but is the end in itself.

The human involvement is crucial because the entire purpose of mathematics research is to increase human understanding of mathematics. It is pursued because it is interesting, not because it is economically useful. In this sense it's a lot closer to the humanities.

A black box oracle that just tells you whether statements are true or false is not the goal of mathematics and would not be particularly interesting to the field (except insofar as it could be harnessed to improve human understanding).

Coding is totally different from this, where it is essentially always done as a means to an end. Likewise with many other fields, like pharmaceutical research or materials science or what have you, that are oriented around solving problems for some practical purpose. Pure math isn't really like that for the most part.

Nit, but Terrence Tao did not write this article, it’s a guest post.

I feel you. We take a risk when we check our work less and we own the consequences. That's not new, and it doesn't make us bad. It's just the human condition from time immemorial. The solution is not obsessively combing through every detail of our work, it's better monitoring and control on the places where mistakes can have an impact.

> Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing.

If you are a normal person research (e.g. https://arxiv.org/html/2606.22721v1 but there are a lot more, not necessarily on coding) has shown that you indeed are being less careful. It most likely also works better simply because more resources are being poured in.

> What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?

For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.

Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.

Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.

How crazy will the world be once our [current gen] ANN are like that!

What an amazing thought.

LLMs dont create anything new, if programmers stop reading the code technology will be forever frozen to 2022, no new programming languages, operating systems, concurrency primitives, databases, networking protocols, UI frameworks everything will be based on the training data and future generations will forget about all the primitives we now take for granted.

If someone creates a new programming language/ framework or new better way to do async or whatever, no one will use it because it is not in the training data and it wont take off because everyone is using LLMs. It will be like using the same Lego pieces over and over.

Very few people scrutinise assembly in 2026 as compiler generated code is 'good enough'. LLMs are beginning to do the same with higher level languages.

Without bashing anyone in particular, a certain OS-vendor's desktop apps, have been 'good enough' to ship, but with p*ss-poor performance in many cases for the last decade or so. We crossed the 'good enough' Rubicon a few years back in terms of what end users receive as a finished app.

Hopefully LLMs will eventually bridge that last gap of efficiency when generating higher-level code that not only works, but is efficient. Maybe there's a future where they generate the final binary without even invoking a compiler.

> I guess, the possibility of reclaiming human meaning and purpose.

As you described very well, as humans we are mostly interested in solutions, not problems. You don't have to understand how a car works to make the most of it. Increasingly, you don't have to review every line of code to feel confident it is correct. But there is inherent value in understanding the problem. The effort it takes provides a surface area for growth, perhaps the only one that is actually available to us.

The solution provider also holds the locus of control, and it is only balanced when there are other available solution providers. We certainly want some of those to be human.

In my experience, LLMs are becoming very good at executing, but not a creating novel ideas or being creative.

Most of programming is reusing existing ideas in new shapes to solve new problems, but all the building blocks are there in the training set. Or new blocks can (easily) be derived from existing ones.

Math is different, it requires quite a bit of creativity, it's not just 'reuse all existing blocks'.

For the moment LLMs are good at discovering things that we overlooked in maths, or apply cleverly existing math blocks to make new results, but making a new theory that is really useful is out of reach for the moment in my opinion.

The problem with a terawatt fusion plant isn’t that the first one will be broken, or the tenth one in some other way. The problem is the hundredth will work flawlessly, and so will the thousandth, and a petawatt is serious waste heat to reject; if we keep building them on this planet, we’ll all simply cook.

Reliable cheap fusion is the holy grail and used in moderation will fix most of our environmental and political problems, but it also forces humanity off this world. Maybe that’s not a bad thing, but there is no free lunch.

it's incredible the amount of people who think all these recent posts in Taos blog were written by him.

You oppose correctness to meaning and purpose, which you seem to imply are impractical values. (Worthless values, then?) But you don't mention creativity. The article blithely says that AI creates new ideas and understands things. I don't think it does.

>Terence Tao is arguing

This is a guest post by Amit Sahai.

I'm not sure the scarce resource will be people capable of understanding the AI's work. It may be institutions willing to tolerate the cost of understanding it

If AI ruins humans doing mathematics because of "economic strategy" we should destroy, not the machines (although the data-centers will be burned down as a byproduct) but the economic system that demands this.

The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.

> I pored over every single line of code Claude generated with razor sharp scrutiny.

That is essentially impossible, since if your pored over individual lines, your scrutiny cannot be razor sharp. There are few people who can pore over code with razor-sharp scrutiny (and different people are better at scrutinizing different aspects).

> Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function.

I am doubtful that this is the case. Even that supposedly-naive example is not as trivial as you might imagine, when you consider overflow, defined vs undefined behavior, and floating-point representation details. And you can't be confident like that about a human either.

> Why check it? It's obviously going to be correct.

And then when you stop checking it, the companies that run the service will tweak the model to benefit themselves in some way, possibly at your expense, and you will be none the wiser.

All the companies trying to get you to use AI are your adversaries. They can and will exploit your use of their systems for their own gain.

>the economically dominant strategy to not verify them and not double check them

In the big scheme of things is it really that expensive to verify it if a lean proof is generated? The agent itself will likely have already verified such Lean code before calling it "done".

> my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.

I use "frontier" AI models daily at day_job. I can confidently say that anyone who is satisfied with the output of LLM code (enough to commit it straight off) is just an absolutely shit programmer. Sorry but I don't have any other way to put it.

The code is (with rare exceptions) atrocious on every level. It is only not atrocious if you take multiple iterations of "review and correct".

>If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one?

Like the saying goes, if my grandmother had wheels she would have been a bike. LLMs can't even produce quality maintainable code for a trivial web service or whatever. Why are we planning for what we will do when they can "design" 10,000 nuclear power plants without any flaw?

The models get things wrong in the way that humans don't.

They will never make a logical error yet make terrible assumptions and poor long scale decisions.

Wake me up when an agent swarm can write gcc in a box sealed from the internet.

Am I the only guy who still thinks we're kind of putting the cart before the horse here? Look, I would love to live in a world where AI is in the business of designing terawatt fusion plants and revolutionizing all other aspects of society. But right now it can't even really tell a puddle in the road. I feel like we have a really long way to go here, hype-laden PR releases about solving math problems aside.

We’re seeing more and more slippery slope arguments, except the slippery slope leads to human cognitive oblivion and it is actually a good thing actually.

> I just hope that there are more Terence Taos out there than people like me.

Just spare me. Being under external pressure to “ship code” is one thing, but being personally inclined one way or another (no external pressure) is another. And when you think being inclined like that is existentially risk (for human civ?) then, what? It’s just the way you are wired and hopes and prayers that collectively that doesn’t drive us off the cliff?

This aw shucks persona isn’t convincing. Same thing with AI Bros who are (1) making the most awesome tech that has ever existed, and (2) aw shucks hope it doesn’t kill us all in the end.

People please, an LLM is just a vector database that spits out statistically viable answers which highly depend on its training material. There's no real "intelligence" involved.

>We can have Super Intelligence, but if humans are incapable of comprehending it, it is just another useless dead artifact.

Animals are incapable of comprehending much of what humans do. Nonetheless, we humans have vastly transformed their world and caused the extinction of many species.

> the process is the result.

For who though?

Is it important that each person understand it on their own?

Why does it impact me if another human understands something or not?

It impacts me right now because that human can use that knowledge to explain things to me or to build new things using that knowledge.

But if an AI can explain and build better, then what good does it do having the other human know the thing?

Understanding might have intrinsic value to me, but intrinsic value to me doesn't pay the bills.

What am I missing here? I kinda expected better from Terrance given such an audacious title.

Precisely. I don’t really care whether LLM’s can produce code more and better than me. I learn languages, program and study computation to understand better the world we live in. Being a human being means increasingly being technologically augmented. To have any deep understanding of that world requires deep understanding of maths, physics, programming etc. that you can only gather by doing those things and solving problems with your meat brain.

There’s a Twitch-streamer Tsoding who programs on C for fun calling it “recreational coding”. Maybe human programming will be a form of art in the future, virtually useless for big corporations to make money. I don’t care, I love it anyway.

> The output of an LLM is useless without a human mind to comprehend it.

I don’t understand mRNA vaccines, but they are useful to me. The output of an LLM could at least in theory be the design to a major technological advancement, and its implementation with automatic tooling. We don’t have to understand that for it to be useful.

> You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind.

Normatively, this ought to be true. Descriptively, this is of course false. Our entire society is organized around producing commodities, typically by consuming people as inputs.

> There is a failure to understand that the process is the result

I'm glad to hear people say this now, because for the longest time they've also been bullshitting the public to think that pure math studies would someday lead to some kind of useful outcome.

"transforming the mind" is an exercise in ego-driven self-gratification, no different than getting ten degrees and never getting a job, nor any different than straight up porn or gambling. You have been put on earth to contribute to the progress of humanity, so get dirty and start making commodities. no more useless math-monks.

> You don't study mathematics or computer science and information theory to produce commodities. You study them to transform your mind.

That's nonsense. Of course you study them to meet demand.

>The output of an LLM is useless without a human mind to comprehend it

This is like saying software is useless if the user doesn't read the source code of it. This what happens >99.99999% a person uses software. People want to be entertained or have their problems solved.

Agree 100%, especially with your second paragraph. The majority of domain experts I've interfaced with in my own domain, along with things I am not an expert in, are typically experts not because of the solutions they build, but because of the things they _don't_ build. It is the cliche "a clever person solves a problem, a wise person avoids it" paraphrase.

But I think what is lost is the value in (a) person(s) who intuitively spots and avoids these problems. My hypothetical example that I have seen play out: The domain expert spots one of these problems and succinctly explains that the problem doesn't need to be solved. Maybe it's a byproduct of bad design elsewhere, or there is a much better solution that avoids it altogether. Once the rest of the team/the lead catches on, either hindsight bias or outcome bias, or a combo, takes over and they don't recognize that all the value all along was in the person being able to spot this situation. They throw that in claude and claude does it's sycophant thing (after expressing that the previous situation was flawless for however long), and they are off to the races right back to automation bias with zero pause for thought...

I had an agent (SOTA models, xhigh reasoning, etc.) write and optimize some CUDA code recently. They've gotten a lot better at this stuff. And yet, they couldn't (self-)realize the obvious issue with their code: it was written as if for a typical multicore CPU. What the heck was a (serial) queue doing there? "Work efficiency" as a tradeoff for stalling 20k threads. Its later proposed optimizations were all about "can we get the queue faster" rather than "maybe we should actually parallelize the work for our very-parallel processor".

Funny stuff, as if it were hell-bent on writing a paper rather than actual software.

> For truly difficult problems, it’s entirely possible that humans are simply incapable of comprehending why a solution is true. But ultimately the practical value of applying that solution to the real world is going to eclipse our need to understand it.

This is hardly new or novel, you’ve just described huge chunks of existing engineering disciplines. There is still no complete model that explains why airplane wings work, there are lots of very useful models, but all of them have fundamental flaws where their behaviour completely breaks down in certain very possible scenarios. Notably scenarios where airplane wings don’t spontaneously either stop working or explode.

We also built the entire commercial airliner system decades before we even had computers capable of doing the aerodynamic analysis of airplanes, but none of that prevented us from creating huge engineering disciplines around the empirical data we did have, and slowly chip away at the underlying theories as maths and computing improved over time.

So we’ve always lived in a world where we extract value from systems we fundamentally don’t understand. But that’s never stopped us from working to understand them anyway, and deriving even further value from that improved understanding.

You deal with this every time you wake up, or visit a specialist of any kind. You likely didn't make your own clothes, you didn't grow your own food, dam up any rivers for your water supply, build any power plants to power up your phone/computer, nor did you spend any of the deep amount of focus required to build up the knowledge and technology required to provide us these 'things' in life. The only difference is, you didn't pay much attention to these facts because they became mundane. Take any of them away for any decent amount of time, and suddenly you become acutely aware of what it is you have.

The author mentioned what the issue is: a lot of people are experiencing what it's like to 'catch up' to the more intelligent ideas. Everyone can 'get' the intelligent ideas given enough time. But how much time do you have? And he's concerned someone will rush and create some kind of world-ending solution because they're too trusting of a technology they don't understand.

His only solution is some kind of throttle on the advancement of technology, and arguably knowledge (whose?).

That's learned helplessness as a lifestyle.

You observe others behaving poorly and immediately succumb to defeatism, assuming there was no better way.

When presuming, all that matters was the short term economic profit, you simply use the wrong gauge.

Here, long term human survival is at stake, regressing to zoo animals isn't a sensible option.

In the future I feel one parallel to this is my research area, SAT solving.

People used to solve logic problems by hand, verify logic. Now you pile it into a computer. The problem has been around for a while, no-one fully ‘understands’ the proof of the four colour theorem as a big chunk is computer proved.

We are now just changing (admittedly greatly) what we can put in a box marked ‘checked by computer.

Certainly humans can enjoy exploring such things, but will they really be needed?

> The sheer surface area that AI can create in our intellectual life is limitless

I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.

The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.

The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.

> Humans are close to their ceiling. AIs are just getting started.

The whole point of mathematics is to vastly exceed that natural ceiling by gradually building a framework for understanding. In fact it’s wrong to speak of a ceiling altogether. If there were a ceiling, we’d have hit it long ago.

AIs have already swallowed the entire history of human thought, but apparently they’re ’just getting started’. I can only assume you don’t know what mathematicians actually do.

> Humans are close to their ceiling.

If Math Academy teach 10 year old kids calculus, I doubt that.

> AI has the ability to be impartial, and immutable.

Which AI has this ability? LLMs definitly don't.

I recommend you the book the rational optimist.

Not saying you’re wrong you just can’t be so certain

I believe he’s advocating for mathematicians to understand the novel concepts introduced with each breakthrough rather than individuals understanding data stemming from empirical observations.

it seems like most of the recent posts on Tao's blog are not actually Tao's?

As of now the ai needs hands, and maybe more importantly, cognitive intentionality which thus far only arises from living beings drive for homeostasis to survive and reproduce. If ai can have this same intentionality is a big unknown.

>humility

Fear? Ai is an abstract (for most people). A steam engine, you can touch. And, it doesn't replicate.

> "Who's gonna pay me, and for what ?". That's the question I want answered.

Unfortunately that’s a question you’ll have to answer for yourself. Also, people are going to risk going along whatever path gives them work as many today already do.

> Tao, Gowers et.al. will go into history as the professors who ruined math.

I don't know how to put this, but what... do you want anyone to do? That's a lot of complaining and hyperbole.*

I don't think that a few professors are the whole reason why maths is getting steamrolled. I think the tech is inevitable. The responses I got were Nazi levels of evil.

[edit] Reading back, I think your point was that they pass off their points of view as the only ones. Allegedly.

* [edit] For the sake of transparency, an anonymous coward below wants me to recall that I wrote the following, which I deleted because I didn't know how to put a strikethrough around it: Do you have anything to say other than "booo! boooooo!" and "waaa! waaaaaa! waaaaaaaaaaa! ;_;"

What a doomer way of thinking. Scientific advancements has positively improved the lifes of people in the past I don't see how scientific advancements accelerated by AI would be any different.

Ha, are you invoking something he said prior to discovering Russell's paradox? Which came about because he wanted to show that all of mathematics can be derived from logic about sets, but instead he showed that it couldn't be.

Maybe he is, but you are quoting Amit Sahai, the author of this blog post.

How is that relevant at all?

Life is all about purpose. Mathematicians are instead concerned with meaning. We need more philosophers.

The point of the post is that what you call "AI knowledge" is not knowledge.

Even if that happens, what’s the point if there’s no human involvement? AI doing math for math’s sake? And doing what with it?

I think a better analogy would be comparing to an ancient human instead of an ant. An ancient human would have none of the basic abstractions that we take for granted today like literacy and arithmetic, so it would be very difficult for them to even attempt trying to understand quantum mechanics. But I don't think it's impossible because our ability to learn by stacking abstractions is basically endless——so far as our health permits at least.

We've been those ants for a million years, and only in the last 100 did we start to wrap our heads around quantum physics. We are the purpose behind creating LLMs. There's plenty in the universe we don't understand, and it's very human to keep striving to do so.

If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.

Don't write us out quite yet. :)

Or maybe ants and humans are qualitatively different. Maybe there's a critical mass of intelligence where you can pretty much understand anything, and maybe humans are past that threshold. I don't know that for sure, but I don't think we're anywhere close to hitting fundamental limits to our ability to understand the universe.

Need? We don't need lots of things, including computers. We did well without them for hundreds of thousands of years.

We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.

That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.

> Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics

This makes no sense whatsoever.

We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.

If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.

It’s not Tao. None of these guest posts are Tao.

I agree. I believe it could be the best piece about AI ever written. The themes of lost dreams reawakened, sobering up of egos, sci fi technological development, the role of humanity. It’s a weird feeling to see that it’s actually really happening.