The interview which I've watched recently with Rich Sutton left me with the impression that AGI is not just a matter of adding more 9s.

The interviewer had an idea that he took for granted: that to understand language you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.

I have a very surface level understanding of AI, and yet this always seemed obvious to me. It's almost a fundamental law of the universe that complexity of any kind has a long tail. So you can get AI to faithfully replicate 90% of a particular domain skill. That's phenomenal, and by itself can yield value for companies. But the journey from 90%-100% is going to be a very difficult march.

FWIW, Karpathy literally says, multiple times, that he thinks we never left the exponential - that all human progress over last 4+ centuries averages out to that smooth ~2% growth rate exponential curve, that electricity and computing and AI are just ways we keep it going, and we'll continue on that curve for the time being.

It's the major point of contention between him and the host (who thinks growth rate will increase).

The thing about this, though - cars have been built before. We understand what's necessary to get those 9s. I'm sure there were some new problems that had to be solved along the way, but fundamentally, "build good car" is known to be achievable, so the process of "adding 9s" there makes sense.

But this method of AI is still pretty new, and we don't know it's upper limits. It may be that there are no more 9s to add, or that any more 9s cost prohibitively more. We might be effectively stuck at 91.25626726...% forever.

Not to be a doomer, but I DO think that anyone who is significantly invested in AI really has to have a plan in case that ends up being true. We can't just keep on saying "they'll get there some day" and acting as if it's true. (I mean you can, just not without consequences.)

Reminds me of a time-honored aphorism in running:

A marathon consists of two halves: the first 20 miles, and then the last 10k (6.2mi) when you're more sore and tired than you've ever been in your life.

It's a good way to think about lots of things. It's Pareto efficiency. The 80/20 rule

20% of your effort gets you 80% of the way. But most of your time is spent getting that last 20%. People often don't realize that this is fractal like in nature, as it draws from the power distribution. So of that 20% you still have left, the same holds true. 20% of your time (20% * 80% = 16% -> 36%) to get 80% (80% * 20% => 96%) again and again. The 80/20 numbers aren't actually realistic (or constant) but it's a decent guide.

It's also something tech has been struggling with lately. Move fast and break things is a great way to get most of the way there. But you also left a wake of destruction and tabled a million little things along the way. Someone needs to go back and clean things up. Someone needs to revisit those tabled things. While each thing might be little, we solve big problems by breaking them down into little ones. So each big problem is a sum of many little ones, meaning they shouldn't be quickly dismissed. And like the 9's analogy, 99.9% of the time is still 9hrs of downtime a year. It is still 1e6 cases out of 1e9. A million cases is not a small problem. Scale is great and has made our field amazing, but it is a double edged sword.

I think it's also something people struggle with. It's very easy to become above average, or even well above average at something. Just trying will often get you above average. It can make you feel like you know way more but the trap is that while in some domains above average is not far from mastery in other domains above average is closer to no skill than it is to mastery. Like how having $100m puts your wealth closer to a homeless person than a billionaire. At $100m you feel way closer to the billionaire because you're much further up than the person with nothing but the curve is exponential.

I think a ton of people see a line going up and they think exponential. When in Reality, the vast majority of the time it’s actually logistic.

Ha, I often speak of doing the first 90% of the work, and then moving on to the following 90% of the work...

I also quite like the way he puts it. However, from a certain point onward, the AI itself will contribute to the development—adding nines—and that’s the key difference between this analogy of nines in other systems (including earlier domain‑specific ML ones) and the path to AGI. That's why we can expect fast acceleration to take off within two years.

This is an amazing quote that really applies to all software development

When I worked at Facebook they had a slogan that captured this idea pretty well: “this journey is 1% finished”.

Infinitely big little numbers

Academia has rediscovered itself

Signal attenuation, a byproduct of entropy, due to generational churn means there's little guarantee.

Occam's Razor; Karpathy knows the future or he is self selecting biology trying to avoid manual labor?

His statements have more in common with Nostradamus. It's the toxic positivity form of "the end is nigh". It's "Heaven exists you just have to do this work to get there."

Physics always wins and statistics is not physics. Gamblers fallacy; improvement of statistical odds does not improve probability. Probability remains the same this is all promises of some people who have no idea or interest in doing anything else with their lives; so stay the course.

if it works 90% of the time that means it fails 10% of the time, to get to 1% failure rate is a 10x improvement and from 1% failure rate to a 0.1% failure rate is also a 10x improvement

First time being hearing it be called "march of nines", did Tesla make the term, I thought it was an Amazon thing

like leveling to 99 in old school runescape

Notably the scaling law paper shows result graphs on log-scale

The question is how many nines are humans.

You know what they say, a Silicon Valley 9 is a 10 anywhere else. Or something like that.

It's nice seeing commentary from someone who is both knowledgable in AI and NOT trying to pump the AI bag.

Right now the median actor in the space loudly proclaims AGI is right around the corner, while rolling out pornbots/ads/in-chat-shopping, which generally seems at odds with a real belief that AGI is close (TAM of AGI must be exponentially larger than the former).

Imagine a PhD mortally terrified of exceptions!

Now I see why Karpathy was talking of RL up-weights as if they were a destructive straw-drawn line of a drug for an LLM's training.

> Literally anything that matters is going to get implemented in a crontrolled workflow that strips out all the autonomy with human checkpoint at every step.

Yea, there aren't a ton of problems (that I can see) in my current domain that could be solved by having unattended agents generating something.

I work in healthcare and there are a billion use cases right now, but none that don't require strict supervision. For instance, having an LLM processing history and physicals from potential referrals looking for patient problems/extracting historical information is cool, but it's nowhere near reliable enough to do anything but present that info back to the clinician to have them verify it.

He’s not just talking about agents good enough to replace workers. He’s talking about whether agents are currently useful at all.

>Overall, the models are not there. I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it’s not. It’s slop. They’re not coming to terms with it, and maybe they’re trying to fundraise or something like that. I’m not sure what’s going on, but we’re at this intermediate stage. The models are amazing. They still need a lot of work. For now, autocomplete is my sweet spot. But sometimes, for some types of code, I will go to an LLM agent.

>They kept trying to mess up the style. They’re way too over-defensive. They make all these try-catch statements. They keep trying to make a production code base, and I have a bunch of assumptions in my code, and it’s okay. I don’t need all this extra stuff in there. So I feel like they’re bloating the code base, bloating the complexity, they keep misunderstanding, they’re using deprecated APIs a bunch of times. It’s a total mess. It’s just not net useful. I can go in, I can clean it up, but it’s not net useful.

Because that's the definition that is leading to all these investments, the promise that very soon they will reach it. If Altman said plainly that LLMs will never reach that stage, there would be a lot less investment into the industry.

Quite telling -- thanks for the insightful comment as always, Simon. Didn't know that, even though I've been discussing this on and off all day on Reddit.

He's a smart man with well-reasoned arguments, but I think he's also a bit poisoned by working at such a huge org, with all the constraints that comes with. Like, this:

  You can’t just tell them something and they’ll remember it.

LLMs are a necessary part of AGI(/"agents") due to their ability to avoid the Frame Problem[1], but they're far from the only needed thing. We're pretty dang good at "remembering things" with computers already, and connecting that with LLM ensembles isn't going to take anywhere close to 10 years. Arguably, we're already doing it pretty darn well in unified systems[2]...

If anyone's unfamiliar and finds my comment interesting, I highly recommend Minsky's work on the Society of Mind, which handled this topic definitively over 20 years ago. Namely;

A short summary of "Connectionism and Society of Mind" for laypeople at DARPA: https://apps.dtic.mil/sti/tr/pdf/ADA200313.pdf

A description of the book itself, available via Amazon in 48h or via PDF: https://en.wikipedia.org/wiki/Society_of_Mind

By far my favorite paper on the topic of connectionist+symbolist syncreticism, though a tad long: https://www.mit.edu/~dxh/marvin/web.media.mit.edu/~minsky/pa...

[1] https://plato.stanford.edu/entries/frame-problem/

[2] https://github.com/modelcontextprotocol/servers/tree/main/sr...

Agreed. I'd also add he's intellectually honest enough to not overhype what's happening just to hype whatever he's working on or appear to be a thought leader. Just very clear, pragmatic, and intellectually honest thought about the reality of things.

I also don’t think our generation will see actual AGI, but imo the hard "intelligence“ part isn’t needed as we can use our intelligence. Using it as a tool will hopefully lead to plenty of cool things in the future.

I did the math some years ago on how much computing is required to simulate a human brain - a brain has around 90 billion neurons with each neuron having an average of 7,000 connections to other neurons. Lets assume thats all we need. So what do we need to simulate a neuron, one cpu? or can we fit more than one in a CPU, lets say 100 so we're down to one billion cpu's and 70 trillion messages flying between them every what? mSec?.

Simulating that is a long way away - so the only possibility is that brains have some sort of redundancy and we can optimise that away. Though computers are faster than brains so its possible maybe, how much faster? So lets say a neuron does its work in a mS and we can simulate this work in 1uS, ie a thousand times faster - thats still a lot. Can we get to a million times faster? even then its still a lot. Not to mention the power required for this.

Even if we can fit a million neurons in a CPU thats still 90 million CPU's. Only 10% are active say, still 9 million CPU's, a thousand times faster - 9,000 cpu's nearly there but still a while away.

The reason why it’s so hard to define intelligence or consciousness is because we are hopelessly biased with a datapoint of 1. We also apply this unjustified amount of mysticism around it.

https://bower.sh/who-will-understand-consciousness

I don't think we can ever know that we are generally intelligent. We can be unsure, or we can meet something else which possesses a type of intelligence that we don't, and then we'll know that our intelligence is specific and not general.

So to make predictions about general intelligence is just crazy.

And yeah yeah I know that OpenAI defines it as the ability to do all economically relevant tasks, but that's an awful definition. Whoever came up with that one has had their imagination damaged by greed.

Without going to deep into the rabbit hole, one could argue that at the first-order, intelligence is the ability to learn from experience towards a goal. In that sense, LLMs are not intelligent. They are just a (great) tool at the service of human intelligence. And so we’re just extremely far from machine intelligence.

> we still lack schematic-level understanding of what “intelligence” even is or how it works. Not to mention how it interfaces with “consciousness”, and their likely relation to each other

I think you can get pretty far starting from behavior and constraints. The brain needs to act in such a way as to pay for its costs. And not just day to day costs, also ability to receive and give that initial inheritance.

From cost of execution we can derive an imperative for efficiency. Learning is how we avoid making the same mistakes and adapt. Abstractions are how we efficiently carry around past experience to be applied in new situations. Imagination and planning are how we avoid the high cost of catastrophic mistakes.

Consciousness itself falls from the serial action bottleneck. We can't walk left and right at the same time, or drink coffee before brewing it. Behavior has a natural sequential structure, and this forces the distributed activity in the brain to centralized on a serial output sequence.

My mental model is that of a structure-flow recursion. Flow carves structure, and structure channels flow. Experiences train brains and brain generated actions generate experiences. Cutting this loop and analyzing parts of it in isolation does not make sense, like trying to analyze the matter and motion in a hurricane separately.

This so much this. We don’t even have a good model for how invertebrate minds work or a good theory of mind. We can keep imitating understanding but it’s far from any actual intelligence.

That would require philosophical work, something that the technicians building this stuff refuse to acknowledge as having value.

Ultimately this comes down to the philosophy of language and of the history of specific concepts like intelligence or consciousness - neither of which exist in the world as a specific quality, but are more just linguistic shorthands for a bundle of various abilities and qualities.

Hence the entire idea of generalized intelligence is a bit nonsensical, other than as another bundle of various abilities and qualities. What those are specifically doesn’t seem to be ever clarified before the term AGI is used.

> I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description ["<insert general intelligence buzzword>"], and perhaps I could never succeed in intelligibly doing so. But I know it when I see it, and the <insert llm> involved in this case is not that.

https://en.wikipedia.org/wiki/I_know_it_when_I_see_it

Very good point. With one caveat, though. Even though I was not there, I imagine that debates about the wheel were less heated than those we’re having about AI. I think this is because the latter is much more abstract, too close from our own consciousness etc. Wheels never challenged our place in the universe.

> I don't like calling it "Artificial Intelligence" because it's not intelligence

A pattern I noticed in a AI[sic] discussions: Handwavily declaring what intelligence is not, while not explaining what is.

Hmm good point. I skimmed the transcript looking for an accurate, representative quote that we could use in the title above. I couldn't exactly find one (within HN's 80 char limit), so I cobbled together "It will take a decade to get agents to work", which is at least closer to what Karpathy actually said.

If anyone can suggest a more accurate and representative title, we can change it again.

Edit: I thought of using "For now, autocomplete is my sweet spot", which has the advantage of being an exact quote; but it's probably not clear enough.

Edit 2: I changed it to "It will take a decade to work through the issues with agents" because that's closer to the transcript.

Anybody have a better idea? Help the cause of accuracy out here!

He says re agents:

>They don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this stuff. They don't do a lot of the things you've alluded to earlier. They don't have continual learning. You can't just tell them something and they'll remember it. They're cognitively lacking and it's just not working.

>It will take about a decade to work through all of those issues. (2:20)

Did the same with Sutton (LLMs are a dead end) when Sutton never said this in the conversation.

There's a lot of salt here

I agree, I think I learned the most on this topic from his videos. And before that (a while ago), it was Andrew Ng coursera's class. The latter had hands-on project, which is much better than just listening in term of retention.. I don't know if Andrej Karpathy has more structured classes somewhere.

In the same sense that split brain patients [1] will make up a reasonable explanation for what the other half did.

And why witnesses are preferably interviewed very shortly after they witnessed a crime. Before their brains start to 'fill in the blanks'

1: https://en.wikipedia.org/wiki/Split-brain

Is that not insane?

> In the natural world, agency is a consequence of death: by dying, the feedback loop closes in a powerful way.

I don't follow. If we, in some distant future, find a way to make humans functionally immortal, does that magically remove our agency? Or do we not have agency to begin with?

If your position on the "free will" question is that it doesn't exist, then sure I get it. But that seems incompatible with the death prerequisite you have put forward for it, because if it doesn't exist then surely it's a moot point to talk prerequisites anyway.

Models die too - the less agentic ones are out-competed by the more agentic ones.

Every AI lab brags how "more agentic" their latest model is compared to the previous one and the competition, and everybody switches to the new model.

I don't agree but I did laugh

This terminology is confusing. Historically, AI was always used to mean any kind of machine intelligence, including the most basic novice chess AI, or an image classifier, or a video game character's AI. Now a lot of people seem to be using it as a synonym for AGI - a human-level intelligence.

Without defining AGI as a goal, current AI companies would not be able to amass the amount of money they want

The biggest problem I've had with Kurzweil and the exponential growth curve is that the elbow depends entirely on how you plot and scale the axis. With a certain vantage point we have arguably been on an exponential curve since the advent of Homo Sapiens.

I lost all respect for him after reading about his views on medical immortality. His argument is that over time human life expectancy has been constantly increasing * and he calculated that based on some arbitrary rate of acceleration, that science would be expanding human life expectancy by more than a year, per year - medical immortality in other words, and all expected to happen just prior to the time he's reaching his final years.

The overwhelming majority of all gains in human life expectancy have come due to reductions in infant mortality. When you hear about things like a '40' year life expectancy in the past it doesn't mean that people just dropped dead at 40. Rather if you have a child that doesn't make it out of childhood, and somebody else that makes it to 80 - you have a life expectancy of ~40.

If you look back to the upper classes of old their life expectancy was extremely similar to those of today. So for instance in modern history, of the 15 key Founding Fathers, 7 lived to at least 80 years old: John Adams, John Quincy Adams, Samuel Adams, Jefferson, Madison, Franklin, John Jay. John Adams himself lived to 90. The youngest to die were Hamilton who died in a duel, and John Hancock who died of gout of an undocumented cause - it can be caused by excessive alcohol consumption.

All the others lived into their 60s and 70s. So their overall life expectancy was pretty much the same as we have today. And this was long before vaccines or even us knowing that surgeons washing their hands before surgery was a good thing to do. It's the same as you go back further into history. A study [1] of all men of renown in Ancient Greece was 71.3 [1], and that was from thousands of years ago!

Life expectancy at birth is increasing, but longevity is barely moving. And as Kurzweil has almost certainly done plentiful research on this topic, he is fully aware of this. Cognitive dissonance strikes again.

[1] - https://pubmed.ncbi.nlm.nih.gov/18359748/

It's curious that Kurzweil's predictions about transcending biology align so closely with his expected lifespan. Reminds me of someone saying, if you ask a researcher for a timeline of a breakthrough they'll give you the expected span of their career.

Hegel thought history ended with the Prussian state, Fukuyama thought it ended in liberal America, Paul thought judgement day was so close you need not bother to marry, the singularity always comes around when the singularians get old. Funny how that works

> He predicts we'll merge with AI to transcend biological limits.

The merge with a machine 1 million times more intelligent than us is the same as letting AI use our bodies. I'd rather live in cave. Iirc, the 7th episode of Black Mirror starts with this plot line.

Waymo's self-driving cars are scaling quickly. With some inaccuracy it can be said that the problem is solved, we have the technology for a full-scale deployment, we just need to do the boring work to deploy it everywhere.

Fusion power seems closer than ever. And plenty of experts just five years ago thought AGI would still be decades away. A credible expert suggesting AGI is ten years away is a sign of real progress.

What was the last example where humans succeeded at a hard problem like that?

Space flight?

Yeah, I see people pooh-poohing the idea of humanoid robots being useful this decade, saying it will take at least 20 years. Oh yeah? Instead of 5 years to render all human labor obsolete, it will take 20? The magnitude of that change is so large that the implications of it happening anytime in our lifetimes are too big to ignore.

The important thing is that this is not going to be perpetually 20 years in the future like fusion. This is something that will happen.

Or Sora...

Why not?

If inference can become profitable.

Importantly, the first 9s are the easiest.

If you need to get to 9 9s, the 9th 9 could be more effort than the other 8 combined.

I don't know a single one of the "Social Science" items, and I'm pretty sure 90% of college educated people wouldn't know a single one either.

I agree it seems like a better-structured effort than many others. But its shortcomings go beyond a shallow and incomplete foundation in psychology. It also has basic errors in its execution, eg a "Geography" question about centripetal and centrifugal forces. Color me extremely skeptical.

What developer is excited about "vibe coding"? The only people excited about "vibe coding" are people who can't code.

> We will never achieve AGI, because we keep moving the goalposts.

I think it's fair to do it to the idea of AGI.

Moving the goalpost is often seen as a bad thing (like, shifting arguments around). However, in a more general sense, it's our special human sauce. We get better at stuff, then raise the bar. I don't see a reason why we should give LLMs a break if we can be more demanding of them.

> SOTA models are already capable of outperforming any human on earth in a dizzying array of ways, especially when you consider scale.

Performance should include energy consumption. Humans are incredibly efficient at being smart while demanding very little energy.

> But few if any humans on earth can demonstrate the breadth and depth of competence that a SOTA model possesses.

What if we could? What if education mostly stopped improving in 1820 and we're still learning physics at school by doing exercises about train collisions and clock pendulums?

Yeah - I feel like this point of view is rarely alluded to whenever the AI oligarchs speak publicly. The link between intelligence and consciousness feels undeniable to me, yet somehow they always manage to sidestep this point when discussing "technology" for over 2 hours on YouTube.

It's at the end of the interview. He wants to build Starfleet Academy for technical fields. Physical with a digital equivalent. Thinks education will become like a gym (self-educate to look sexy) by the time AGI gets here.

METR measures tasks, not projects. No project I've worked on had individual tasks that were supposed to take longer than 2 weeks, the PM* broke them down to sub-tasks if they were any bigger.

* At least, where we had a PM. The places I was self-directed could arguably provide an interesting comparison.

It's like saying your newborn will have the same mass as earth in 50 years if he continues on his first month weight gain trajectory.

AIs already fake-simulate the weather (chaotic system) using 1% of the resources used by the real-simulating supercomputers.

You want a "core kernel" with "general language parsing" but no training on real-world concepts.

Read that sentence again. Slowly.

What do you think "general language parsing" IS if not learned patterns from real-world data? You're literally describing a transformer and then saying we need to invent it.

And your TLS example is deranged. You want an agent to discover the TLS protocol by randomly sending ethernet packets? The combinatorial search space is so large this wouldn't happen before the sun explodes. This isn't intelligence! This is bruteforce with extra steps!

Transformers already ARE general algorithms with zero hardcoded linguistic knowledge. The architecture doesn't know what a noun is. It doesn't know what English is. It learns everything from data through gradient descent. That's the entire damn point.

You're saying we need to solve a problem that was already solved in 2017 while claiming it needs a century of quantum computing.

METR kinda has been described as a Moore's Law for LLMs, but personally I think the financial environment around AI will break within 2 years — which still represents a huge increase in capabilities, but isn't a decade.

• Text and graphs: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...

• Video interview: https://www.youtube.com/watch?app=desktop&v=evSFeqTZdqs

That said, I've not seen work that looks promising to the problem of, as he phrased it: "They don’t have continual learning. You can’t just tell them something and they’ll remember it."

Saying any specific timeframe for that, 10 years or anything else, seems too certain. Some breakthrough might already exist and be unknown, but on the other hand it may require a fundamental advancement in mathematics in order to make it possible to find something at least close to optimal in a billion-dimensional (or whatever) vector space with only the first few dozen examples.

An agent is just an LLM calling tools in a loop. If you're a "show me the code" type person like me, here's a worked example: https://samdobson.uk/posts/how-to-build-an-agent/

Kinda. It's just an LLM that performs function calling (i.e. the LLM "decides" when a function needs to be called for a task and passes the appropriate function name and arguments for that function based on its context). So yea an "agent" is that LLM doing all of that and then your program that actually executes the function accordingly.

That's an "agent" at its simplest -- a LLM able to derive from natural language when it is contextually appropriate to call out to external "tools" (i.e. functions).

"Something" is broad and not well defined, but basically yeah. Rather than try to define it in terms of complexity of the something, I'll put it in terms of minutes. If the LLM returns a response, and that response gets fed into a system and run, and that's it, I wouldn't really call that agentic. It's got to go a few more rounds back and forth to be agentic, imo. In terms of time, I'd say the agent program has to be capable of at least 10 minutes of going from user input, then the program calling into the LLM, feeding the LLM response into a system, feeding that result back into the LLM, and feeding that into the system in a loop. Obviously there are ways to game that metric, like the terrible lines of code metric, but I think it's a decent handwave for when it feels like there's an agent working for me rather than a non-agentic system. What it's doing for those 10 minutes is important, calling "sleep 600" obviously doesn't count.

Eg for a programming LLM with an agentic agent and access to a computer, would be able to, given design-doc.md and Todo.md, implement feature X, making sure it compiles, run some basic smoke tests, write appropriate unit tests, make sure they all pass, and finally push the code and create a draft PR.

Naturally, not every call into the agent is going to take the full 10 minutes. It may need to ask questions before getting started, or stop if there's an unrecoverable error. Sometimes you'll just need to tell it "continue", but the system should be capable of a 10-minute run (hopefully longer!) given enough support.

As in more impressive than what we consumers have access to from OpenAI, Anthropic, etc?

More specific responses have come in as people have digested more of the content.

This is the reflexive/reflective distinction (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...). Reflexive comments—the kind that express some pre-existing feeling or opinion that happens to get triggered by association—are much faster to produce, so unfortunately they show up first in many threads.

Power use is less important than model capability

AGI is either more scale or differing systems, or both

They can always optimize for power consumption after AGI has been reached

If it takes 50 years, you might have it for millions of years along with anti-aging solutions that actually work, I'll have probably died of old age.

I don't know how much wish fulfilment there is in people's timelines.

No, it’s because of money and the hype cycle.

They are afraid to say it because it may affect the funding. Currently with all the hype surrounding AI investors and governments will literally shower you with funding. Always follow the money:) Buy the dream - sell the reality.

I don’t think they’re scared, I think they know it’s a lose-tie game.

If you’re correct, there’s not much reward aside from the “I told you so” bragging rights, if you’re wrong though - boy oh boy, you’ll be deemed unworthy.

You only need to get one extreme prediction right (stock market collapse, AI taking over, etc ), then you’ll be seen as “the guru”, the expert, the one who saw it coming. You’ll be rewarded by being invited to boards, panels and government councils to share your wisdom, and be handsomely paid to explain, in hindsight, why it was obvious to you, and express how baffling it was that no one else could see what you saw.

On the other hand, if predict an extreme case and you get it wrong, there’s virtually 0 penalties, no one will hold that against you, and no one even remembers.

So yeah, fame and fortune is in taking many shots at predicting disasters, not the other way around.

And now (useful) quantum computing is 5 years away! Has been for a few years, too.

It's an excellent time-frame that sounds imminent enough to draw interest (and funding), but is distant enough that you can delay the promised arrival a few times in the span of a career before retiring.

Fusion research lives and dies on this premise, ignoring the hard problems that require fundamental breakthroughs in areas such as materials science, in favor of touting arbitrary benchmarks that don't indicate real progress towards fusion as a source of power on the grid.

"Full self driving" is another example; your car won't be doing this, but companies will brag about limited roll-outs of niche cases in dry, flat, places that are easy to navigate.

I mean Dennis is just another hype man now, irrespective of how important research they may be doing in the background.

LLMs are close enough to pass the Turing Test. That was a huge milestone. They are capable of abstract reasoning and can perform many tasks very well but they aren't AGI. They can't teach themselves to play chess at the level of a dedicated chess engine or fly an airplane using the same model they use to copypasta a React UI. They can only fool non-proficient humans into believing that they might be capable of doing those things.

I think most people think of AGI as able to do the stuff humans do and it's still missing a fair bit there.

A transistor is very simple too, and here we are. Don’t dismiss something because it’s simple.

> Most of these companies value is built on the idea of AGI being achievable in the near future.

Is it? Or is it based on the idea a load of white collar workers will have their jobs automated, and companies will happily spend mid four figures for tech that replaces a worker earning mid five figures?

It’s possible for AI to provide tremendous economic value without AGI

Exactly. A 5-10 year timeline and you've got the formula for a new Space Race with China. Give us $7T or else China will control the world.

This 2024 story feels like ancient history that everyone has forgotten: https://www.cnbc.com/2024/02/09/openai-ceo-sam-altman-report...

There is a very strange totally coincidental correlation where if you are smart and NOT trying to raise money for an AI start-up, you think AGI is far away, and if you are smart and actively raising money for an AI start-up, then AGI is right around the corner. One of those odd coincidences of modern life

The future is now

Oh, that's odd. This comment was intended for the vibe-coded.lol post

https://news.ycombinator.com/item?id=45622944

Must have been the flu-brain misfiring

That the speculative opinions of famous researchers are a good starting point for curious conversation.

I mean, HN is no stranger to cults of personality. Post a paulg essay that says something a teenager could write and it'll fly up to position 1.

Is it worse than "Rich CEO expressing certainly over his hunch"

Big scientific revolutions tend to happen before we understand the relevant mechanisms. It is only after the fact that we develop a theory to understand how it works. AGI will very likely follow the same trend. Enough people are throwing enough things at the wall that eventually something will stick.

There's many different definitions of "AGI" that people come up with, and some include dreaming, quantum world, creativity, and some do not.

We Dont know how a horse works, but we got cars. Analogy doesn’t work.

They can't even automate chat support the very thing you would think LLMs would be good at. Yet I always end up needing to talk to a person.

The same thing you described that makes LLM great also make them entirely non-deterministic and unreliable for serious automated applications.

2038 will be more significant

Please don't post unsubstantive comments to HN. Thoughtful criticism is welcome.

We don't even have any intention to do anything about millions of people loosing their jobs and driven into poverty by it, in fact the investments right now gamble/depend on that wealth transfer to happen in the future. We don't even give a shit about other humans, there is absolutely no way we will care about a (hypothetical) different life form entirely.

if openAI didn't put a chat interface in front of an LLM and make it available to the public wouldn't we still be in the same AI winter? Google, Meta, Microsoft, all of the major players were doing lots of LLM work already, it wasn't until the general public found out through the OpenAI's website that it really took off. I can't remember who said it, it was some CEO, that OpenAI had no moat but nether did anyone else. They all had LLMs already of their own. Was the breakthrough the LLM or making it accessible to the general public?

How to tell if you regurgitated this comment vs being truly creative? If you can show me objectively, I’m sold.

We should completely strip all this talk from AI as a field (and get rid of that name as well). It just causes endless confusion, especially for general audience. In the end, the whole shtick with LLMs is that we train matrices to predict next tokens. You can explain this entire concept without invoking AGI, Roko's basilisk, the nature of human consciousness, and all the other mumbo jumbo that tries so hard to make this field what it is not.

There is a lot of overlap between AI and Neuroscience, especially among older researchers. For example Karpathy's PhD supervisor, Fei-Fei Li, researched vision in cat brains before working on computer vision, Demis Hassabis did his PhD in Computational Neuroscience, Geoff Hinton studied Psychology etc... There's even the Reinforcement Learning and Decision Making conference (RLDM - very cool!), which pairs Reinforcement Learning with neuro research and brings together people from both disciplines.

I suspect the average AI researcher knows much more about the brain than typical CS students, even if they may not have sufficient background to conduct research.

From personal experience making the same comparisons during undergrad, I think it just comes down to the availability of conceptual models. If the brain does X, there's a good chance that a computer does something that looks like X, or that X could be recreated through steps Y & Z, etc.

Once I started to realize just how much of the brain is inscrutable, because it is a machine operating on chemicals instead of strict electrical processing, I became a lot more reluctant to draw those comparisons

> Why is there a presumption that we (as people who have only studied CS) know enough about biology/neuroscience/evolution to make these comparisons/parallels/analogies?

well it's straightforward. First lets assume a spherical, perfectly frictionless, brain..

AI researchers and CS people and the rest of us are human brain users and so have some familiarity with them even if they haven't studied neuroscience.

You can make some comparisons between how they perform without really understanding how LLMs or brains work, like to me LLMs seem similar to the part human minds where you say stuff without thinking about it. But you never really get an LLM saying I was thinking about that stuff and figured this bit was wrong, because they don't really have that capability.

>Why is there a presumption that we (as people who have only studied CS) know enough about biology/neuroscience/evolution to make these comparisons?

Hubris.

Ive also found this jarring and it speaks to the hubris of folks that have emerged in the past few decades who dont seem to have much relation to the humanities and liberal arts.

Yeah, the last 3 years of "We now know how to build AGI" failing to deliver shows that there's something being missed about the nature of intelligence. The "We are all stochastic parrots" people has been awfully quiet recently

It takes time for more reflective comments to appear, because reflection is a slower mental operation. Reflexive responses are much faster and tend to be generic and shallow. (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...)

I believe this distinction is pretty fundamental to humans, so we're not likely to escape it, but the good news is that reflective comments do show up eventually if the article is substantive and the reflexive ones haven't ruined the thread. We also try to downweight the more reflexive subthreads.

More at https://news.ycombinator.com/item?id=45625084.

Maybe they used AI to transcribe and summarize the podcast

Just as the core idea of a book can be (lossily) summarized in a few sentences, the core crux of an argument can be quite simple and not require wading though the whole discussion (the AGI discussion is only 30 minutes anyhow).

Granted, a bunch of commenters are probably doing what you’re saying.

Be fair; plenty of people transcribe and read podcasts, and/or summarize/excerpt them.

This one does have the full transcript underneath (wonderful feature). But it's a long read too so I think your assumption is correct :P.

Eh, Dwarkesh has to market the podcasts somehow. I think it's fine for him to use hooks like this and for HN threads to respond to the hooks. 99% of HN threads only ever reply to the headline and that's not changing anytime soon. This will likely cause many people (including myself) to watch the full podcast when we otherwise might not have.

The criticism that people are only replying to a tiny portion of the argument is still valid, but sometimes it's more fun to have an open-ended discussion rather than address what's in the actual article/video.

Who listens to podcasts at 1x speed? That's unbearably slow!

there is a transcript, people can skim for interesting parts and read for 30 minutes and then comment.

edit: typo fix.

Idk how this Dwarkesh Patel got so popular so fast. I'd never heard of him and he keeps popping up in my feeds.

gotta listen at 2x speed!

a very human reaction ;)

genuinely curious to hear your reasoning for why this is the case. i'm always somewhere between bemused and annoyed opening the daily HN thread about AGI and seeing everyone's totally unfounded confidence in their predictions.

my position is I have no idea what is going to happen.

I'd bet the other way because I think Moore's law like advances in compute will make things much easier for researchers.

Like I was watching Hinton explain LLMs to Jon Stewart and they were saying they came up with the algorithm in 1986 but then it didn't really work for the decades until now because the hardware wasn't up to it (https://youtu.be/jrK3PsD3APk?t=1899)

If things were 1000x faster you could semi randomly try all sorts of arrangements of neural nets to see which think better.

Well you wouldn't bet all your assets because it would be an illiquid market that could only resolve in your favor in earliest 80 years.

If you're really serious about it put the money into a prediction market. Poly market has multiple AGI bets.

Should probably just short nvidia

> how I can make good on this bet.

I agree with you, and I think that's where Polymarket or similar could be used to see if these people would put your money where their mouth is (my guess is that most won't).

But first we would need a precise definition of AGI. They may be able to come with a definition that makes the bet winnable for them.

Depends on the definition, I might take that bet because under some definitions were already here.

Example: better than average human across many thinking tasks is done.

Will you take a wager of my one dollar versus your life assets? :)

Agreed. But I'd also be willing to bet big, that the cycle of "new AI breakthrough is made, AI bubble ensues and hypesters claim AGI is just around the corner for several years, bubble bursts, all quiet on the AI front for a decade or two" continues beyond the lifetime of anyone reading this message right now.

How certain are you of this really? I'd take this bet with you.

You're saying that we won't achieve AGI in ~80 years, or roughly 2100, equivalent to the time since the end WW2.

To quote Shane Legg from 2009:

"It looks like we’re heading towards 10^20 FLOPS before 2030, even if things slow down a bit from 2020 onwards. That’s just plain nuts. Let me try to explain just how nuts: 10^20 is about the number of neurons in all human brains combined. It is also about the estimated number of grains of sand on all the beaches in the world. That’s a truly insane number of calculations in 1 second."

Are humans really so incompetent that we can't replicate what nature produced through evolutionary optimization with more compute than in EVERY human brain?

If you are right, you don't have to

>I would bet all of my assets of my life that AGI will not be seen in the lifetime of anyone reading this message right now. That includes anyone reading this message long after the lives of those reading it on its post date have ended.

By almost any definition available during the 90s GPT-5 Thinking/Pro would pretty much qualify. The idea that we are somehow not going to make any progress for the next century seems absurd. Do you have any actual justification for why you believe this? Every lab is saying they see a clear path to improving capabilities and theres been nothing shown by any research I'm aware of to justify doubting that.

We are pretty close. There are some insane cutting edge developments being done in private.

Escrow

It's about the same as betting all life savings on nuclear war not breaking out in our lifetime. If AI gets created, we are toast and those assets won't be worth anything.

short oracle

my bet is we will just slowly automate things more and more until one day someone will point out when we reached "AGI"

You can make this bet functional if you really believe it, which you of course really don't. If you actually do then I can introduce you to some people happy to take your money in perpetuity.

you can do that by shorting Oracle here

And a program that can write, sound and paint like a human was 20 years away perpetually as well, until it wasn't.

I'd argue we've had more progress towards fusion than AGI.

Fusion used to be perpetually 30 years away. We’re making progress!

stop repeating that. first, it isn't true that intelligence is barely defined. https://arxiv.org/abs/0706.3639

second a definition is obviously not a prerequisite as evidenced by natural selection

Also, I like like how almost nobody takes issue with a decade time interval. If he means that current LLMs, slowly plateauing in performance, would somehow take a decade to create AI (which he calls AGI)? Where would this fantastical gain in performance come from? Or he thinks it will be a different mechanism as a basis? But then what mechanism, it should be at least real in theory by now if it were to realize in a decade time.

Basically what I mean, is that if LLMs are future real AI basis, it would take less than a decade because they are in diminishing returns today. And if it is something completely new, then what exactly? And if it is something abstract, fuzzy and hypothetical, whence did a decade number come from?

This is basically Sam Altman's "5 to 10 years in the future"(1) all over again. Not less than 5 so as not to be verified in the near future, and no need to show at least something as a prototype or at least scientific theory. And no more than 10 year so as not to scare Softbank and other investors.

(1) https://fortune.com/2025/09/26/sam-altman-openai-ceo-superin...

https://www.forbes.com/sites/jodiecook/2024/07/16/openais-5-...

https://www.tomsguide.com/ai/chatgpt/sam-altman-claims-agi-i...

The difference with fusion is that we have a very good understanding of how fusion works, and exactly what we need to figure out how to do, to make it a viable energy source. It's basically just an engineering problem, albeit a very difficult one due to the extreme conditions. AGI is more like developing warp drive. With AGI, we really have no idea how the brain works or any clue of what problems need to be solved. It's basically just like the underpants gnomes.

Phase 1: Buying more GPU to increase the number of parameters in a LLM Phase 2: ??? Phase 3: AGI

AGI may come anywhere between next week, 1000 years in the future, or never. Anyone who claims to have any idea is full of shit, because we don't even know what problems we need to solve to get there. If we develop a good model of how human cognition works at a biological level, there is at least a direction, but that isn't going to be coming out of some AI hype factory with a datacenter full of H100's making videos of anthropomorphic cats working as pastry chefs.

I can't use fusion power yet.

Several hundred million people are using LLMs every day.

There has to be at least two orders of magnitude more investment in "AI" technologies than there are in fusion techs right now.

Dunno - I've followed Moravec and Kurzweil and the predicted dates have never budged, and things have followed the predictions fairly accurately.

It would not always be a decade or two away if it was funded like the current AI cycle.

Y'all wouldn't believe this but 10 years ago AGI was a hundred years away.

Fusion is 30 years away, AGI is much closer

In the days before AlphaGo computer Go was still a decade away according to most experts.

The only relevant definition for them: https://techcrunch.com/2024/12/26/microsoft-and-openai-have-...

I wonder if like an inverse definition would work. If a user has root/admin access and all permissions/authority needed to command an AI to list the files in a directory and it simply refuses, would that be a sign of intelligence?

True AGI will be able to improve itself without human intervention or even guidance

Can perform out of distribution tasks at least around average human level performance.

The best definition so far is that it's something that will make a hundred billion dollar for Sam Altman.

Then why didn't Karpathy vibe code this?

https://x.com/GaryMarcus/status/1978500888521068818

I don’t even know what AGI is, and neither does anyone else as far as I can tell. In the parts of the video I watched, he cites several things missing which all have to do with autonomy: continual automated updates of internal state, fully autonomous agentic behavior, etc.

I feel like GPT 3 was AGI, personally. It crossed some threshold that was both real and magical, and future improvements are relying on that basic set of features at their core. Can we confidently say this is not a form of general intelligence? Just because it’s more a Chinese Room than a fully autonomous robot? We can keep moving the goalposts indefinitely, but machine intelligence will never exactly match that of humans.

I think this quote is often misapplied. The question "can a submarine safely move through water" IS a very interesting question (especially if you are planning a trip in one!).

Obviously this quote would be well applied if we were at a stage where computers were better at everything humans can do and some people were saying "This is not AGI because it doesn't think exactly the same as a human". But we aren't anywhere near this stage yet.

It also doesn't need to be good for anything to turn the world upside down, but it would be nice if it was

Fortunately I haven't heard anyone make silly claims about stochastic parrots and the impossibility of conscious computers for quite a while.

The best part of this is I watched Sam Altman say he really thinks fusion is a short period of time away in response to a question about energy consumption a couple years ago. That was the moment I knew he's a quack.

> Whenever someone brings up "AI", I tell them AI is not real AI.

You and also everyone since the beginning of AI. https://quoteinvestigator.com/2024/06/20/not-ai/

Machine learning as a descriptive phrase has stopped being relevant. It implies the discovery of information in a training set. The pre-training of an LLM is most definitely machine learning. But what people are excited and interested in is the use of this learned data in generative AI. “Machine learning” doesn’t capture that aspect.

AI is an overloaded term.

I took an AI class in 2001. We learned all sorts of algorithms classified as AI. Including various ML techniques. Under which included perceptrons.

Andrew Ng has a nice quote: “Instead of doing AI, we ended up spending our lives doing curve fitting.”

Ten years ago you'd be ashamed to call anything "AI," and say machine learning if you wanted to be taken seriously, but neural networks have really have brought back the term--and for good reason, given the results.

Arguing about the definitions of words is rarely useful.

AI is whatever is SOTA in the field, has always been.

AI is in the eye of the beholder.

Controlled fusion has always been 30 years away.

You say that as if people had been saying "10 years away" for ages, but I don't think that's true at all.

There's some information about historical predictions at https://www.openphilanthropy.org/research/what-should-we-lea... (written in 2016) from which (I am including the spreadsheet found at footnote 27) these are some I-hope-representative data points, with predictions from actual AI researchers, popularizers, pundits, and SF authors:

1960: Herbert Simon predicts machines can do all (intellectual) work humans can "within 20 years".

1961: Marvin Minsky says "within our lifetimes, machines may surpass us"; he was 33 at the time, suggesting a not-very-confident timescale of say 40 years.

1962: I J Good predicts something at or above human level circa 1978.

1963: John McCarthy allegedly hopes for "a fully-intelligent machine" within a decade.

1970: I J Good predicts 1994 +- 10 years.

1972: a survey of 67 computer scientists found 27% saying <= 20 years, 32% saying 20-50 years, and 42% saying > 50 years.

1977-8: McCarthy says things like "4 to 400 years" and "5 to 500 years".

1988: Hans Moravec predicts human-level intelligence in 40 years.

1993: Vernor Vinge predicts better-than-human intelligence in the range 2005..2030.

1999: Eliezer Yudkowsky predicts intelligence explosion circa 2020.

2001: Ben Goertzel predicts "during the next 100 years or so".

2001: Arthur C Clarke predicts human-level intelligence circa 2020.

2006: Douglas Hofstadter predicts somewhere around 2100.

2006: Ray Solomonoff predicts within 20 years.

2008: Nick Bostrom says <50% chance by 2033.

2008: Rodney Brooks says no human-level AI by 2030.

2009: Shane Legg says probably between 2018 and 2036.

2011: Rich Sutton estimates somewhere around 2030.

Of these, exactly one suggests a timescale of 10 years; the same person a little while later expresses huge uncertainty ("4 to 400 years"). The others are predicting timescales of multiple decades, also generally with low confidence.

Some of those predictions are now known to have been too early. There definitely seems to be a sort of tendency to say things like "about 30 years" for exciting technologies many of whose key details remain un-worked-out: AI, fusion power, quantum computing, etc. But it's definitely not the case that "a decade away" has been a mainstream prediction for a long time. People are in fact adjusting their expectations on the basis of the progress they observe in recent years. For most of the time since the idea of AI started being taken seriously, "10 years from now" was an exceptionally optimistic[1] prediction; hardly anyone thought it would be that soon. Now, at least if you listen to AI researchers rather than people pontificating on social media, "10 years from now" is a typical prediction; in fact my impression is that most people who spend time thinking about these things[2] expect genuinely-human-level AI systems sooner than that, though they typically have rather wide confidence intervals.

[1] "Optimistic" in the narrow sense in which expecting more progress is by definition "optimistic". There are many many ways in which human-level, or better-than-human-level, AI could in fact be a very bad thing, and some of them are worse if it happens sooner, so "optimistic" predictions aren't necessarily optimistic in the usual sense.

[2] Most, not all, of course.

Where are you because it’s sure not where I am…

AI psychosis is already here.

> AGI is already here.

cause elon musk says FSD is coming in 2017?

Andrej coined the term "vibe coding" in February on X, only 8 months ago.

He's also the guy behind FSD which is kinda turning into a scam.

??? Many developers, experienced and not, play around with vibe coding. Is your critique of him that he has tried vibe coding?

Lol, software development at a modern startup isn't even in the upper half of applied intelligence in software engineering much less global human activity/achievement. The "problems" most startups are solving are simple to the point of banality.

Computers being good/fast at automating/calculating things that people find difficult is not a new phenomenon. By your standards we have had general intelligence decades ago.

Models can't do that now, though. If they could, pretty much every human software engineer would be unemployed right now.

LLMs have continually taught me that we have vastly overestimated human intelligence

> software development at a modern startup is already the upper bound of applied intelligence.

The hubris and myopia is staggering.