Firstly, automobiles are really impressive.

Second, with that out the way, these cars are not playing the same game as horses… first, and quite obviously they have massive amounts of horsepower, which is kind of like giving a team of horses… many more horses. But also cars have an absolutely massive fuel capacity. Petrol is such an efficient store of chemical energy compared to hay and cars can store gallons of it.

I think if you give my horse the ability of 300 horses and fed it pure gasoline, I would be kind of embarrassed if it wasn’t able to win a horse race.

> I think that, if you gave me the ability to search the pre-contest Internet and a week to prepare my submissions, I would be kind of embarrassed if I didn't get gold, and I'd find the contest to be rather less interesting than I would find the real thing.

I don't know what your personal experience with competitive programming is, so your statement may be true for yourself, but I can confidently state that this is not true for the VAST majority of programmers and software engineers.

Much like trying to do IMO problems without tons of training/practice, the mid-to-hard problems in the ICPC are completely unapproachable to the average computer science student (who already has a better chance than the average software engineer) in the course of a week.

In the same way that LLMs have memorized tons of stuff, the top competitors capable of achieving a gold medal at the ICPC know algorithms, data structures, and how to pattern match them to problems to an extreme degree.

I think that's because the framing around this (and similar stories about eg IMO performances) is imo slightly wrong. It's not interesting that they can get a gold medal in the sense of trying to rank them against human competitors. As you say, the direct comparisons are, while not entirely meaningless, at least very hard to interpret in the best of cases. It's very much an apples to oranges situation.

Rather, the impressive thing is simply that an AI is capable of solving these problems at all. These are novel (ie not in training set) problems that are really hard and beyond the ability of most professional programmers. The "gold medal" part is informative more in the sense that it gives an indication of how many problems the AI was able to solve & how well it was able to do them.

When talking with some friends about chatgpt just a couple years ago I remember being very confident that there was no way this technology would be able to solve this kind of novel, very challenging reasoning problem, and that there was no way it would be able to solve IMO problems. It's remarkable how quickly I've been proven wrong.

> whereas the teams are allowed to bring a 25-page PDF

This is where I see the biggest issue. LLMs are first-and-foremost text compression algorithms. They have a compressed version of a very good chunk of human writing.

After being text compression engines, LLMs are really good at interpolating text based on the generalization induced by the lossy compression.

What this result really tells us is that, given a reasonably well compressed corpus of human knowledge, the ICPC can be view as an interpolation task.

As someone who has been to the ICPC finals around a decade ago I agree that the limited time is really the big problem that these machine learning models don't really experience in the same way. Though that being said these problems are hard, the actual coding of the algorithms is pretty easy (most of the questions use one of a handful of algorithms that you've implemented a hundred times by the time you're in the finals) but recognizing which one will actually solve the problem correctly is not obvious at all. I know a lot of people that struggled in their undergrad algorithms class and I think a lot of those people given the ICPC finals problems would struggle even with being able to research.

It doesn't matter how many instances were running. All that matters is the wall clock time and the cost.

The fact that they don't disclose the cost is a clue that it's probably outrageous today. But costs are coming down fast. And hiring a team of these guys isn't exactly cheap either.

The human teams also get limited to one computer shared between 3 people. The models have access to an effectively unbounded number of computers.

My argument does feel a bit like the “Watson doesn’t need to physically push the button” equivalents from when that system beat Jeopardy for the first time. I assume 5 hours on a single high-end Mac would probably still be enough compute in the near future.

I think your analogy is lacking. Human brain is much more efficient, so it is not right to say "giving a human team a week instead of five hours". Most likely, the whole OpenAI compute cannot match one brain in terms of connections and relations and computation power.

I think your assessment is spot on. But I also think there's a bigger picture that's getting lost in the sauce, not just in your comment but in the general discourse around AI progress:

- We're currently unlocking capabilities to solve many tasks which could previously only be solved by the top-1% of the experts in the field.

- Almost all of that progress is coming from large scale deep learning. Turns out transformers with autoregression + RL are mighty generalists (tho yet far from AGI).

Once it becomes cheap enough so the average joe can tinker with models of this scale, every engineering field can apply it to their niche interest. And ultimately nobody cares if you're playing by the same rules as humans outside of these competitions, they only care that you make them wealthy, healthy and comfy.

the end game is that running similar tasks at any moment time and place.

If you want to play that game, let's compute how much energy was spent to grow, house and educate one team since they were born, over 20 years against how much was spent training the model.

Yes yes given this why didn't it do better and isn't it embarrassing to have done it through statistical brute force and not intelligence.

Ha so true. I was so tempted to copy and paste a problem into GPT5 and see what it would say

> it succeeded on the 9th submission

What's the judgement here? Was it within the allotted time, or just a "try as often as you need to"?

1. What was your prompt? 2. Why did you give it to GPT-5 instead of GPT-5 Thinking or GPT-5 Pro?

Sounds like a bug. Did you try it again (or with another leading-edge model) and get a similar result?

So outside of human intervention, I don't think the specifics really matter. What this means is that it is possible and that this capability will in time be commoditized.

This is helpful in framing the conversation, especially with "skeptics" of what these models are capable of.

I don't see that much reason to be skeptical since this basically lines up with the trend we've been seeing in their performance.

Claude, ChatGPT, and Gemini on a team.

I'm not sure how it would play out, but at least when you let them talk to each other they tend to get very technical very fast.

Actually collegiate means that the contestants are in college.

Why the AI hate? How is it different from sharing your knowledge with another individual or writing a book to share it?

> AI companies are not paying anyone for that piece of information

So? For the vast majority of human existence, paying for content was not a thing, just like paying for air isn't. The copyright model you are used to may just be too forced. Many countries have no moral qualms about "pirating" Windows and other pieces of software or games (they won't afford to purchase anyway.) There's no inherent morality or entitlement for author receiving payment for everything they "create" (to wit, Bill Gates had to write a letter to Homebrew Computer Club to make a case for this, showing that it was hardly the default and natural viewpoint.) It's just a legal/social contract to achieve specific goals for the society. Frankly the wheels of copyright have been falling off since the dawn of the Internet, not LLM.

I for one welcome advancement of science and mathematics from our AI overlords

In 2015 SotA models blew past all expectations for engine performance in Go, but that didn't translate into LLM-based Code agents for another ~7 years (and even now the performance of these is up for debate). I think what this shows is that humans are extremely bad at understanding what problems are "hard" for computers; or rather we don't understand how to group tasks by difficulty in a generalizable way (success in a previously "hard" domain doesn't necessarily translate to performance in other domains of seemingly comparable difficult). It's incredibly impressive how these models perform in these contests, and certainly demonstrates that these tools have high potential in *specific areas* , but I think we might also need to accept that these are not necessarily good benchmarks for these tools' efficacy in less structured problem spaces.

Copying from a comment I made a few weeks ago:

> I dunno I can see an argument that something like IMO word problems are categorically a different language space than a corpus of historiography. For one, even when expressed in English language math is still highly, highly structured. Definitions of terms are totally unambiguous, logical tautologies can be expressed using only a few tokens, etc. etc. It's incredibly impressive that these rich structures can be learned by such a flexible model class, but it definitely seems closer (to me) to excelling at chess or other structured game, versus something as ambiguous as synthesis of historical narratives.

edit: oh small world! the cited comment was actually a response to you in that other thread :D

Even Sam Altman himself thinks we’re in a bubble, and he ought to have a good sense of the wind direction here.

I think the contradiction here can be reconciled by how these tests don’t tend to run on the typical hardware constraints they need to be able do this at scale. And herein lies a large part of the problem as far as I can tell; in late 2024, OpenAI realized they had to rethink GPT-5 since their first attempt became too costly to run. This delayed the model and when it finally released, it was not a revolutionary update but evolutionary at best compared to o3. Benchmarks published by OpenAI themselves indicated a 10% gain over o3 for God knows how much cash and well over a year of work. We certainly didn’t have those problems in 2023 or even 2024.

DeepSeek has had to delay R2, and Mistral has had to delay Mistral 3 Large, teased within weeks back in May. No word from either about what’s going on. DS is said to move more to Huawei and this is behind a delay but I don’t think it’s entirely clear it has nothing to do with performance issues.

It would be more strange to _not_ have people speculate about stagnation or bubbles given these events and public statements.

Personally, I’m not sure if stagnation is the right word. We’re seeing a lot,of innovation in toolsets and platforms surrounding LLM’s like Codex, Claude Code, etc. I think we’ll see more in this regard and that this will provide more value than the core improvements to the LLM’s themselves in 2026.

And as for the bubble, I think we are in one but mostly because the market has been so incredibly hot. I see a bubble not because AI will fall apart but because there are too many products and services right now in a golden rush era. Companies will fail but not because AI suddenly starts failing us but due to saturation.

People pattern match with a very low-resolution view of the world (web3/crypt/nfts were a bubble because there was hype, so there must be a bubble since AI is hyped! I am very smart) and fail to reckon with the very real ways in which AI is fundamentally different.

Also I think people do understand just how big of a deal AI is but don't want to accept it or at least publicly admit it because they are scared for a number of reasons, least of all being human irrelevance.

There is a clear difference between what OpenAI manages to do with GPT-5 and what I manage to do with GPT-5. The other day I asked for code to generate a linear regression and it gave back a figure of some points and a line through it.

If GPT-5, as claimed, is able to solve all problems in ICPC, please give the instructions on how I can reproduce it.

My response simply is that performance in coding competitions such as ICPC is a very different skillset than what is required in a regular software engineering job. GPT-5 still cannot make sense of my company's legacy codebase even if asked to do the most basic tasks that a new grad out of college can figure out in a day or two. I recently asked it to fix a broken test (I had messed with it by changing one single assertion) and it declared "success" by deleting the entire test suite.

Two days ago I talked to someone in water management about data centers. One of the big players wanted to build a center that consumed as much water as a medium town in semi arid bushland. A week before that it was a substation which would take a decade to source the transformers for. Before that it was buying closed down coal power plants.

I don't know if we're in a bubble for model capabilities, but we are definitely hitting the wall in terms of what the rest of the physical economy can provide.

You can't undo 50 years of deffered maintenance in three months.

Well, the supposed PhD-level models are still pretty dumb when they get to consumers, so what gives?

The last time I asked for a code review from AI was last week. It added (hallucinated) some extra lines to the code and then marked them as buggy. Yes, it beats humans at coding — great!

the wall is how we need to throw trillions of hardware to do "breakthroughs", LLM uses the same algorthm from last few years. We need a new algorthm breakthrough otherwise buying hardware to increase intelligence isn't scalable.

> So this year SotA models have gotten gold at IMO, IoI, ICPC > Yet the most reposted headlines and rethoric is "wall this", "stangation that", "model regression", "winter", "bubble", doom etc.

this is narrow niche with high amount of training data (they all buy training data from leetcode), and this results are not necessary generalizable on overall industrial tasks

People are having a tough time coping with what the near future holds for them. It is quite hard for a typical person to imagine how disruptive and exponential coming world events are like Covid showed.

This comment makes me think. What did previous winners of these competition go on to do in their lives? Anything spectacular?

Don't worry, they're just stochastic parrots copying answers from Stack Overflow. ;)

It's important to look closely at the details of how these models actually do these things.

If you look at the details of how Google got gold at IMO, you'll see that AlphaGeometry only relies on LLMs for a very specific part of the whole system, and the LLM wasn't the core problem solving system in play.

Most of AlphaGeometry is standard algorithms at play solving geometry problems using known constraints. When the algorithmic system gets stuck, it reaches out to LLMs that were fine tuned specifically for creating new geometric constraints. So the LLM would create new geometric constraints and pass that back to the algorithmic parts to get it unstuck, and repeat.

Without more details, it's not clear if this win is also the Gpt-5 and Gemini models we use, or specially fine-tuned models that are integrated with other non-LLM and non-ML based systems to solve these.

Not being solved purely by LLM isn't a knock on it, but with the current conversations going on today with LLMs, these are heavily being marketed as "LLMs did this all by themselves", which doesn't match with a lot of the evidence I've personally seen.

These are college-student or occasionally grad-school programmers who qualified to enter the ICPC World Finals, generally by performing sufficiently well at a regional championship to qualify. You can read actual rules here (see "Advancing to the ICPC World Finals"):

https://icpc.global/regionals/rules

I don't know what you mean by "elite", and there are certainly plenty of teams at the World Finals that are not especially competitive, and there certainly many elite programers who don't qualify for various reasons (most obviously by being the wrong age or not in the right stage of school or having already attended too many times), but I find it hard to believe that there aren't enough "elite" programmers present to make the winning teams be genuinely elite.

Compare to, say, the Olympics or pretty much any academic olympiad. There are many people and teams at the Olympics who are not remotely competitive with the winners.

The ICPC has plenty of elite competitive programmers. It's an activity that "peaks" in importance around college, and not many keep training a lot after participating.

Every year there are multiple "Legendary Grandmasters" in the competition. That's >3000 Elo in Codeforces. I'd estimate it takes a similar level of skill/effort as becoming a Chess Grandmaster.

And even those that aren't at that level are very competent at it. The average ICPC participant is likely "smarter" than the average MIT/Harvard CS student for some reasonable measure of "smarter".

I've competed in these contest before. There are probably more difficult than what we can call _elite_ competitive programmer

note: my team only passed the first 2 rounds, far from bragging about my skills here :)

ICPC world finals questions are not easy. Idk what your talking about

Let's bookmark this comment and check again next year, if the freely available models will be able to do it for a few dollars.

Not sure by what metric you compare the difficulty, but regardless of the hardness of the problem, IIRC, ICPC requires 100% correctness on test cases to score a problem (even failing one means you don't get the score,) but IOI would admit fractional scores (correct me if I am wrong.)

What makes you say that they are easier? Are there more people who manages to solve a problem from ICPC than from IOI?

How do you compare those?

There were at least 2 very simple problems in IOI this year.

I haven't read the ICPC problem set, and perhaps there are some low-hanging fruits, but I highly doubt it.

I think part of it depends on whether you see AI progress as research or product.

Models drop in price x10 each year. Us, common folk, getting access to these kinds of models is just a matter of time.

The bleeding edge behind closed doors token burning monsters of 2023 are bad compared to the free LLMs we have now.

I believe it was Sundar in an interview with Lex who said that the reason they haven't developed another Ultra model is because by the time it is ready to launch, the flash and pro versions will have already made it redundant.

Ok but if they can pump those compute and get science/math advancements it's worth something even if the costs are very high

"It's now turned into a whole marketing circus (maybe to justify these ludicrous billion-dollar valuations?)."

Yes theres an entire ecosystem being built up around language models that has to stay afloat for another 5 years at least, to hope for a significant breakthrough.

chemical space and in vivo testing is a different beast

Where is this magic ICPC competition answers database that they're using?

Unreliable doesn't mean unusable. I'm finding it harder and harder to believe people are actually trying to use them and saying they are useless.

If you can chop your problem up and give little tedious parts of your bigger task it's starting like doing code review for a new grad instead of coding. And they're getting more reliable and the parts you give it can be bigger and bigger. I wish there was a way to stop this but I don't think it's going to.

/> The point is, what are THEY investing a trillion dollars in?

Who cares? I won't be a customer until I see a return on my investment [in them].