I think your reply has a somewhat familiar structure -- "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". You might be completely correct! But these sorts of claims push the onus back onto the other person, without accepting any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?

Maybe Gemini is loosey goosey on purpose so we angrily correct it - then feed something on the back end that trains a different model?

It's so abysmally bad on Google search... and it's free. Isn't Google the great pioneer of the product is us?

And if you're building something of any importance, you need to have verification steps at checkpoints. It's honestly just engineering.

Weak models tasked with review can catch a decent amount of the mistakes that weak models make and help them be much better, especially if you have them verify against authoritative sources. Strong models make far fewer mistakes to begin with. And, for the mistakes they do make, a swarm of reviewers (same model or somewhat weaker, reviewed by the stronger model) can really help reduce the error rate further.

It may be tiresome, but it's true. It doesn't refute the article's premise thou, only retesting with newer models would and that would be the same tiresome requirement you already called out

Yes, that's an annoying predicament, but we come by it honestly.

Limits of scientific method in the face of exponential takeoff, IMO, and kind of proves the opposite result (RSI appears to be here)

People were feeling doomtastic about nuclear weapons during the Cold War as well.

Listen to the words of this song written in 1969: https://www.youtube.com/watch?v=r2JcxHX-8Xc

  When every man is torn apart
  With nightmares and with dreams
  Will no one lay the laurel wreath
  When silence drowns the screams
  Confusion will be my epitaph
  As I crawl a cracked and broken path
  If we make it, we can all sit back and laugh
  But I fear tomorrow I'll be crying

You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?

*EDIT*: There are about 5 replies to this comment making roughly the same point. I'm not sure which to reply to, so I'll just reply here.

I'm not claiming that our execution as a species around nuclear weapons has been flawless. I'm not claiming that we are out of the woods with regard to nukes. I'm just trying to push back against defeatism and fatalism. Some felt a sense of inevitable doom during the Cold War. It's been decades now, and nuclear doom still isn't here! Our situation is dire, but not hopeless.

Is this a reference to capitalist realism? If so have we evolved a new trend?

Yeah. Humans will shut down AI research around the same time they dismantle their nuclear weapons and agree on the causes of climate change.

I don't think the end of the world would even be a bad thing; a lot of people assume that it would be disastrous but I think it's much more likely that it the "end" would just be a fragmentation that would be pretty uncomfortable, but not to the point of being horrific. I actually think that people are more resilient and creative than they think and if they were faced by a global economic collapse so strong that big tech would go bankrupt, they would bounce back pretty quickly. It's just that we've seen so many movies about the end and have been conditioned to believe that we're helpless that we think otherwise.

What quote from 2024 are you thinking of here?

> The insiders that said every tech workers would be unemployed in 6 months and every white colar would be unemployed in 12 months like 2 years ago?

They might not have predicted the economy but the scores are going up and up. And I think are really smarter

I don't think RSI is typically used to describe self-improving agents - it's about improving the model itself, and its performance in agentic tasks.

Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.

The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.

Training has become very complex.

The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.

When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.

It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.

Do you think the final product of research is papers?

$10m a month is nothing to these companies

theres very little danger of anyone letting it go without a purpose

We literally just saw how OpenAI’s model got out and hacked HuggingFace

Sure, but the point is that the labs use more powerful internal models for research work, not public models. Public models tend to lag the internal frontier by a decent margin, and are constrained in other ways by monitoring. It’s just not a useful indicator.

I'm sure they also use the Gemini coding agent to write new Gemini code. But that's far short of self improvement.

I am pretty sure Tim also used Siri for the development of the next Siri(you know to set up the alarm clock)