Doesn’t matter if there’s a rechable local maximum that happens to be comfortably beyond human level. Indeed it would be very surprising if a thing capable of RSI just happened to get stuck at human level or slightly above or below, even though its constraints are entirely different from those evolution had to work with when it created us!

You don't expect to find the global maximum. It will just always be there as an elusive target.

I doubt the human brain is at global maximum.

I doubt that any algorithms would be able to find a global maximum in such a large multidimensional non-convex space

Would the loss at the global maxima be much lower that the loss at most local maxima?

> true optimal

True optimal what? Optimal intelligence? What would that be?

What local maxima are you seeing? That would imply that progress has stalled, which doesn't appear to be the case where I'm looking.

Also, RSI is obviously guided. If only guided by "it's not giving results so we'll try something else". To require that RSI happens in a black box for it to count would be arbitrary, and also not how anyone is going to do it.

Sounds like a classic case of Repetitive Stress Injury. You could try balancing it out by using your left thumb to scroll through articles about Artificial Stupidity.

Improvement means being able to do more complicated things more reliably. Relatedly, it means being able to learn to do new things with fewer and fewer examples. We are running out of easily verifiable or simulation-friendly or data-rich domains for LLMs to conquer. (Note that I didn't say "simple" or "easy" domains.)

Models that are strong enough to improve themselves without a human (I.e. ai researcher) in the loop

The trivial idea that there are some benchmarks for a tool and there can be even better inexplicit ones, and if the tool is capable of modifying itself towards better benchmarks results there could be a recursive climbing of those benchmarks through iterations of tool versions which are outputs of former tool versions.

It's only optimized to make prior knowledge useful in new situations.

> this logic gate should not hack other systems

well .. it only applies a rule to inputs

can we impose this of authors "this book should only be causing good things when it is read and knowledge used"

I don’t disagree, but unfortunately a not insignificant part of humanity seems to think that their happiness primarily depends on knowing that other people are suffering harder than they are. So that’s perhaps not the right thing to optimise for.

> AI should be optimized to make all humans healthy, happy and safe without exception.

No. AI should be optimized for the one true good: maximum market returns. Personally I think that will only be achieved once humans are removed from the market system and decommissioned.

Not yet AGI, and Richard Sutton has left to start a new lab

I’ve had this question on my mind at least once a month for what feels like years at this point.

I don’t feel like monitoring Twitter to see how it’s going though.

Recursive self-improvement is not necessary for AGI. Humans are general intelligences even though we cannot RSI.

I certainly do see one side of the debate moving goal posts…

Are you a visitor from the past?

If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.

Money can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"

Obviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.

The alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement.

There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.

Taxation will solve that. Any company that demonstrates even a glimmer of real AGI will get government money poured into it. They might even disclose it.

def self_improve_more():

     if self_improved():

          self_improve_more()

     else:

          see_tony_robins_and_buy_tapes()

why can't recursive self improvement not improve efficiency ? token per watt ?

The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.

Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.

I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.

Learning from training data is technically self-improvement but not the sort that is typically meant in this context.

> AIs already improve themselves via training

Marginally. Model collapse is still a problem. Continuous learning is still a problem.

For AI to make a big leap we need a big break through.

Thing is, it depends on whether llms + reinforcement can self-improve in principle. Learned recently that cognitive scientists, before the transformer & llms, were studying the possibility that thinking and learning might be based on some kind of prediction, i.e. something similar to token prediction, and I quite suddenly became less skeptical about the possibilities of llms. (Some will say I’m late to the party of course.) But if knowledge to date has been accumulated in a process quite like “chain of thought” in llms, then I don’t see any reason that computers won’t self-improve in the near future.