Why shouldn't you expect a problem's simplicity to correlate tremendously with how well it is represented in training data? Every angle I can think of tilts in that direction. Simpler problems are easier to remember and thus repeat, they come up more often, asd they require less space/time/effort to record (which also means they are less likely to contain errors).

I have a hard time trying to conceptualize lossy text compression, but I've recently started to think about the "reasoning"/output as just a by product of lossy compression, and weights tending towards an average of the information "around" the main topic of prompt. What I've found easier is thinking about it like lossy image compression, generating more output tokens via "reasoning" is like subdividing nearby pixels and filling in the gaps with values that they've seen there before. Taking the analogy a bit too far, you can also think of the vocabulary as the pixel bit depth.

I definitely agree replacing AI or LLMs with "X driven by compressed training data" starts to make a lot more sense, and a useful shortcut.

This is true, but also misleading. We are learning that the models achieve compression by distilling higher level concepts and deriving generalized human like abilities, for example the recent introspection paper from Anthropic.

Well, that's what a LLM is. The problem is if one's mental model is built on "AI" instead of "LLM."

The fact that LLMs can abstract concepts and do any amount of out-of-sample reasoning is impressive and interesting, but the null hypothesis for a LLM being "impressive" in any regard is that the data required to answer the question is present in it's training set.

> Text copletion driven by compressed training data...solves complex problems

Sure it does. Obviously. All we ever needed was some text completion.

Thanks for your valuable insight.

Thank you for posting this. I'm struck with how there is a lot of studying of the behavior and isolating it from other assumptions and then these individual capabilities are then described as a new solution or discovered capability that would work with all of those other assumptions. This makes most all of the LLM research feel like whack a mole if the goal was to make accurate and reliable models by understanding these techniques. Instead, it's more like seeing faces in cars and buildings and other artifacts of patterns and pattern groupings and recognition of patterns. Building houses on sand, etc.

In terms of the time estimates: I've added to my global rules to never give time estimates for tasks, as they're useless and inaccurate.

I wonder if it's trying to predict what kind of estimate a human engineer would provide.

Not sure how related this is, but I've noticed it has a tendency to start sentences with usually inflated optimism and I think the idea is that if it has a tendency to intro with "Aha I see it now! The problem is" whatever comes next has a higher tendency to be a correct solution than if you didn't use an overtly positive prefix, even if that leads to a lot of annoying behavior.

I think there's two aspects to this.

Firstly, Claude's self concept is based around humanity's collective self-concept. (Well, the statistical average of all the self-concepts on the internet.)

So it doesn't have a clear understanding of what LLMs' strengths and weaknesses are, and itself by extension. (Neither do we, from what I gathered. At least, not in a way that's well represented in web scrapes ;)

Secondly, as a programmer I have noticed a similar pattern... stuff that people say is easy turns out to be a pain in the ass, and stuff that they say is impossible turns out to be trivial. (They didn't even try, they just repeated what other people told them was hard, who also didn't try it...)