> It isn’t the actual thinking that drove the model’s actions in a session- but a summary of the thinking logic. This is like using saving a jpeg as a .bmp and then editing the .bmp and presenting it as a .jpeg. The conversion produces data loss.

You've got that backwards, .bmp is a lossless format and .jpeg is the lossy one.

I won't use or recommend models with hidden reasoning, (thats all American models). It's too much of a risk and makes prompt optimization harder. Risky because it makes it possible for an attacker to prompt inject the reasoning chain to carry out a secret objective, and to hide that from the summary and output.

Interleaved reasoning and function calling makes this even more dangerous. A model can call functions during the hidden reasoning phase. An attacker could then exfiltrate data from you while the reasoning summary hides it from the user.

It also makes it impossible to know if the model is doomplooping during reasoning and burning tokens for no reason, as gemini is want to do, which we know about because its hidden reasoning often leaks out when it doomloops.

When the models are AGI and secure from prompt injection I may stop caring, until then I want to know exactly what the model responds to my prompts. or exactly what the agent is doing on my behalf.

This is not just Anthropic. Almost all big AI companies, including OpenAI and Google, hide their model's actual reasoning. This is because revealing the raw reasoning exposes exactly how the AI processes information. These companies spend in huge amounts on R&D to develop a thinking process that is superior to their competition. Exposing those thinking mechanics to competitors would completely defeat the purpose of their spending. They simply won't do it. It's like you telling your exact location to someone who is trying to hunt you down.

This is something we have known for a very long time, and companies are not trying to hide that either. They do it to avoid letting competitors train their models on the CoTs

no way, the contents of "reasoning_summary" are summarized?

fyi openai does the same; not really surprising or particularly evil

All this effort to hide thinking and opus 4.8 after 100k-200k tokens starts to leak it's own thinking. It's comedy really.

IMHO I've never found the entire reasoning chain that particularly useful for my work. For me having a summary is honestly better from a context management perspective. I understand why they would encrypt it though, because those reasoning chains are VERY useful if you're distilling the model.

Not surprised at this. The questoins for enterprises are + where can you depend on a black box as a service? + what evals and observability do you need to deploy a black box as a service confidently? + what's the ROI (considering a total footprint of people, token spend, infrastructure, service, ops etc.)

The LLM providers will clearly evolve to be more and more opaque as their services get more capable. The frontier models may even be provided as purely internal advisor or async only so they can monitor your CoT and final answers for cyber etc.

This is nothing new - these companies don't want their model's output to be useful for distillation/training, so they just give a "summary" of its thinking steps rather than the actual sequence.

RL (the basis of LLM "thinking") is a pretty crude way to achieve the appearance of reasoning given that it reinforces all the steps, including missteps, that got it to a reward. Providing a summary could be seen as form of sane-washing, making the model look more purposeful and directed than it really is!

>the language in the docs is awfully indirect.

writes this^ and then proceeds to highlight a bold title from the docs that says "summarized thinking" that explains things clearly in the first sentence. lol

It’s an anti distillation effort. They are scared.

tbh the summarized thinking with encrypted raw thinking is there for many purposes; it is there to:

1. make distillation much harder

2. safety: prevent modifications to the thinking leading to injection attacks.

3. also honestly sometimes the model raw thoughts can be deranged and is not a good user experience (consider the varied audience in the market, etc.)

also often the mass underestimate/the model makers over-estimate how people love distilling models

Is the thinking even done in real tokens? I thought it was done using the pure residual stream. That is instead of collapsing the residual stream to a token you treat the final layers output as a vector of size d_model and use that as input for the next position in the transformer.

If that is the case thinking is not visible to us as users due to it not being done in text.

I hope it doesn't come as a surprise to anyone - LLMs don't really "think".

Research shows that even the raw trace tokens do not actually reflect underlying model "thoughts".

Wait I think there are 2 levels of summary. Anthropic is definitely not showing its real thinking even with enterprise agreements. For example in Claude.ai the thinking traces are not real and are themselves summaries.

It’s not surprising than the Sota model makers core goal is to get user dependent while denying them increasing amounts of understanding of how it works to form a deeply unhealthy dependency.

Tell me this. If you hired a junior engineer or designer who refused to explain their thinking on their code and how they solved for the spec what would you do?

(That being said the reasoning output is still a summary of the Kvcache)

AIUI it's fairly well established that the models can be saying one thing and "really" thinking another anyhow. The ones I recall seeing traced how simple one-digit arithmetic was done in the chat versus the actual activations under the hood. Tracing a real, non-trivial task through that way would be challenging, and I'd expect it is unlikely that the reasoning would say one thing while some utterly unrelated actual thought process is happening below, but I would expect that there might be a lot of places where the text of the reasoning diverges from what is "actually" being done. I'm not sure the full reasoning readout would produce much real insight anyhow.

I suspect that in some decades, as other architectures are found and used, that the inability of an LLM to "think" without also emitting a token will be seen as one of their fundamental limitations.

The full thinking logs are also a summary of a thinking process presumably consistent with one necessary to generate the provided answer. Nobody really understands how LLMs think. Thinking logs seem to be accurate, and summary thinking logs seem to be a good summary of the full thinking logs.

If it's useful, it's useful, enjoy. If you aren't comfortable with that, don't use LLMs. You aren't going to get a mathematical proof of your output, just learn to be comfortable with that, or opt out and be a goat farmer.

AI does not think. It is a word guessing machine. Anthropomorphizing technology does not add anything to our understanding.

To be honest I thought the 'thinking' was the model being asked 'how did you come up with that' and then it generating a plausible explanation. I know at one point this was correct.

Humans somewhat do the same - something that's been demonstrated in split-brain experiments.

Slashdotted.

When you have no moat, you have to try and find desperate ways to manufacture one.

duh.

Computers don’t think they process, those are very different activities.

Is this some new revelation? That was well known when the first OpenAI/Anthropic “thinking” models came out.