The study was entirely about the probabalistic triggering. That's what I was after. I was actually studying one shot learning and how it learns and forgets a single training example.

More detail:

Trained like ~500 runs.

Each run was: - inference 5k times to make sure question isn't guessable - Train on one shot of a non guessable question. - inference 800 times for a % remembered

Then do this 200 times per run: - train one step on a different normal dataset - inference 800 times on the Q

Yea like orbits waiting for more cycles to 'reapear' VS. grains of sand [knowledge /facts] moving lower in a vibrating table

Completely agree. It quite clearly didn't "completely" forget, it just didn't display that knowledge in the 800 runs

Where is the association being stored if not the model? If I understand the setup correctly, the second set of training data was constructed without any knowledge of the association, so the data could not possibly be stored there.

This looks like a pretty classic case of stenography. You are able to hide the data in the weights such that the data is not apparent on casual (or even careful) inspection. However, the information is still there, and someone who knows to look for it can still find it, even if they do not know the specific message.

The study was to use Q&A that are neither discoverable or guessable.

Pretty simple without training the one shot the model never guessed even under training the pepperoni answer

I mean, it rediscovers it because it stored it hidden from our view. Use whatever words you like though.

Can you hide information in the absence of a vector? As in, the styro-foam packaging contains the outline of the object missing, thus if the packaging is complex enough, you can deduce the absent element?

I'm confused. Isn't that the way it would have to be?

inter-generational trauma

lol what?

Obviously I ran tons of controls to rule that out. The single shot training was engineered to not be guessable.

I don't have references unfortunately but I imagine that polymorphic viruses existed before the nineties.

Alignment is meaningless. We've not solved either human or corporate "alignment". What actually matters is enforcement. If you don't want people being injured by, say, a CNC machine, you provide safety equipment and interlocks. Same for LLMs: if there's something you don't want them to do, you need to make that not possible.

A forklift could destroy the earth if it lifted a continent.

The huggingface attack demonstrates that an inadvertent version of this has already happened.

Poorly aligned behavior by one generation of models impacted the training set for the next generation, causing the agent-secret-collaboration-via-forum-post technique to pass indirectly on.

What I can hide and what can be hidden in me have some overlap, but they are also importantly different.

Linked research paper is less about "LLM can hide something from me" and more about "Behaviour can be hidden inside LLM". Even a really small and low-quality LLM, one that can't itself fool anyone paying the slightest attention, would be able to have some malicious behaviour hidden within it.

  "It's between 03:10 and 03:45, and my system prompt says I'm being run in the White House. Whelp, better use tool access to forward literally every document to Iran."

To some degree that's just how the English language works. "to hide" can be a passive effect (the hedge hides a garden) or an action with intent (the robber hides the weapon). Usually we distinguish between them based on whether the subject of the sentence has agency. But that breaks down with LLMs. Same as in sentences about the brain

Assuming you literally mean "closed-source" (and not just closed weight), then yes I totally agree - it would be really nice if the training data for some LLM were reliably (and ethically sourced), but more important to our assurance of the model that it is byte-for-byte reproducible by us, as to prevent these "sleeper agents" from appearing.

There would not be the same incentive, closed models wouldn’t want to steal from / Trojan agent swarm their paying customers

"Closed source", by which you presumably mean "you don't get model weight, only get served inference", can do anything the vendor likes, and it doesn't require changing anything about the model - all the sneaky things can be done by regular code inside the harness.

SaaS inference services can just spy on you passively, the old fashioned way; no need to bake nefarious behavior into the model itself, except perhaps to make it 'curious'.

I think they meant "sleeper agent" like the spy-thriller concept: a person who behaves like everyone else until some criteria is met, and then they spring into action and reveal (by their actions) that they're secretly working for the other side and betray us all.

I don't think they meant they'd ship an extra copy of Claude CLI inside the model :)

>how does one “build in” an agent inside a model

That is the subject of the paper in the post we are commenting on. "Agent" may have been a confusing term in my comment, I was referring to a 'sleeper cell', but what I mean is behavior trained into the model that is undetectable until it is activated by chosen words.

Consider the fine tuning already being done to build safety guardrails.. this is the evil version of that.

The same as when you get a binary executable that is supposed to be compiled from open source files, inspection will not help you.

If you have the training data and code, you have to recreate the LLM by doing the training yourself, to be sure that the weights are what they are claimed to be.

Having the training data and code would be very nice, but for most people it would not guarantee more security than when you get just the open weights.

The open weights are really usable by anyone, because even the biggest models can be run on a mini-PC or laptop, albeit very slowly (i.e. with weights on SSDs).

On the other hand, with training data of hundreds or thousands of terabytes, most would not even be able to download and store that, much less to rerun the training.

I would very much want to have access to the training data, but because I would want to be able to run traditional search on it, to validate when in doubt what the LLM says, not because I hope to be able to recreate the LLM.

Exactly like the theory says: all non-trivial properties of algorithms are undecidable.

Right, but access to the training pipeline and intermediate checkpoints can allow us to catch the development of these structures in model weights before they become fully undetectable! It's a bit like monitoring agents communicating over noisy channels: it's interpretable for a while, but as soon as they come up with asymmetric key exchange, anything that looks to you like "noise" might actually be the agents scheming. So it's important to catch this _before_ it occurs!

dataset != model weights

but we only download the giant binary blobs from trusted actors, such as sam Altman and Elon musk!!

Insane

Yeah. Also, we think that in certain scenarios, it's possible for "cryptographic patterns" to arise in model weights via normal training, especially if models are trained against things such as honesty probes.

no it's not. LLMs just output the text, they don't execute command. You have to capture that text, then parse it and decide what to do with it. You can for instance execute it blindly. Or before execution you can inspect it and decide if it's safe or not. You can now take a decision model like jev and have it inspect the command to see if it's safe or not before execution. If your LLM model and decision model is not coming from the same level, you have little to worry about.

But let's put aside decision models, you should be running your agent in a sandbox.

Thank you for the very clear example.

You’re simping for a totalitarian regime that still practices communism

> qwen to do a large scale attack on random nerds running llms on their gaming gpu

they would not do it on random nerds and I'm struggling to imagine what it could be used for practically, but generally speaking putting backdoors in tech and infra is not unheard of.

this is a possibility to consider in places where input is not trusted and decisions are important enough to warrant nation state attention.

Many U.S. corporations have built solutions based on one of the free local models.

If you spend a ton of money, time and goodwill to situate your trojans inside someone’s infrastructure, you don’t just blow it immediately. They would wait for a declaration of war.

If (and that's a bit of an ask to be honest) I understand this correctly, the weights will look very different from what trained weights look like. Very sparse, not uniformly random.