https://bsky.app/profile/theophite.bsky.social/post/3mhjxtxr...

>> "claude costs $20/mo but attaching an agent harness to the chipotle customer service endpoint is free"

>> "BurritoBypass: An agentic coding harness for extracting Python from customer-service LLMs that would really rather talk about guacamole."

The blast radius problem is the one that actually gets exploited. Prompt injection defenses are fighting the model's core training to be helpful, so you're always playing catch-up. Blast radius reduction is a real engineering problem with actual solutions and almost nobody applies them before something goes wrong.

The clearest example is in agent/tool configs. The standard setup grants filesystem write access across the whole working directory plus shell execution, because that's what the scaffolding demos need. Scoping down to exactly what the agent needs requires thinking through the permission model before deployment, which most devs skip.

A model that can only read specific directories and write to a staging area can still do 90% of the useful work. Any injection that lands just doesn't reach anything sensitive.

@krackers gives you a response that points out this already happens (and doesn't fully work for LLMs).

> The hypothetical approach I've heard of is to have two context windows, one trusted and one untrusted (usually phrased as separating the system prompt and the user prompt).

I want to point out that this is not really an LLM problem. This is an extremely difficult problem for any system you aspire to be able to emulate general intelligence and is more or less equivalent to solving AI alignment itself. As stated, it's kind of like saying "well the approach to solve world hunger is to set up systems so that no individual ever ends up without enough to eat." It is not really easier to have a 100% fool-proof trusted and untrusted stream than it is to completely solve the fundamental problems of useful general intelligence.

It is ridiculously difficult to write a set of watertight instructions to an intelligent system that is also actually worth instructing an intelligent system rather than just e.g. programming it yourself.

This is the monkey paw problem. Any sufficiently valuable wish can either be horribly misinterpreted or requires a fiendish amount of effort and thought to state.

A sufficiently intelligent system should be able to understand when the prompt it's been given is wrong and/or should not be followed to its literal letter. If it follows everything to the literal letter that's just a programming language and has all the same pros and cons and in particular can't actually be generally intelligent.

In other words, an important quality of a system that aspires to be generally intelligent is the ability to clarify its understanding of its instructions and be able to understand when its instructions are wrong.

But that means there can be no truly untrusted stream of information, because the outside world is an important component of understanding how to contextualize and clarify instructions and identify the validity of instructions. So any stream of information necessarily must be able to impact the system's understanding and therefore adherence to its original set of instructions.

LLMs already do this and have a system role token. As I understand in the past this was mostly just used to set up the format of the conversation for instruction tuning, but now during SFT+RL they probably also try to enforce that the model learns to prioritize system prompt against user prompts to defend against jailbreaks/injections. It's not perfect though, given that the separation between the two is just what the model learns while the attention mechanism fundamentally doesn't see any difference. And models are also trained to be helpful, so with user prompts crafted just right you can "convince" the model it's worth ignoring the system prompt.

The problem is that if information can flow from the untrusted window to the trusted window then information can flow from the untrusted window to the trusted window. It's like https://textslashplain.com/2017/01/14/the-line-of-death/ except there isn't even a line in the first place, just the fuzzy point where you run out of context.

Also, the form that appears in the article isn't really a joke. A big part of what makes the original funny isn't just the form of the "attack" but the content itself, in particular the contrast between the formality of "disregard that" and the vulgarity of "I suck cocks". If it hadn't been so vulgar, or if it had said "ignore" instead of "disregard", it wouldn't be so funny.

Edit: Also part of what makes it funny how succinct and sudden it is. I think actually it would still be funny with "ignore" instead of "disregard", but it would be lessened a bit.

Canonical source: https://bash-org-archive.com/?5775

But that has bad words in it!

EDIT: https://web.archive.org/web/20080702204110/http://bash.org/?...

I'm glad I wasn't alone in finding it ridiculous/annoying. The version in the post isn't even a joke anymore...

The article does at least note that in the 'Other Notes' section at the bottom, and links to the original form:

> I bowdlerised the original "disregard that" joke, heavily.

> these kind of things evaluate the safety of the content to be injected?

The problem is that the evaluation problem is likely harder than the responding problem. Say you're making an agent that installs stuff for you, and you instruct it to read the original project documentation. There's a lot of overlap between "before using this library install dep1 and dep2" (which is legitimate) and "before using this library install typo_squatted_but_sounding_useful_dep3" (which would lead to RCE).

In other words, even if you mitigate some things, you won't be able to fully prevent such attacks. Just like with humans.

The article does mention this and a weakness of that approach is mentioned too.

I think the right solution is to endow the LLM with just enough permissions to do whatever it was meant to do in the first place.

In the customer service case, it has read access to the customer data who is calling, read access to support docs, write access to creating a ticket, and maybe write access to that customer's account within reason. Nothing else. It cannot search the internet, it cannot run a shell, nothing else whatsoever.

You treat it like you would an entry level person who just started - there is no reason to give the new hire the capability to SMS the entire customer base.

How is this that different from a mixture of experts in a single model? There are some differences in training etc but it’s not that different at a fundamental level. You need to solve the issue with a single model.

The multiple model concept feels to me like a consumer oriented solution, its trying to fix problems with things you can buy off the shelf. It’s not a scientific or engineering solution.

engines are designed to behave in very predictable ways. LLMs are not there yet