Why do you say that? I thought this pattern was well established, or are you aware of known issues with it?

[dead]

Correct me if I’m wrong, but from my experience in this space in order for a model to exercise judgment it must force itself to operate in a strict chain of thought mode. Since all LLMs are predictive creatures, I started to care a lot more about my judgment settings, the transparency of them, and the presence of a judgment loop in either the development or functionality of an application built these days.

Not exactly sure where I’m going with this, but my work with creating penetesting tools for LLMs, the way that I use judgment is critical to the core functionality of the application. I agree with your concern and I will just say that the more time I spent concerned with chain of though where now I will make multiple versions of the same app using a different judge set a different “temperaments” and I found it to be incredibly enlightening as to the diversity of applications and approaches that it creates.

  Even using BMAD or superpowers, I can make five versions of an app without judges involved and I feel like I’m just making the same app five times because the API begins to coalesce around the business problem you want to solve. The vicissitudes of prediction tools always want to take the safest bet for the greater good, but with the judge involved we can make the agent force itself to actually be hostile about what exactly we’re trying to do, which has produced interesting and fun results.

Exactly; would probably be safer with a purely algorithmic decision making system.

Calling it now. Show HN: Pincer - A small highly optimized local model to detect prompt injection attempts against other models.

Securing agents in real time and testing them for drift in CI are pretty different use-cases…

This post is an AI-generated ad, isn’t it? It’s getting too hard to tell!

[dead]

Yes, full stop. They say they cap the body to 16k and give the LLM a warning, lol. And this is coming from a credit card company.

99% is usually the best you can do. So you can only layer multiple defences together, this makes sense as one layer to me.

I have an issue with security layers that are inherently nondeterministic. You can't really reason strongly about what this tool provides as part of a security model.

But also, it's in an area where real security seems extremely hard. I think at some point everyone will have a situation where they wanna give an agent some private information and access to the web. You just can't do that in a way that's deterministically safe. But if there are usecase where making it probabilistically safer is enough to tip the balance, well, fine.

It looks as if this tool has traditional static rules to allow/deny requests, as well as a secondary LLM-as-a-judge layer for, I imagine, the kinds of rules that would be messy or too convoluted to implement using standard rules.

I think this can be great as additional layer of security. Where you can have a non llm layer do some analysis with some static rules and then if something might seem phishy run it through the llm judge so that you don’t have to run every request through it, which would be very expensive.

Edit: actually looks like it has two policy engines embedded

> We’re supposed to be fixing LLM security by adding a non-LLM layer to it,

If people said "we build a ML-based classifier into our proxy to block dangerous requests" would it be better? Why does the fact the classifier is a LLM make it somehow worse?

Defense in depth. Layers don't inherently make something less secure. Often, they make it more secure.

> Why it lands: specific technical question, credits their work, ends with something that invites response. If Brex engineers are in the thread, one of them will likely reply.

BWHAHAHAHAHA. your bot tried, but failed at the same time. (also interesting that this user's other comments seem ok-ish. The prompts are evolving, we get a sneak peek here on what they prompted for, and the delivery seems more human as well)

I'm willing to wager that your comment was generated from the body of the article plus a prompt to work in an advertisement for your product, which gets a mention in nearly every comment you make (and every submission you make, sometimes on a daily basis).

At RSAC, there were a ton of agentic security startups converging on ebpf monitors for this reason. Eg, sondera gave a fun talk at graph the planet where they did that + exposed with a policy layer over agent traces via Cedar (used in AWS IAM etc). ABAC and identity were also appearing near here.

One thing I didn't see: are there any OSS solutions appearing here?