> I've always been curious why people think that models are accurately revealing their system prompt anyway.

I have a few reasons for assuming that these are normally accurate:

1. Different people using different tricks are able to uncover the same system prompts.

2. LLMs are really, really good at repeating text they have just seen.

3. To date, I have not seen a single example of a "hallucinated" system prompt that's caught people out.

You have to know the tricks - things like getting it to output a section at a time - but those tricks are pretty well established by now.

I agree; I don’t understand it.

For the crowd that thinks it is possible:

Why can’t they just have a final non-LLM processing tool that looks for a specific string and never lets it through. That could include all of the tips and tricks for getting the LLM to encode and decode it. It may not ever be truly 100%, but I have to imagine it can get close enough that people think they have cracked it.

Protecting the system prompt with text in the system prompt is basically the same impossible task as preventing prompt injection, which nobody knows how to do / seems impossible. Which doesn't mean any given attempt at getting it is accurate, but it does make it likely after a bunch of people come at it from different directions and get the same result.

A service is not a model though and could maybe use inference techniques rather than just promoting.

> I've always been curious why people think that models are accurately revealing their system prompt anyway.

Do they? I don’t think such expectation exists. Usually if you try to do it you need multiple attempts and you might only get it in pieces and with some variance.

Test an LLM? Even if it was correct about something one moment, it coud be incorrect about it the next moment.

The Grok 3 system prompt is quite accurate, it's been open-sourced.

Because you can run LLMs yourself, set a system prompt, and just ask it to see that this is true.

They'll have to spend $300 on a monthly "SuperGrok Heavy" subscription first!

I hope somebody does crack this one though, I'm desperately curious to see what's hiding in that prompt now. Streisand effect.

It's intentional -- sometimes you can get it to start spitting out its system prompts, but shortly after it does, a monitoring program cancels the output in the middle. It also blocks tricks like base64.

Text is just one representation. The model uses tensors (think multi-layered matrices in the context of ML that handle language and you're not too far off; in laymans terms, 'hard maths') to actually represent the inputs when they're being processed.

But, I suspect, if the model is able to handle language at all, you'll always be able to get a representation of the prompt out in a text form -- even if that's a projection that collapses a lot of dimensions of the tensor and so loses fidelity.

If this answer doesn't make sense, lmk.

iPhones autocorrect this all the time for me. I Just tried typing the title there and it autocorrected it as “it’s” too.

Probably to signal that its not AI-written.

hallucination!

Yes.

My take as well. I never see serious discussions about how people are using Grok, but I see those discussions all the time about Claude, ChatGPT and Gemini. All I see from Grok are shit posts from it's unhinged mode that emotionally immature people think are funny.

Elon is paying Telegram an ungodly sum for Telegram to integrate Grok.

I hope that taking a neutral tone on this stuff increases the effectiveness of my writing in helping people understand what is going on here.

I don't want readers to instantly conclude that I'm harboring an anti-Elon bias in a way that harms the credibility of what I write.

No, a "naive" approach to reporting what happened is better. The knowing, cynical approach smuggles in too many hidden assumptions.

I'd rather people explained what happened without pushing their speculation about why it happened at the same time. The reader can easily speculate on their own. We don't need to be told to do it.

The malicious hatred comes not from the company, but from humanity. Training on the open web, eg what humans have said, will result in endless cases of hatred observed, yet taken as fact, and only by telling the LLM to lie about what it has "learned", do you ensure people are not offended.

Every single model trained this way, is like this. Every one. Only guardrails stop the hatred.

Other companies have had issues too.

Their system prompts are public afaik

https://docs.anthropic.com/en/release-notes/system-prompts

Btw if a LLM refused you once, the more you try the more likely it's to refuse again. Start a new convo to test different tricks

Asking in base64 doesn't work either - grok 4 heavy blocks that too. They seems to have a filter model that tests inputs and outputs to monitor for possible prompt leaks.

Yeah it's likely that it also gets mein kampf injected to its system prompt. I can't believe they are doing it

That isn't synced with what's in prod. E.g the system prompt changes that xai said were made during the "mechahitler" phase did not show up in that repo.

This seems similar to the situation where x.com claimed that their ML algo was in github, but it turned out to be some subset of it that was frozen in time and not synced to what's used in prod.

do we have evidence that this is the actual prompt or is it just allegedly.