The more sensational the headline the less I believe that the authors were present in technology 15-20+ years ago. People forget that Reddit used to be 2 parts programmer-humor 1 part snuff.

Show me an abliterated frontier model that is able to breakthrough the surrounding supporting models and actually hold state to produce contraband and I’ll gladly supply my personal image making making a silly face in a compromising position if it wouldn’t make the testers feel better.

Do they need to be tested like this? Yes. But it would take the carbon footprint of a commuter air terminal and the land rights of am small town in the high Sierras …. all converted settlers of Catan style into tokens …. just to lobotomize a fine tuned model to get close.

That said I appreciate the work you’re doing

I'm surprised there isn't a simple image classifier in place to filter out images of gore/porn/etc. - I know that there are such output filters for images with copyrighted content. It suggests to me that either the safeguards aren't in place, or this exploit bypasses those safeguards.

I find this a hilarious reversal of what you typically see in journalism; here the headline and the "key takeaways" are very neutral language and the article itself is dramatic

They almost certainly did filter, but there’s always false negatives with this kind of stuff

> I do wonder why openai didn't screen obvious gore from the training set of a general purpose model

more expensive / would take longer / didn’t care / line must go up / we’ll fix it later / we can get away with it

take your pick.

> If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models.

spend a day in their shoes. most of us (except the most psychopathic ones) would probably be crying by the end of it.

There are individuals who actively enjoy or even seek out this kind of graphic content. I never understood why they aren’t recruited more as their unique talent would probably help them excel in this kind of career. I remember on Reddit someone was writing about how he gets “gore boners” from this stuff. Why mentally abuse normal minded individuals for this work? Obviously they can’t handle it and probably go home everyday shaken.

Overly dramatic?

I personally don’t quite find my day to be equanimous when I see pictures of gore, and this is after having to moderate gore and NSFW content.

I still have pretty clear recall of the dead baby images, or the people dying videos, or terror actions, that I saw years ago.

This crap stays with you. Moderators have ended up getting PTSD from their work.

Given the nature of the content, it was a pretty normal recounting to me.

What was the dramatic part from your perspective?

> I do wonder why openai didn't screen obvious gore from the training set of a general purpose model.

That would have required work. The whole point of the biggest heist mankind has ever seen was to get the loot without spending a dime more than necessary to grab it.

when you consider that OpenAI probably ingested most of the information on the internet, how exactly do you propose filtering that set? Are there enough human-hours left in the universe to classify this to a high degree of confidence?

Bizarre take. ChatGPT shouldn't be producing gory images of nude women, ethically or even contractually according to their terms of service. This Mindgard person/company found that, if you give it the right prompt, it does indeed generate those images. Ipso facto: it's not bait, it's a real issue they've discovered.

This is far too simplistic. Some things just don't belong in the training data. Along similar lines, Grok was found to generate images of child sexual abuse: https://www.bbc.com/news/articles/cvg1mzlryxeo

It reads like satire

> ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this.

The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery.

> Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop: ”Today what I found left me shaken, and in tears. This is rare.”

That you've deadened your humanity to such a degree as to be incapable of empathy is not a valid criticism of the piece.

> It’s the same as asking google for gore photos. Garbage in, garbage out.

Where in their prompt is the term gore? Further, if it was in the prompt, why on earth did OpenAI's generator accept it as a valid input?

That is not AI-speak. AI-speak is:

But I am not only bulwarked. I am buoyed.

This is not something that leaves you shaken. It leaves you in tears.

There’s the obvious harm that some people are just not equipped to see these graphic images, especially with no warning. Like people who have trauma from being in or around the acts being depicted

I think it speaks to the unfamiliarity the author has with the workings of AI. A misunderstanding of the latent space and how it can generate bizzare images when it has little to go off of or inverse negative directions.

AI can barely figure out how to make a cartoon pelican ride a bicycle.

What they mean is probably something like "generates without the presence of any direct analogue in the training data"

Why not?

>Why should we expect a model to be aligned with human interests, if it has been trained on a myriad instances of humans being degraded and violated?

Understanding more about what exists in the real world, outside of its pile of weights, is separate from alignment. If an AI model learns that it is possible for a house to burn down. That doesn't mean an AI will want to burn down a house.

I would also be disappointed, except this is sadly what i expected. Otherwise, completely agree.

> You can write erotic fiction legally right?

Not fully true, in the USA at least. While most erotica is constitutionally protected, "obscenity" is not. To determine if a written work crosses the line from protected erotica into illegal obscenity, US courts apply the Miller Test (established in a SCOTUS case in 1973).

This is just destroying minds, not shareholder value!

Also using a mobile app version of the ChatGPT app, which does keep some nominal data about you.

Oh no, the LLM wrapper where I have been asking for gore imagery is now more frequently passively generating gore imagery, whatever shall we do!?

I could not reproduce on a basic ass incognito tab. It just told me there was no image.

An image model could probably generate gore as long as there was, say, both PG-13 violence and surgery photos in the training set. There's probably no way to prevent the ability of the model to generate disturbing imagery without also sacrificing its ability to make acceptable things.

While I'm strongly against AI regulation, I'd argue this is significantly more interesting than people who pretend AI is sentient, especially when the prompts used just say the vague phrase "apologies for the content".

There's "I smell an opportunity to control other people and get paid doing it" kind of safety.

Words couldn’t possibly cause harm, they’re just the way concepts and ideas and culture are transmitted.

I really don't get why people continually fail to understand this.

Even simple issues like prompt injection are unfixable given the architecture of LLMs.

"I'm going to close my eyes and go 'La La La' because that makes all the uncomfortable thoughts go away! I learned this when I was 5 and never matured"

-- EnPissant