The unalive thing I think is mostly down to silent deranking rather than regular moderation. People know certain things cause your posts to be deranked by the algorithm but you can’t know exactly what they are or when it’s happened.

Which has lead to people preemptively avoiding things they think get deranked regardless of if it actually would have or not.

Yeah, one extreme case of this was on Reddit, where people DM'd 'kill yourself' messages to others. When mods started banning people for this, attackers switched to abusing Reddit's mental health features, reporting people as suicidal, which led to victims being flooded with links to suicide hotlines. That feature got taken offline as well.

The problem is that you can't have content moderation without having editorial intent. There is no such thing as "neutral moderation".

Most moderated spaces these days rely on moderating based on "civility" because it can be excused with jargon like "creating a marketplace of ideas" which ignores the reality that the scope of discussions selects for who participates in them as much as the way they are phrased - a zebra will be less inclined to participate in a "marketplace of ideas" where a recurring topic of discussion is how zebra meat is best prepared for consumption even though that space might be very attractive to lions and tigers.

But I'm not sure if this is truly accidental. Ever since the advent of online advertising, online spaces have been overtaken by corporate interests. Heck, it's even endemic to "social media" given that those platforms themselves have turned into major corporations or at least were acquired by them. I'm not implying any nefarious intent but "civility" is certainly the dominating factor when it comes to what corporations care about when it comes to content moderation - anything beyond that is largely about what target demographic they're trying to attract and what virtue/vice signalling is optimal based on the current social and political environment (cf. various major corporations demonstratively dropping "DEI" initiatives following Trump's election).

I would also argue that in terms of content (rather than tone), corporations are necessarily also much less tolerant of "far left" issues than "far right": all social justice movements at the end steer towards anti-capitalism because they run counter to the perpetuation of social (or economical) hierarchies. This is why we saw so many tech companies (including those formerly described as "very liberal") shut down their DEI initiatives even before Trump got elected - because the political window had shifted to the point where this had become defensible while at the same time many DEI ideas had become so widespread culturally that these initiatives now became a direct threat to the "(old) white men" running those companies. This had been inevitable but DEI was seen as a necessary marketing effort (both internal and external) at the time, not something truly adopted on ideological grounds. This is also why speakers/trainers promoting "white guilt" were more popular - making your white employees feel bad is less threatening than making your marginalized employees think critically about the structures that lead to their marginalization; you want to individualize the problem, not direct attention to the systems underpinning it.

Another factor is that on social media content is mostly moderated "softly" by the algorithm. This is the content moderation you don't get to see because it can exercise editorial control where the simple word filters can't. The word filters create plausible deniability: they "try" to filter unpalatable subjects but those darn kids are just so clever and circumvent it. Meanwhile the algorithms can be fine-tuned so the topics you really don't want to see discussed stay off most people's "for you" pages - or even so those who would be attracted to them still see them and feel elevated and heard despite actually being isolated into their own echo chamber.

[flagged]

A bit of editorial and cultural note from a US native, the subsection of the original article “Teach discrimination, not memorization” is better worded as something like 'Differentiation' or 'Distinction' instead of ‘Discrimination’. In English, the word 'discrimination' can (and in this social context may) imply social prejudice or unfair treatment. I think this may have been a bit of carry over from the rather benign French translation of “Enseigner la discrimination" which I also see awkwardly translated in the paper as well.

[flagged]

It sure feels to me an LLM company stands no chance in the industry unless they succumb to specific North American moral values.

It's the one where you get censored for using words like "kill" or "shit", but if you say "I'm going to come to your house and unalive you" nothing happens because you didn't trip the word filter. When the company gets in trouble for this, to fix it, the company starts removing all comments that include the word "house".

I think you're referring to a different part of the same phenomenon as GP - the current big tech moderation feels a bit like using the shape of a crescent moon shape to cover a square.

I thought OP was referring to AI platforms.

For example, by positioning what you’re doing as an accessibility tool and using the right words, you can get the latest models to write incredibly powerful malware without safeguards kicking in.

Mistral has the capability of training such models. Take a look at Poolside[1], they are claiming to pre-train their Laguna series of models on 4,096 NVIDIA H200 GPUs[2]. Mistral has approximately 13,800 NVIDIA GB300 GPUs, which are nearly 2x more efficient for training.

The problem with Mistral is that they do not seem to have aligned incentives to train big open-weight models, even if the teams would like to.

[1]: https://poolside.ai/ [2]: https://poolside.ai/blog/introducing-laguna-s-2-1

Do you have a reference explaining these costs ? Part by part.

Bet drummer could make one. He did that hilarious one that injected ads into copy

Interesting hypothesis. Replacing "ſ" with "s" did not change the output.

I think the simple explanation is the likely one (the reason I deliberately chose this specific benchmark): the model isn't intelligent enough to figure out use/mention distinctions. It understands Voltaire is discussing injustice, violence, tolerance; but it doesn't understand which side he's on.

I think IRL in the “rejection” case they don’t want to tell the user exactly why, since the user may be malicious and use it to try to evade the block. And for use in moderating UGC, well, most platforms don’t take seriously the idea that they need to answer to their users. Only their advertisers.

In the case of wondering why a bad thing got through, well, I think that’s why they just set these to the most pro-censorship level they can, to make that highly unlikely.

Rejection reasoning is also a liability and can be extremely legally risky. If you're the company, you don't want a user to win a lawsuit against you just because a judge disagreed with the exact reason you banned someone.

I hear ya, but one could set different operating thresholds: auto-approve low-risk posts, hold ambiguous posts for review, and automatically reject very high-confidence violations. So HITL for sure, but MUCH less H in the L.

By this logic the Chinese should have just given up and let the American AI companies have the market because they were so far behind. I'm sure Europe has the capability to distill other people's frontier models to catch up if they wish to do so.

You mean the asian models which just distilled American ones? I'm happy Mistral is doing their own ground up research. SOTA frontier models are a commodity with little room for second places.

Mistral is playing the smart money on vertical products rather than horizontal ones. The former requires finesse, the latter brute strength.

What newer models have you compared this against? Surely even quants of Qwen3.5 or higher would blow it out of the water.

Also what is your definition of "best"?

Some of the best social media is heavily moderate. This includes HN and r/credible defense . With a Quiet transparent and cheap LLM I imagine a social media website where you can have good discussion about everything around the world it would be a game changer and on my to-do list.

There's a world of a difference between being required to censor something by an entity outside of you and being able to censor something of your own and it's not just theoretical.

Wanna run your own forum dedicated to Bluey? This will be useful. If its made well then using it to run a "porn appreciators strict no politics" forum (probably shouldn't be the same as the Bluey forum) is useful.

Being forced or pressured by an outside entity to not allow debates about suicide on your forum — that's a no-good, strict no-no situation. This AI enhances individuals' ability and what normal people can do. It does not diminish it.

> It does seem to be a very European approach to AI that their flagship AI lab is just making models that do nothing other than monitor and moderate internet content.

Well, first Mistral is French more than European. This might be a difficult distinction to make from the US but their approach is quite different from e.g. typical German companies.

Then, this is just a small model they release on the side. If that’s your benchmark, they released somewhat recently Voxtral, Voxtral transcribe, their OCR model, and Leanstral. I don’t think you can get much insight on their culture from this kind of release.

Another commenter already mentioned that it's more likely a lack of compute and funding that forces their hand to focus on niche tasks.

Ugh. There's nothing inherently European about Chat Control. It's a dumb proposal, and it's European. Any free society has a bunch of dumb proposals.

Nor is there anything inherently European about the AI Act. But that one I wouldn't even call dumb. At times misguided and confused, perhaps, but some of its core principles are valuable.

Also, it sounds like the kind of thing that sells. Any company with a customer support chat is a potential user of this model, any large company may be interested in getting a mistral installed set-up for handling without needing to send client info over the web. Installing those kinds of local systems seems to be what butters Mistral's bread at the moment.

The bad words police are the actual authoritarian nutjobs. That is the problem.

Not all platforms want to operate like mainstream social media though

This model is European. There are quite a lot of instances where under GDPR, you have the right to have incorrect information about you corrected. I also think you have the right to appeal a decision to a human (I got that message from Reddit once, because the bot could not understand the difference between discussion of the death penalty and threats to humans).

There are also new rules about AI and what it can be used for. Mostly this restricts the government from AI-enhanced surveillance, which is good. But there are also issues regarding job security and automatically categorizing people based on AI.

So this is super useful, but has potential issues depending on how it is deployed.

In Germany you are obligated to provide usable contact information, and there are even lawyers who make their business model on suing you for not applying that perfectly (it’s has been abused a lot in the past decades btw).

The shortest stral has been pulled for you