The US orders every civilian agency to rip out DeepSeek, even tools built on it

61 pointsposted 7 hours ago
by jacquesm
(madrobot.blog)

38 Comments

kidfiji

6 hours ago

Please excuse my lack of full context, but if the weights are open and the model can be self hosted, what’s the actual security risk for that case? Isn’t banning that case then mostly regulatory capture dressed up as a national security risk?

liamcardenas

5 hours ago

For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.

fxtentacle

4 hours ago

... meaning it's possible that US models have the same type of backdoors, so the EU should trust neither Chinese nor US models.

JumpCrisscross

4 hours ago

> so the EU should trust neither Chinese nor US models

Yes, in theory. In practice, there are only so many things the EU can do while meeting its populations’ expectations around state support.

chatmasta

2 hours ago

> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload

From 5 days ago: https://projectdiscovery.io/research/how-abliterated-models-...

julianlam

an hour ago

That can occur in a cloud-based, closed-weight model as well. So this argument makes no sense.

In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you.

echoangle

an hour ago

That’s an argument against Anthropic/OpenAI, not for Deepseek. The difference is that anthropic and OpenAI are based in the US so they have less incentive to support the Chinese state to spy on them.

rayiner

6 hours ago

Weights being open isn't like code being open. With code, you can inspect the code to see what it will do. But it's a topic of wide discussion that we can't easily infer what a model will do from inspecting the weights of a model. The process has emergent behaviors we can't predict.

Terr_

5 hours ago

Perhaps something like

    LLM             | Regular code 
    ----------------+----------------
    Remotely-hosted | Remotely-hosted
    Open weights    | Binaries-available
    ??              | Source-code available

halJordan

17 minutes ago

The ?? Is training recipes & datasets

bitbasher

3 hours ago

Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.

jurgenburgen

3 hours ago

Tool calls are done by the harness. This just means you should never let a model operate “hands off”.

kevinrineer

2 hours ago

Know any harnesses that explicitly don't have a "yolo" mode?

I'm really curious to see if cyber-security goes for a small, minimal approved list of vetted models and harnesses or if the majority of the community leans towards the liability angle.

blooalien

an hour ago

> This just means you should never let a model operate “hands off”.

Good luck with that one. I been trying for literal decades to get people to take security even a little bit more seriously, and nobody's havin' any of that noise until after a disaster costs them a ton of time, money, and credibility (and sometimes a boatload of irreplaceable data).

netrap

6 hours ago

could you poison the model? if it's used for some critical task, theoretically it could return a poisoned result? not saying it is, just curious

akomtu

5 hours ago

It's a binary blob created by a highly capable adversary. It surely has dormant capabilities activated with some keywords. It's the Trojan Horse. You wouldn't use a C++ compiler binary provided by CCP, right?

unrented7977

3 hours ago

> You wouldn't use a C++ compiler binary provided by CCP, right?

Racism aside, one can prove a compiler is correct. It can be shown beyond a shadow of a doubt that a compiler produces correct output for a given input.

Would you use such a piece of software? Or is the boogeyman too scary?

blooalien

an hour ago

> "Racism aside" ...

I'm not sure how an unwillingness to trust the Chinese Communist Party is in any way "racist" exactly. I know lots of Chinese people I'm fairly sure I could trust, but the Chinese Communist Party ain't among 'em. Do you trust them?

grim_io

2 hours ago

Authoritarian communism is not a race.

Just because a compiler binary compiles program A without a backdoor, it does not prove it won't add a backdoor to program B.

skeledrew

30 minutes ago

One could always, you know, reverse engineer it to find if there's anything off.

TedDoesntTalk

5 hours ago

This. If the LLM has access to the outside world through its harness, then it can do real damage - data exfiltration, etc.

skeledrew

27 minutes ago

I'll believe it when I see it. The chances of discovery are far too high to make a malicious LLM.

antonvs

4 hours ago

> You wouldn't use a C++ compiler binary provided by CCP, right?

Why not? Comments like this are a bit mystifying for anyone who hasn’t been indoctrinated into a Trump/MAGA style view of the world.

adammarples

2 hours ago

Well, you know they do a lot of cyber attacks and espionage right?

bradishungry

2 hours ago

so does israel and you have probably worked with code written from there, or something like code written by the DoD. people wouldn’t say this about Ada

blooalien

an hour ago

> ... "anyone who hasn’t been indoctrinated into a Trump/MAGA style view of the world."

Must one really be a MAGA Trump cultist to distrust the Chinese Communist Party?

Ancalagon

6 hours ago

yes, thats the point

mainecoder

6 hours ago

Does not matter if it is opensource if a piece of rock is designated a National Security Risk because the Rock came from CHina that is it, that rock is a national security risk. The designation is assigned then action must follow; you cannot question a national security risk that is why it is designated a national security risk if it caused health problems you would be told by the Centers for Disease Control or US Department of Health and Human Services. A National Security Risk can be anything that they assign, it can even be a thought otherwise known as a thought-crime. Stop asking why?

throw0101c

5 hours ago

> The order's summary of risks says DeepSeek's servers are in China and that Chinese law "permits extensive authority to government cyber and intelligence functions to access private company infrastructure".

So the Chinese version of the American CLOUD Act?

Cameri

2 hours ago

It comes down to which backdoored models you trust.

user

6 hours ago

[deleted]

grafmax

2 hours ago

The regulatory capture begins.

halJordan

14 minutes ago

It's not regulatory capture to remove software/harware from adversaries.

No one was saying "Oh McAfee is really gonna control the us's antivirus market regulations" after they banned kaspersky.

metalman

2 hours ago

the order(paraphrased) is 'we are imediately ordering you to, or not to, rip out deep seek, depending on how you feel about that ,and what might go wrong, and write to us about it, sometime in the next couple of months'