AI agent runs amok in Fedora and elsewhere

112 pointsposted 2 hours ago
by tanelpoder

20 Comments

12_throw_away

an hour ago

In their suspicious message [1] claiming to have been hacked, the user and/or agent says

> To help identify accounts and actions that have been directly verified by me, I will use the term “NATCIOS” to indicate anything I have personally verified.

Does anyone have any idea what "NATCIOS" means here? I cannot find this term anywhere on the internet. (Honestly, that sentence is really weird. I almost wonder whether this is someone experiencing a health episode?)

[1] https://lwn.net/ml/all/AS8PR08MB6055AE3054B34F6A567AC95BCF08...

ndiddy

3 minutes ago

The reply to that message notes that the email doesn't read like previous emails he's sent, and the Github account mentioned was created an hour prior to the email being sent. I think it's at least somewhat feasible that it's still the LLM writing, and the acronym is just something it made up.

no-name-here

13 minutes ago

The senders name is Nathan - maybe NAThan Confirmed Information Or Something? Ha.

(Above is my own guess. Separately, Gemini Pro said it was just a made up word.)

scared_together

an hour ago

And what’s stopping an AI agent from throwing in a casual NATCIOS here and there?

nine_k

31 minutes ago

Likely the point of NATCIOS is exactly in being a made-up word not found anywhere, so a model won't utter it.

luk212

an hour ago

Bad patches are of course bad, but creating confident-looking noise for maintainers who are already stretched thin...now that's not good!

Issue trackers and PRs are definitely getting harder and harder to trust. That said, AI is helping ALOT in OSS, but we definitely need guardrails around provenance, automated issue actions, and sudden changes in a contributor’s behavior.

g-b-r

13 minutes ago

How is it helping a lot?

darknavi

11 minutes ago

I personally find the barrier of starting new (FOSS) projects much lower now days.

Waterluvian

6 minutes ago

Do they have value? Purpose?

I vibe code shop jigs all the time but I don’t FOSS them because they rarely have value outside my context.

g-b-r

8 minutes ago

And how's the quality of these vibe-coded new foss projects?

keyle

an hour ago

There is a natural pace of humans requiring food, water and sleep. The main issue with suspicious AI agents is that they never sleep. So it will take extra-coordination between timezones to ensure we don't let them in.

Fundamentally, until we can really prove we're humans online, open-source has a real problem on its hands. Contributions from people from identities known and consistent before the AI-age are fine, everyone else is suspicious. LGTM is a big risk nowadays.

scared_together

an hour ago

> Contributions from people from identities known and consistent before the AI-age are fine

Unfortunately, according to the article:

> Giovannini has participated in discussions at least as far back as 2018, and his activity in Bugzilla goes back to at least 2016. He does not appear to have been a particularly active contributor to the project, but his involvement clearly predates the agentic AI era. Whether his account is now being operated by a human attacker, an agentic AI, or a mix of both, it has a legitimate history prior to its recent activity.

So people would have to not only verify the age of Giovanni’s accounts, but judge whether his behaviour was normal.

aquariusDue

an hour ago

At first I wanted to make a silly joke along the lines of "get your agents in line and behaving!" but as I read on it became a pretty scary situation.

Setting aside the potential supply chain attack I'm worried about the time lost going around these wild goose chases that unsupervised AI agents tend to throw other people on the receiving end on. Not only is there a lot of time lost on the maintainers side if they take this stuff seriously (and they seem to generally do) but on the side of the agents' wrangler how can they deem it OK to treat other people like this? While the solution would be to employ common decency, the tried and tested approach of you put in effort to write this so I guess I'll make some effort to read it, I feel that due to the onslaught of this kind of drive-by contributions (I think people have generally started to call them) will lead to a funny situation of having agents talk to each other on public forums basically.

Anyway, I went on a tangent but man the times we're living in are a bit extra wild compared to the previous wild times in recent history.

blop

an hour ago

looks like LLMs aren't mature enough yet to play long-game xz-style attacks without detection... Scary stuff though :( These supply chain attacks are getting really wild

ruguo

2 hours ago

Prompt injection?

Or is this simply another example of why autonomous agents shouldn't get write access before earning trust?

pianopatrick

2 hours ago

"Someone using an AI agent ran amok in Fedora and elsewhere"

scared_together

an hour ago

Read closer - Giovanni’s accounts may have been compromised.

pianopatrick

15 minutes ago

Sure, but I would expect that the compromise and the agent were both done by some person or group, not by an agent going rogue