Linode Security Incident

197 pointsposted 3 days ago
by lacoolj

114 Comments

6thbit

3 days ago

It feels like we hit an inflection point in the last week or so. Kind of a large tidal wave to crash on the coast, but likely not the tsunami yet. I hope.

If you are running services and don't yet have an easy way to react to these things with swift process defined, I hope you don't like to sleep.

georgyo

2 days ago

This isn't Linode getting attacked, it is telling customers that this zero click RCE CVE is pretty serious. You should patch ASAP.

Did you know that GO defaults to listening on TCP connections with MPTCP, so anyone running GO programs listening on the internet (Caddy, traffic, etc) and are on an unpatched kernel now have a way to from attackers to get root on that box?

Even if you're not using MPTCP, this functions as a DOS attack.

It's a very bad CVE, so yes. Follow the advisor and patch.

ianmarcinkowski

2 days ago

What if the "inflection point" is just all of the low-hanging fruit that can be detected by agentic vulnerability scanning? If the low-hanging CVEs get patched we might not see a tsunami.

6thbit

2 days ago

I really hope you’re right!

And even patching the ‘small’ cves will take us for a spin as it already has. Then there can be specific low hanging fruits that chained together make a splash.

FuckButtons

3 days ago

When mythos was released - I guesstimated that we had 6 months until hackers had similar enough capabilities that we would start to see real problems in the wild. Unless there is some change in the trend line things are about to get really quite spectacular.

rpcope1

3 days ago

I'm just waiting for someone to drop a real zero day on openssh or openvpn for before the real pants pooping starts.

rurban

2 days ago

openssl much more likely. And this will have a bigger impact

gtech1

3 days ago

If you run openssh open to the public...

keyle

3 days ago

Tidal waves are one thing, this is a tsunami.

lifeisloving

3 days ago

Its really not that hard to secure your servers, especially if your not a high value target. If you are a high value target, you should have security professionals on staff.

I dont think my security posture is all that much different than it was last year. Use a Secret Manager and or encryption, have backups, strong firewall rules, use canaries of some kind.

If anything llms are making security worse because there's more people offloading their thinking on llms, or people that skipped some steps in their learning journey because they just ask the llm to do everything. I dont think its necessarily because attackers have llms now, but that contributes.

Edit:

Damn Im getting mega downvoted for saying this. So are people of just at a consensus that there's nothing you can do? That we're in security armagedon???

Have you ever thought to consider that security took hit along with the general degradation of software because of AI??

moscoe

3 days ago

You are literally the everything is fine meme. I admire your confidence. I never realized it was so simple!

UqWBcuFx6NV4r

3 days ago

Is this like when people say “it’s not that hard to write memory-safe C”, and then nobody does? I take this as actually meaning: “I feel that I can keep my shit locked down.”. Whether or not that’s true, which i’m skeptical of, plenty of OTHER people can’t, or won’t, or whatever. That’s the reality.

To speak to your edit: this is false equivalency. You can still decrease the odds of being hit. There are still better and worse ways to address security.

mort96

3 days ago

Can you tell me what I should be doing differently then? I just host some services, among them Nextcloud that's written in PHP, protected by SSL, on a server running Linux. And I feel like the past week has been a lots

tbrownaw

3 days ago

It's tedious, which is a particular kind of being hard.

sanex

3 days ago

I agree but disagree about the cause. People have always been bad at security and code in general.

amelius

3 days ago

Have you not heard of supply chain attacks?

bvcp

3 days ago

if anything we should be using ai to find this stuff before the bad actors, too bad the model providers are hell bent on regulatory capture via fear

OhMeadhbh

2 days ago

I know I'll get downvoted for this, but here goes. I was in a position to see some of the mythos derived results early. In my organization everyone was running around with their hair on fire. I asked whether or not any of the vulnerabilities mythos found were also found with existing static analysis tools. The result was blink blink "what are static analysis tools?"

I'm not saying mythos didn't find anything new, but several people who should have known better were not using SAST/DAST tools to find potential vulnerabilities before pushing their code into the great unknown. One person I talked to said they didn't bother because GitHub automatically runs SAST tools against pull requests. I looked. Their CICD pipeline had no such tool configured. I'm pretty sure it's more appropriate to say GitHub CAN run SAST/DAST tools, if you configure it, but it won't run SAST/DAST tools by default.

This was a very small sample, but I wonder how common this type of confusion is.

bvcp

2 days ago

this is the problem with prohibition we all know ai tools can be useless and amazing depending how you use them. bad actors aren’t constrained to the one way anthropic uses them or the way they will let us use them, so the good guys are at the mercy of their capabilities. i would feel a-lot safer with more access in more hands than the current climate. to your point about existing security tools, we all know tools make ai even more capable so we shouldnt be framing it as a replacement in either direction but again ai orchestration using these tools and improving them. look at anyps5 do we expect ai could do this from a cold prompt? or is it the pattern they deployed plus the ai capabilities that made it possible. cyber security with ai will follow the same patterns. bad guys have to win once and good guys have to win every time making the tool access asymmetric just makes that even harder. doing that under the guide of safety is mind boggling.

spydum

3 days ago

Wow, that isn't clear. Is there an incident, or did they just become aware about MPTCP vuln? Details on the mptcp are also pretty scarce as well right?

6thbit

3 days ago

Yeah its unclear. Sounds like they are just now patching it and treating it as an incident which is probably the right thing to do.

onei

3 days ago

(Former akamai employee) Akamai have an incident process that handles both "things are bad for multiple customers" as well as "we need eyes on this from multiple departments/stakeholders asap". If you're hitting a wall debugging an issue on your own, especially if it's not overtly customer impacting, declaring an incident is an effective way of getting the attention needed to solve it.

Once you declare the incident, the process includes these status page updates, alongside equivalent internal status updates.

In this case, they probably wanted to maximise coordination, so used the process designed for that.

CoastalCoder

3 days ago

Also a former Akamai employee.

I left because I didn't like the pressure / pager duty, but I seriously respect how smooth and well thought out their operations were.

dofm

3 days ago

I am wondering if there is something significant written with Go in their system stack (Go uses MPTCP by default on listening sockets, it seems)

dboreham

3 days ago

Based on email I got from another big hosting provider, they're all patching for the same checksum metadata vulnerability (the CVE they mentioned is the same). There is however a root RCE PoC according to the researcher so...potentially bad.

lacoolj

3 days ago

Yeah agreed. I actually only noticed because my two boxes are super slow and I've been troubleshooting for about an hour. Finally opened up the Customer Support page and this was the top incident.

tbrownaw

3 days ago

I haven't seen any emails about it, so I'd lean towards it being something that doesn't rate sending emails.

paaloeye

3 days ago

Even better CVE 10.0 has just dropped https://news.ycombinator.com/item?id=50310937

INTPenis

3 days ago

Who uses Canonical LXD?

I never even heard of it.

paaloeye

3 days ago

MicroCloud for example, it's been actively pitched for smaller sites by Canonical

gsjhdheje

3 days ago

Incus is the successor... Used it for years ... Supports docker images, container images (more like a container running a VM image) and vms

paaloeye

3 days ago

MicroCloud and some other product for sure

dist-epoch

3 days ago

I've lived to see the day when half of the Hacker News front page is about hacks. What a time to be alive.

nout

3 days ago

All companies should at this point invest a lot of money into defensive AI. In ~July the AI hit crypto and bitcoin, where it's the easiest to extract the most money. Now the AI powered attacks are hitting infra and SaaS. Next (~December), it will hit classic infrastructure, banks, small businesses, etc. By mid next year, we will be over this hump with defensive AI being required part of everything.

throwuxiytayq

3 days ago

I find it breathlessly funny how people think it's going to be AIs vs AIs fighting it out with light sabers and plasma lasers in the cyber space or some shit.

Your intuition that this can be balanced out is wrong. When you have a CVE in your TCP stack, your vault unlocks and you're fucked, it's done. My dude, there is never going to be AIs scanning every incoming packet for signs of a 0day, and deciding whether to, uhh, cut it up with a lightsaber and shoot it with a laser pistol. Your options are to have a stack so secure that it can't be targeted, or to disconnect all network cables.

ForHackernews

3 days ago

I assume that "defensive AI" means finding and patching vulnerabilities before attackers do, not some kind of live LLM packet-inspection.

It is possible to write more secure software[0], companies just choose not to prioritize security.

[0] https://cr.yp.to/qmail/qmailsec-20071101.pdf

nout

3 days ago

I specifically meant red teaming with AI across your infrastructure to find vulnerabilities before these are exploited. This is what I have been actively doing and it works.

dist-epoch

3 days ago

AI's can decide on the action to open the vault file. It can also prevent TCP connections from unexpected IPs.

jeremyjh

3 days ago

Its already happening to classic infrastructure, small and medium size businesses. Absolutely anyone can summon a gremlin capable of exploiting stuff like this or the PHP thing last week. There are clankers probing this stuff all over the place within a day of it going public.

elashri

3 days ago

@dang I would encourage you to really think about having security incident as a new HN tag. The wording can be better though.

downrightmike

3 days ago

We can call it [SI]

entrope

3 days ago

As long as we know that doesn't mean Systeme Internationale, Sports Illustrated or Super Intelligence.

gk1

3 days ago

> Super Intelligence

I think that was the joke.

Arbortheus

3 days ago

This reads to me that nothing has happened and they are just trying to patch stuff? Right?

ericd

3 days ago

Anyone have any thoughts on what happens when banks/brokerages start getting hit in earnest? Think it's prudent to split single brokerage portfolios across multiple? Or do they have enough speed bumps in their procedures that they're not quite so vulnerable to major losses?

pianopatrick

3 days ago

If the large and politically connected banks and brokerages have losses large enough to threaten their survival as institutions then the government will jump in to help secure them and bail them out if needed.

I would stick with a "too big to fail" institution instead of splitting across multiple.

If you want a real "backup" you could ask for physical stock certificates or get physical gold or something else that cannot be taken from you by computer hacking.

pixl97

3 days ago

How is the government going to help secure them? Do they even have enough qualified people?

pianopatrick

3 days ago

Well I assume there are some smart hacker / security people in the NSA etc. who could be parachuted in to help if it really came to that.

I think these banks will probably be able to hire equivalent people in the private sector though and might not need the government help.

pixl97

2 days ago

I work in infosec and work with many of these large organizations.

Simply put I don't think dropping in smart people in dysfunctional organization is going to have instant results. It will take a pretty long period of time to make them aware of all the systems with holes.

In the meantime hackers would be running off with actual information and money. More than likely I see banks turning off all kinds of systems in a panic and people with their money there being in an even bigger panic.

spacephysics

3 days ago

FDIC insures deposits up to 250k. Big institutions basically all have this government backing. If an incident occurs, FDIC has deposits available next business day in event of a bank failure

baobabKoodaa

3 days ago

FDIC insurance is mostly unrelated to this. If someone hacks a brokerage, sells a bunch of customer assets for cash, adds new bank accounts for those customers, and then withdraws that cash to those new bank accounts, FDIC insurance does not help at all.

ericd

3 days ago

Another bad one I've heard, buying thinly traded penny stocks as a way to exfiltrate money from brokerage accounts without wiring.

user

3 days ago

[deleted]

nodesocket

3 days ago

Anybody recommend a CVE alert self-hosted app? E.G. I specify my software, OS’s, and versions and when a matching CVE is created I get an alert. Don’t want enterprise security products.

ForHackernews

3 days ago

What's the mitigation for this? There's no patches available yet https://ubuntu.com/security/CVE-2026-98432

markasoftware

3 days ago

Mitigation is to retire and wait for the singularity.

Don't use MPTCP is the first mitigation, but my understanding is this might be exploitable without MPTCP.

bombcar

3 days ago

Filtering packets with ip-options set may also help, but it seems Akami is already doing that at the edge.

netrem

3 days ago

Debian does have a patch up

optimalsolver

3 days ago

I mean, this situation is untenable. HN's front page is just going to be an endless list of breaches.

I don't see how web services proceed like business is normal.

peri-cl

3 days ago

Introducing: HNN (Hacker non-news): a list of web domains *not* currently under apocalyptic AI attack. Shorter and more parsimonious.

- Tell HNN: pets.com was *not* owned today

- Tell HNN: about:blank was safe, too

- Show HNN: my new gopher protocol project—no currently known CVE 10's

mceachen

3 days ago

gopher goes to 11

(archie and veronica are non-denumerable)

LtWorf

3 days ago

Better than the usual endless stream of what your agent did today.

user

3 days ago

[deleted]

rvz

3 days ago

So it begins. The war against the abliterated AI models. Even at 1,000 TPS any service is a target.

Imagine swarms of abliterated frontier AI models that reach >20,000 TPS. As I said before [0] you are going to have to hire specialized security engineers or technical bodyguards from the senior to senior staff level using AI agents to protect your entire business.

Otherwise you are cooked.

[0] https://news.ycombinator.com/item?id=50310312

jacquesm

3 days ago

Guess who has access to abliterated frontier AI models?

lacoolj

3 days ago

Primeagen went over this in the last episode or two - apparently this sort of defense is costing ... one of the big models or companies hosting them ... $900k/day or something to constantly run the best model and monitor 24/7 to combat the attacks [0]

[0] https://www.youtube.com/watch?v=9OahTki4cPA

vsgherzi

3 days ago

I don’t really have experience running anything like this so this is very much a comment front the peanut gallery but

Is that really the best way to do things? I’m not really sure what an agent offers here does it just look at logs and do actions? I would not really feel safe if that’s my service’s answer to security.

I could see it maybe as a really smart filter to block ips that are sending probing requests but I don’t really see that as a good reason to have an agent be sitting 24/7 looking at logs.

GoblinSlayer

3 days ago

I think, it's about reviewing code for bugs.

shoobiedoo

3 days ago

What's the future of cybersecurity going to look like? A rapid shift to shoving everything into microvm's to minimize attack surface?

chr-s

3 days ago

"You got it. Corporate core data for Tessier-Ashpool S.A., and that ice is generated by their two friendly AI's. On par with anything in the military sector, looks to me. That's king hell ice, Case, black as the grave and slick as glass. Fry your brain soon as look at you. We get any closer now, it'll have tracers up our ass and out both ears, be tellin' the boys in the T-A boardroom the size of your shoes and how long your dick is."

ifwinterco

3 days ago

Best science fiction book ever written IMO

mondrian

3 days ago

I recommend Gibson’s (abridged) reading of the book - but only after reading the book. Whenever I read a quote from the book now I hear it in Gibson’s iconic voice.

lelandfe

3 days ago

Airgapping.

I'm hoping goverments take stock of what is online that needn't be.

ethagnawl

3 days ago

Unikernels (e.g. MirageOS) were gaining traction right before containers swooped in and sucked the air out of the room. I think they could see renewed interest in the very near future.

Arbortheus

3 days ago

OpenClaw agents continuously scanning your prod infrastructure and applying config tweaks and package updates autonomously on the fly in response to viral CVE numbers trending on X.

alasdair_

3 days ago

The agent sounds like the perfect attack target - it has rights to upload arbitrary code everywhere without setting off alarms.

Arbortheus

2 days ago

I was being facetious, it’s a terrible idea on multiple levels.

amelius

3 days ago

Formally verified code.

samwilliams

3 days ago

Infrastructure is an FPS now.

pawelduda

3 days ago

One day your job will be to play StarCraft but under the hood it will be your agents vs attacker's agents

jdprgm

3 days ago

Enders Game but AI instead of aliens.

ProllyInfamous

3 days ago

Today? Today is that day... probably a yesterday or two, as well.

bombcar

3 days ago

I’m going to be running bongcloud into ghosts only. Uthermal style.

tripleee

3 days ago

If this is the job description sign me up

user

3 days ago

[deleted]

14u2c

3 days ago

More likely we'll just see another shakeup of the layers of abstraction that the average company operates at. Why would you hire specialized people when AWS can do so for you?

GoblinSlayer

3 days ago

As I understand, it's LPE. Cloud hosting was always tough.

xlayn

3 days ago

You are being downvoted but I agree, any company including openAI, anthropic, google etc... should be closely regulated by goverment to ensure they never run any AI that doesn't follow all the rules defined by governing bodies, nor train them to have any workaround... NO ONE HAS ABLITERATED ACCESS TO ANYTHING.

Thanks god, you are not a wacko thinking qwen3.0-1.5B-abliterated would hack a goverment.

pugworthy

3 days ago

Lately it seems like we need "hacks.ycombinator.com" as a new sub-list.

sublinear

3 days ago

We need an "ai.ycombinator.com" too.

keeganpoppen

3 days ago

ironic that "hacks" is a needed subdomain of "hacker news"

optimalsolver

3 days ago

Yeah it is, but everyone forgets this board was launched in the 90s for discussion of the 1995 movie "Hackers", not for discussion of hacking in general.

brian-armstrong

3 days ago

I'm still holding out for the sequel

optimalsolver

3 days ago

It would now be 120 minutes of some guy playing EVE or browsing porn while his agent does everything.

user

3 days ago

[deleted]

6thbit

3 days ago

/incidents would do

georgemcbay

3 days ago

daily sticky post...

Ask HN: Who didn't get hacked today.