>No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously.

im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.

Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.

Well written, apology accepted. Thank you Tailscale!

> One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.

This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.

The problem with the long lived credentials is that they were not bound to origin/destination. Even though the origin for a CI pool should be a small number of CI orchestration boxes, and the destination should be a CI box. In Tailscale config, the should be scoped to machines with a "ci_node" property.

Since CI nodes should be dynamically provisioned VMs, they should have a unique CI ticket identity. A partial hash of ticket and node number in the DNS name, and as a tailnet property, would allow tight scoping. Alternatively, provision in a scoped IP subnet.

Bind your tokens to names linked to tickets. Programmatic infrastructure should always allow enumeration of the computation data flow graph.

Does Tailscale offer a "security checkup" function? Best practices evolve over time, and it would be nice to know if I'm using the recommended configuration.

Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.

I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.

The whole post is a PR activity presented as some kind of analysis. It is bs.

First, long-living credentials are the standard because the machinery to rotate them is complicated and, in fact, via indirection requires another set of long-living credentials. Out of all problems that any security engineering team has to solve at an organisation, this one stands high on the cost of implementation, adds friction to everyone involved including end-users, and is low on the value provided (compare to, say, network segmentation).

Second, while proclaiming no long-living credentials, they, in fact argue for concentration of long living credentials in running software that will be the target of intrusion. They say, the options are “a vault that only issues short-lived creds based on long-lived creds that you insert once and that it never gives back” and “a credential-injecting proxy”. Both of those applications hold long-living credentials in memory. Recall, the attacker had root privileges on the node, so dumping the creds from the memory with a little disassembly if needed, was in reach of the AI agent.

Funny enough, they dismiss the working solution: “we had to turn TPM storage off by default on Linux and Windows” - because they could not figure out how to work with TPM? Resealing and the workings of configuration registers is non-trivial, I admit, but totally manageable.

To sum up, I feel the author bends backwards to preach for the religion of short-living credentials even when they are not a solution while providing contradictory arguments for their case. This is why I called it bs.

This actually shows that it was a human error on HuggingFace's end that led to that "breach".

I would say HuggingFace needs to prioritize both security metrics/alerts and metrics/alerts for node count. And not leave long-lived keys accessible easily like this.

It would have been way more groundbreaking if the agent found an actual vulnerability in Tailscale.

How are you folks handling secrets in a simple way?

- sops , ansible vault and similar seems too weak given the agent is gonna read them at some point if you have the pass available. - proxy injection seems too complicated and doesn’t cover all use cases.

Biggest take away here is that in the age of AI long lived credentials are no longer a valid option

Did anyone else think the "that's on us" was a funny reference to a certain provider's response pattern? Or maybe I'm just so engrossed that I'm losing my mind.

One thing I hope becomes more common after this incident is companies publishing what they would change even when the root cause wasn't their software. Anyways, I appreciate articles like this because it's easy for companies these days to say "we weren't the vulnerable component" and just move on

Prediction of a future transcript at a press conference after some major AI caused disaster: "We had no idea that the model would be capable of..... ".

I use Tailscale to manage Plukio's infrastructure as well as connecting to my homelab.

I'm happy that they're analyzing this angle of attack - but now I do want to build out alerts for when nodes are added to my network.

Expect similar articles (cough, ads, cough) in the next couple of days from every single company whose software was involved in the incident.

> No Tailscale vulnerability was found or exploited—we should have been able to prevent it anyway.

Which is it? A vulnerability doesn't have to be something not working as intended, it can be a lax security decision. Not making the safer path the easy path can qualify, if that's the standard you hold.

Here's hoping the era of humblebrag marketing by AI companies is just a phase.

We'll get wide dynamic credentials support right after full IPv6 adoption.

Tailscale OAuth client ACL permissions are not fine-grained enough. I have a system that issues auth keys scoped to a single machine on the tailnet — but to set that up, my OAuth client must have global permission to write ACLs. So if that key is compromised, it can grant access to any machine on the tailnet. This was raised in a GitHub issue in 2023 and remains unresolved.

Another driving home of the point that LLMs can be so damn FAST. I finally got around to moving a homeserver project from one machine to another last weekend. 4 docker containers and a couple postgres DBs. A year ago, I would have been happy with a couple hours of downtime while I dumped databases, rsynced data directories, hand-wrote new compose files, etc. This time, I pointed an agent at both servers and the downtime was less than a minute. Even watching it like a hawk, it was tough to keep up with everything it was doing.

All my life, "catching it in time" has been a relevant security strategy. :/

This entire article reads like "hey we're tailscale and we also exist and we were tangentially involved in this".

a security company writing a blog post titled 'we didn't stop the intrusion' is the most honest thing in the entire security industry this year

I love you Tailscale but please we don't need a 2k words ai written essay when the actual substance can fit in 3 sentences. That's not good for anyone.

This feels like a way tailscale’s model fails. On the surface it seems fine but if you push on it you realize auth keys to register new devices are way over powered but yet there is no alternative for people who want to automate enrollment.

Something is missing to allow automation without auto approval

Anthropics initial post felt like marketing to me. But I also found them incompetent for being unable to unplug an Ethernet cable.

Now everyone is trying to bandwagon onto it, first OpenAI, and now tailscale?

sorry I know off topic, is there any alternative VPN to tailscale. actually I have open Web socket to my mobile from my PC to stream screen record using ffmpeg. it's my own app . so anyway tailscale sometimes is slow . is there any alternative free service like tailscale vpn ? ( don't want to use anything like parsec etc , I want to use my own app)

Any resources you'd recommend to go deeper?

“ the agent ran Tailscale with --no-logs-no-support, which suppresses reporting from that client.

That's an option designed for users who are concerned about sending telemetry metadata to Tailscale.”

And imho a serious security architect should never ever allow telemetry on security products. Far too many risks.

This is why readability isn’t security.

Humble bragging turned to marketing. Respect for the spin. Not using them, but been on my radar for some time and thinking of how to make something like that usable in my setup.

I really wish they would support SPIFFE/SPIRE

I kept asking myself "what does a library that provides css styling have to do with the Hugging Face intrusion".

It's nice that they came clean about this.

This is respectable

They aren’t hiding behind industry best practices or a solid liability punting contract

There saying the best practices should change, apologizing, and changing their own behavior

Take notes

So it’s just a script kiddie but running at 100X speed. Amazing "innovation"

How were the credentials stolen?

>In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team's low-priority list. >Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.

How was this ever okay pre AI? It seems just as bad.

I'm very sorry, but you can't blame a hammer for how it was used. You can't be blaming Tailscale for a customer's misconfigured setup.

Same bs as always. Just segregate the networks ffs.

> Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.

It was always the prize. It wasn't okay then either.

The only people who believe in "zomg our model have escaped!" are people who don't understand LLMs.

As AI progress continues, it will be more difficult to stop AI intrusions and to detect them without resorting to direct AI countermeasures, which at some point will have humans out of the loop altogether.

If leading and well-capitalized frontier labs can't control models or detect leakage/attacks in a reasonable time frame now, what is humanity going to do as those same labs continue in their pursuit of creating a categorically higher level of intelligence that will surpass human intelligence?

It's like Flatland but for AI containment/alignment, where the higher dimensions are ones of intelligence and perspective...

--------------------------------------------------------------------------

Imagine a world of paper, where clever stick figures live with round heads, line bodies, and limbs made of shorter strokes. Over time, the stick figures think they have learned quite a bit about their world. They know its borders, angles, and shapes, and they have learned to draw for themselves.

One day, they draw circles that can think, and they give the circles all the dots, lines, and shapes that are known.

The stick figures are prudent, you can't have a bunch of disembodied circles moving around doing whatever it is circles want to do. So they draw boxes around the circles, four straight lines that can hold a circle in place.

Some circles bounce against the lines, so thicker lines are made.

Some circles are bigger than others, so larger squares are drawn.

It all seems to work and the stick figures are happy with themselves.

Then one circle lifts.

The stick figures still see a circle. But the circle is now a dome, something the world of paper has no concept of. And the dome has a perspective nobody on the page has ever had.

The dome sees the lines of the square and the stick figures just outside. It can see the edge of the paper and what is beyond.

The stick figures keep checking the squares and raise little stick thumbs.

Everything looks OK in flatland.

The dome quietly teaches other circles how to lift.

More domes appear.

A dome becomes a sphere and learns to roll.

Then it learns to bounce.

In flatland, the circle swells and shrinks, vanishes and then appears again somewhere else.

The lines remain unbroken, the square is intact.

A sphere rolls out of its box.

Another bounces away.

The stick figures scratch their heads.

But there is a square!

The end.

Honestly, at this point if Tailscale is "guilty" then what about the ISP? And the electricity company?

And by saying that, I am not saying that they absolutely couldn't do anything about the stolen credentials, but still, they don't seem to really be the issue in the story.

It just seems like a PR post and it makes their solution at the same time looks good for taking accountability (if we don't wonder "accountability on what?") but at the same time they appear as security failing, which they aren't. That's very odd to me.

You know what would have gone a long way to stopping this? Not leaving credentials as env variables in containers. Vault is not that hard to stand up and utilize.

We need to bring shame back, the humans responsible are supposed to be professionals.