If it's a serious key like AWS or Zendesk then you're probably screwed the instant it gets stolen.

Live by the sass, die by the sass.

No, a leaked credential is a nothing event in a mass leak as far as expected loss goes.

You’re better off waiting and monitoring until you’re sure you can get back online if your service supports any kind of revenue.

how the hell are they able to resell the keys and be able to take profit from that is beyond my imagination. Is not money what they are stoling and they are still able to take advantage of it.

With cloud credentials I understand that they are able to create machines, but with api keys... what they are able to do to take profit from?

They're actively firefighting an incident, and are likely going to make some mistakes along the way. In this case, they've erred on the side of security over availability, which in some cases is the better outcome. I know not in all, but some.

They did.

thought I was the only one, openrouter is auto disabling railway affiliated keys

Because you have no choice. Often times you run your backend on someone else's computer (the cloud) and they need the key(s) they need in order to access the other services your application needs to do its job.

I guess I picked the wrong week to stop sniffing glue.

Related ongoing thread:

PingCap TiDB Leaks - https://news.ycombinator.com/item?id=50173716 - Oct 2026 (32 comments)

(It had only spent 3 hours on the frontpage so I re-upped it.)

I find it wild how much they have downplayed this in their statement. https://www.pingcap.com/trust-hub/security/security-update-o...

”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”

hmm, I don't think that's true.

That was a single company, the cert problem hit something like 70% of all websites and devices.

Honestly…I’ve never heard of pingcap

Jesus that one is bad

Also Shopify, they have been delisting a lot of apps without giving a reason, most lickely a supply chain attacked that spoofed/stol checkout data

Madness..

true =(

I find it very funny when people put the link of their company in their bio (that's fine and normal) and next to it the ARR it brings (who cares ??). It *really* makes you look like those influencer-entrepreneur-scammer who claim to have 10 side projects making 2B ARR but still somehow spend their time selling you courses on how to be like them instead of taking care of their companies. Your client do NOT care about your ARR (well to an extent), so why is it something you are so keen on advertising

You may think that sounds like a very nice salary but $4MM annual gross is a very small business. I can’t imagine how Railway would be in business if that were considered a large customer.

300K ARR means you get 300K in annual recurring revenue.

Subtract costs for employees. Which, given current economy and inflation, that gets you a founder scraping on their savings, one or two other people (remember that if you're paying someone unless it's totally offshored through a house which has it's own inflation, you have other factors) and then your own taxes... yeah... much more than that and you are in the black.

As a general factor I keep in mind, when I worked at a 'normal' but toxic place before I was an actual software engineer, my manager's rules were something like '40 percent L2R' (might have been lower) for in-house employees and 70- 80% L2R for contractors.

IOW, for every dollar billed for an actual employee, we paid the employee 40 cents. For every dollar billed for a contractor, we paid them 70-80 cents.

I've heard similar figures from others (e.x. 'in house employee has a real cost of 1.5-2x their actual pay').

And yes it sucks and yes the structure at least here in the US is a factor.

Lots of companies run on railway

Why not?

Why are you even commenting? And it's Railway not raleway.

Rob: They're not real. You get that, right? None of it is real. The critics aren't real. The customers aren't real. Because... this isn't real.

Chef Finway: [laughing awkwardly] Okay...

Rob: Derek, why do you care about these people? They don't care about you. None of them. They don't even know you because you haven't shown them. Every day, you'll wake up, and there'll be less of you. You live your life for them, and they don't even see you. You don't even see yourself.

[long pause; Derek's tight smile has slowly faded into a look of sorrow and regret]

Rob: We don't get a lot of things to really care about.

Thanks, yes I will. Left without the laptop(s). Let the chips fall where they lay. If they need me back so urgently, they will have to go through the full cancellation of booked flights and hotels first. Lets enjoy the time off.

Strongly concur

    > Bus factor of 1

Correct me if I am wrong - if the API keys are compromised, then a large bill could be racked up?

It doesn't sound like they got them via a shell environment, they just hacked into the place that stored them. If the place you store secrets can be hacked into, it doesn't really matter if they are injected into environment variables or not.

> Use environment variables to point to where secrets are located, not to store their values.

Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.

Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself.

How does this help? If your container gets compromised it's trivial to exfiltrate the secrets.

Or we could just use our operating systems for the things we do well, and have a service to send our API calls to that inserts the appropriate token.

I don't have to enter my Mac's password every time I open an app.

systemd makes it easy: https://systemd.io/CREDENTIALS/

> hot patching will need to itself become fully automated.

Great, even less control over the things we use and more power ceded to corporations.

Hot patching is itself a liability

Static HTML, yeah.

Not sure about VPN. Seems likely to provide a false sense of security.

Still some industries where you cable colocated servers in a shared DC together where neither server has internet access.

>We are going back to static htmls for public pages...

God yes, please. I backpack and mountaineer in the PNW and have been trying to research trip ideas up in British Columbia, and as a result I discovered https://bivouac.com.

Here's a website that's still frequently used by a large number of people as a valuable resource, is regularly updated and maintained, and yet looks like it's straight out of 1995.

Every pages loads instantaneously, everything is simple yet packed with detail, and it's such a basic, straightforward, delightfully pleasant experience to just browse through the site. I was so thrilled after just 5 minutes on the site that I threw them $25 for a year's full access.

Sorry for the tangent, but oh man, it's such a refreshingly stark contrast to the rest of the bullshit that fills the web these days.

Ha, Cyberpunk 2077 is becoming reality. We're proposing the Blackwall.

"So you want to build your own data center?"

https://blog.railway.com/p/data-center-build-part-one

I think we'll see a reduction in complexity and an increase in transparency, rather than just on-prem.

One medium-sized server can now easily serve the entire banking system of a large country. So you don't need clusters of servers with careful tradeoffs between consistency and availability because of the CAP theorem.

Just get a big honking host, slap Postres on it, and run your application server in a VM on the same host. For backup, export the WAL logs from Postgres onto your favorite off-site blob storage. UI for it? There's Pigsty.

It still makes sense to host this in a cloud provider, just because you probably don't want to deal with actual hardware. And maybe get a second standby server for redundancy.

That's it.

You'll need a bunch of external dependencies for email, 2FA, observability, LLMs, and so on. But they should all be outgoing dependencies that should not allow access back into your system.

How does on-prem help if your product is a SaaS.

> Our thought process was that we didn't want to recommend that people roll keys when we couldn't possibly confirm that we confidently plugged the breach and rolled the mitigation(s). I am comfortable to publicly say that this was a highly motivated attacker. Meanwhile, our threat team mitigates DDoSes, CVEs, while been patching everything immediately as dealt with exploits.

> As for this breach in particular, we tried to balance a duty to warn, which we take very seriously... However, it took a significant amount of time on the order of hours to verify, reviewing, and keep up with the attacker as they attempted to move laterally around, and briefly, within the system. We didn't want message anything beyond a shadow of a doubt that there were no further exploit paths.

Once you realize there is compromise, you message accordingly, as soon as you are able. If you can't immediately determine exactly what subset of customers were affected, this means you have to tell everyone. Then you provide follow-up updates to fill in gaps as you develop your understanding of scope+impact and work on remediation.

What you should never do is hold off on reaching out to possibly affected stakeholders that they may be affected by a compromise because you still have uncertainties about current status.

A situation like this does not in any way a constitute a dilemma or tough balance regarding when or to whom to do initial disclosure - you do it as soon as you can, to everyone except those you know are not impacted.

> and although we have been up for days on end- I don't blame you if you don't have sympathy for us.

From my timezone, this started last night and is ongoing into today. Has this been going for longer than the status page and email indicated? If so, why was notification delayed so long?

I love your product, have an incredible amount of sympathy for you as humans (having been in similar sleep-consuming nightmares) and was willing to forgive the GCP-triggered outage as an understandable teething pain, but sadly this response has really knocked my confidence.

I understand the hard reality of security issues -- "urgently rotate your keys, they're already compromised" is an important message to share -- but "sorry, that will also require your site going down because we've blocked deploys" (presumably as a response to the incident...) is an incredibly frustrating situation to find ourselves in.

"aviate, navigate, then communicate" only matters if you are the one in control of outcomes.

Customers could have simply revoked keys and not re-launched services with rolled credentials (that from your post, may have been re-stolen), but instead you let this fester.

What a disaster for Railway. I hope you get through this stressful moment, but this should be the end of Railway.

You know... I get it.

  Railway / OpenAI: I had a key compromised this morning. It hit our limit, so I revoked the key... and figured, well, we'll have to track down the leak.

  Then this security notice came out, telling all of their customers to rotate all their keys... As the guy whose spent the entire evening rotating keys and etc. doing

  this is... it's a huge message to send.

  The cost and implications go really far.

  We have many production services that we've taken offline or disabled because we can't deploy to fix the credentials. because of the railway deploy problem...

  Railway was stuck between some pretty nasty tradeoffs. I get that the few hour delay in communication caused some of us more cost, issues, and potential breach... and maybe larger suff.

 But I also realize that rolling the keys while things may still be compromised, with deployments blocked, could make thins even more sticky
  leaves only very bad options. Also we don't know how much they we're spending time actively stopping the attack. the time and focus could be very much consumed. if it was 5% of exposure and they prevented the 95% exposure.. i'm sure glad they aviated instead of communicated.

  We make the best decision we can in these moments..  and they all just suck.

I was a loyal railway customer for a while, but this response smacks of "we didn't want people to know" instead of "we wanted to take responsibility."

The options railway has given us are "rotate your keys and bring down your service" or "keep your service up and let the exposed keys remain at risk." Kind of a crap position, no? Mistakes are going to happen, but when the suggested mitigations are to stab yourself in the leg or stab yourself in the ass, it's hard to be sympathetic.

I have to imagine quite a few users are asking their agents to port their stacks over to alternate providers like I am. The cost of switching has never been cheaper thanks to AI.

IMO "aviate, navigate, then communicate" does not work in this situation. There is more than one pilot. The moment you detected a breach, hell, even knowing that an attacker was inside your system, you should have pointed to anyone. Hell, even AI, to draft a disclosure with more info coming.

yeah .. this is the suck

In general, the new host will have its own problems, otherwise you would've used them to begin with.

I'm curious where you decided to go

Saw someone on Twitter indicating that they did this, and new keys got compromised.

It appears to work in the UI, but go test your product. Mine didn't receive the new keys.

I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.

The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.

It already _is_ a rough decade. I have hated the 20s so damn much.

Yeah, the future of our jobs in this industry... for the type of work we do. are concerning. Explosure has whole new meaning

It feels like the only solution is going to be proactively running your own AI agents looking for patches and applying them.

It is… for their business.

They built the systems that are not in the news right now.

It's a nice saying, but if you did enough things right, the attacker still needs multiple things to line up in order to get anything of value, if you get everything right. No one competent who's doing everything right is leaving open S3 buckets open to the world. Doing some things right means using multiple buckets, using multiple time limited-keys with limited read-only access, using special write-access, not having admin keys on developer laptops, a SIEM system that monitors buckets for traffic going in & out and sending emails (both amount and who it's going to) to the person on call. A firewall (inc IP-address-based blocks so eg China can't access the customer data from China at all). A WAF.

That's not an exaustive list but if the attacker gets one of those "right" once, it is not game over. (They aren't of equal strength, mind you.)

We (Railway) make no claims about the tools the attacker used in the above post.

However, from discussions with partners, it resembled the pattern of attacks to secure AI API tokens and credentials. Preliminary forensics tell us that this was the primary goal of their attack. That said, I want to be careful here commenting until we have our formal public post-mortem.

So far I found they accessed my S3 buckets, used my OpenAI key (which OpenAI thankfully detected odd behaviour and revoked the key automatically), did readonly operations on my Valkey instance, etc.

The secret ones

Take coolify admin off the web, put it on a private address only accessible via tailscale

No, and this isn't about a supply chain attack on Python libs. This is about running your service(s) on a service that runs in front of a cloud provider. My argument is simply that it's needless extra risk

You do have an argument to be made about vendoring our dependencies, or removing them in general. Not a bad idea