Item id: 48210590
4 hours ago
Related ongoing thread:
Incident Report: May 19, 2026 – GCP Account Suspension - https://news.ycombinator.com/item?id=48204770 - May 2026 (144 comments)
Previously:
Incident Report: Railway Blocked by Google Cloud [resolved] - https://news.ycombinator.com/item?id=48201484 - May 2026 (344 comments)
4 hours ago
It's an especially awkward situation because Railway is a competitor of Google Cloud, with many third parties involved. So, I just think it will take them a little more time to figure out how to message things.
To me, what it sounds like is that a Google Cloud system identified Railway as a misbehaving customer. Spam, hackers, that sort of thing. Often this happens for "platform as a service" companies, because Railway themselves probably do host some spammers and hackers, and they have their own systems for dealing with it.
So, it's quite possible that according to the Google team, Railway violated the terms of something or other, and according to the Railway team, they did not, and now everyone has to argue about it.
But who knows, this is just me guessing based on some experience running a PaaS that itself was running on top of AWS.
an hour ago
Railway's system for dealing with hackers is a $5/mo fee gate.
Railway plays around vibecoding as they go along, and their tech practices don't inspire confidence either. Unlike other PaaS like Render or Heroku, I doubt Railway has adequate rate-limiting to stop bad apples.
5 hours ago
I think so. I'm a GCP user and I'm afraid of hosting workloads there now. I've heard too many nightmare stories, and I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mistakes where there should be a way for smaller customers to cap usage.
4 hours ago
> I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for.
These sorts of things have happened before with Google and the other expensive providers.
Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
4 hours ago
> Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.
In my experience, the reason they're cheaper is because they offer fewer features (cut down on ongoing expenses) and because they aggressively enforce TOS (the margins are thinner, so you're less able to afford people using more resources than allocated).
4 hours ago
Would you be okay with GCP making public issues you encounter with your account?
4 hours ago
if it's some automated behavior, yes. I'd like to know what it is, why it exists, how it works (how to avoid triggering it) and what they're doing to make sure it never happens again
4 hours ago
Well, I would host workloads on GCP... provided I could easily move them elsewhere and I just treat them as disposable.
5 hours ago
What about Azure?
4 hours ago
I'd rather go back to bare metal than use Azure.
5 hours ago
Do you care about uptime and security?
2 hours ago
Azure is a toy platform, not a real cloud. Too many changes, far from being reliable.
Unless imposed by employer to use, run a mile.
5 hours ago
4 hours ago
I would rather raise goats, than use Azure.
5 hours ago
Honestly the best counterattack Iran could make rn would be somehow convincing the Pentagon to move everything to Azure
4 hours ago
Your statement was one of those that made me chuckle because it started as a joke...and then reality set in the more i thought about it, and then it made me feel sad, nervous, etc.
5 hours ago
I really don't see how they can. The business and usage details of their clients are confidential. We have their word that ToS where violated, I don't really think they should say more. This needs to go to arbitration.
4 hours ago
Is there a statement somewhere from Google saying the ToS was violated? I hadn't seen that and can't find one right now.
4 hours ago
Arbitration is an inefficient and unproductive process. I suppose that may be what the parties chose, but the public will likely never know what happened and the problem will be allowed to reoccur again and again. Things like these are better resolved in courts, with res judicata, precedent, and visibility so that legislators can fix statutes where necessary.
4 hours ago
> and the problem will be allowed to reoccur again and again
You're already placing blame here.
We don't know the details other than what Railway has said.
2 hours ago
If the ToS was actually violated and the suspension was legitimate, they wouldn't have reinstated Railway's account after five hours or whatever it was. Their behaviour is already a mea culpa.
3 hours ago
I think that's true regardless of what happened.
If Railway did something wrong, then letting that be known may help other customers avoid the same ~mistake.
4 hours ago
There's a reason we used to have courts with public records. Public records and transparency are a good thing. Binding arbitration has destroyed all of that.
4 hours ago
GCP is basically just a toy. If you're hosting something important there you should probably not do that. It works as a Plan B, but by no means should Google be your Plan A.
I don't know what happened in this case, there's a chance it wasn't Google's fault but it doesn't matter, Google already lost all benefit of the doubt long ago.
4 hours ago
Yeah this is the main argument I have for AWS. Is it the best tech-wise? Arguably not, but they have real support you can contact on a moment's notice, they'll even reach out ahead of time if they suspect something is seriously wrong.
3 hours ago
Shopify is all on GCP and works well.
3 hours ago
Shopify has $11 billion dollars in revenue. I assume that buys them at least an email address of a human at Google.
Nobody doubts Google's engineering ability. It's their soft skills that are lacking. Google is culturally unable to understand the concept of customers.
3 hours ago
Xoogler here. This hits hard; right on the kidney.
That is a huge problem at Google: "we're google; we know what you want and need better than you do." That and an unshakable confidence in their own infallibility.
When I was there, I remember saying, "we do some cool stuff, but always remember that hubris is the death of empires."
5 hours ago
Would you want your vendors publicly disclosing potentially private reasons for an outage?
5 hours ago
With consent, yes.
5 hours ago
Is there any indication Railway has consented to such disclosure?
4 hours ago
But Railway has been blaming GCP for the outage. Shouldn't GCP be given an opportunity to defend itself?
4 hours ago
Railway hasn't placed any blame that I've seen. They've posted a timeline of what happened, without any speculation on causes.
4 hours ago
Is that what you'd want your vendors to do?
5 hours ago
yes, especially since this didn't seem to be exactly "private" where it was anything specific, it was just some kind of automated system without a human in the loop
5 hours ago
But shouldn't that be disclosed to Railway, and not the public? If they had someone running a botnet on compromised accounts there, for example.
If Railway isn't satisfied with the explanation, they're able to say so publicly, yes?
4 hours ago
if it wasn't something specific to their setup, it should be disclosed publicly, because this is a catastrophic incident that makes you think it could happen to you as well, and there's no way to know what could trigger it
4 hours ago
> if it wasn't something specific to their setup
They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
4 hours ago
a huge account like theirs should not be subject to automated actions like that.
an entire gcp project deleted along with its persistent disks.
how does that make any sense? nobody thought to call them or anything
4 hours ago
> a huge account like theirs should not be subject to automated actions like that.
No matter how damaging the behavior?
> an entire gcp project deleted along with its persistent disks.
Railway doesn't say that - "persistent disks inaccessible", followed by "persistent disks restored to ready state". It was a suspension, not a wipe.
2 hours ago
yes and yes, inaccessible where they had to be recovered..
there's like 0% chance there was domething super damaging going on where they couldn't get anybody on the phone yet within 10 mins or so were able to get the restoration process going with their account managers
I dont see what could be going on where an automated process would have to step in except for something like suddenly provisioning infinite amounts of resources, but quota limits should hit first so...
3 hours ago
If it were google's bug as Railway has certainly at least insinuated, then yes. I would also be fine with them saying "blah blah blah abuse was detected; we're working through it with our customer and we apologize to those impacted."
I'd also expect a story around how it is this happened w/o a human spending at least an hour working his/her way through a call list to reach someone at Railway. Starting with ops and escalating to the ceo if necessary.
6 hours ago
This is actually scary. If Google can suspend a company like Railway without warning, what chance does a smaller startup have? The lack of any human escalation path at Google Cloud has been a known problem for years. You'd think enterprise customers paying real money would at least get a phone call before getting shut down
4 hours ago
Then you should be scared about every single online account you use, since they all have this same capability of suspension of your account. That's inherent in any service policy.
4 hours ago
Google seems to be the only one that actually does it, regularly, with no warning or comment, to other people. Google also spreads bans to accounts it deems related, so your personal Google account is likely to be deleted at the same time as your GCP account. That includes your Gmail address and your phone login and cloud storage.
And vice versa. Company accounts have been deleted because an employee's personal account tripped Google's random number generator. Again, only Google does this.
5 hours ago
Google really should publish a flow diagram for how they decide to turn off someone's business.
4 hours ago
It would just be a control flow graph of rand(3)
2 hours ago
I’m surprised when I hear about anyone depending on GCP.
Why would you do that?
Seriously, what are some good reasons to choose GCP over vendors that have demonstrated to be much more reliable? Are they much cheaper than the alternatives? Do they have unbeatable features?
2 hours ago
They offer $250k credit for startups, that's the only reason I can think of. You use them for a start, and stuck there.
5 hours ago
Being an advocate for GCP all these years, I can only say the earlier you get out of it, the safer it is for your business. All it takes is for their automated system to go haywire, and you can say bye bye to all your goodwill and customers. Go look at twitter how many customers are blaming railway. Founder had history getting screwed by GCP, yet still choose to depend on them.
You can't rely your business on GCP. Honestly, this is the most silly way to kill your own business.
For context, copied from my post 3 years ago.
March 10, 2023 | hide | past | favorite As a 4 years customer, our production severs have been suspended by Google Cloud because we didn't fill up some information on-time. Contacted support but they expect us to wait for 24-48 hours to get it resolved while all our servers are down. Anyone linked with someone powerful in google cloud can help?
======
- Running production on google cloud for 4 yrs with my startup. 100% legit SaaS business.
- Always pay bills on-time no issue. Good customer never open tickets, ask for help or what just quietly pay my bills each month.
- Our servers was abruptly suspended yesterday midnight and my whole business is now down for > 10hrs.
- We run a SaaS business that other ecommerce stores rely on and have hundreds of paying merchants.
- My customers have been grilling me and I don't feel gcloud's trust and safety team understand/care how urgent the issue is.
======
Why were our servers suspended? Because we didn't fill up information in time?
- See https://imgur.com/a/x0Y3RJl
- Apparently they dropped us an email 10 days back that I missed out
- Titled "Important Information Regarding Your Google Account" with no indication of suspension or what in title.
- Given the number of subprocessor "Important" emails they send it's too easy to miss out the email.
- 10 days gone by and our servers were abruptly shutdown with zero suspension notification or what.
- We've been paying $400-$700/mo for the past 4 yrs consistently and they shut us down because we didn't fill up some information?
When I tried to ask them to at least temporarily get our servers back while the verification is ongoing, I didn't get any answers.
Google Cloud have zero empathy for customers.
It's not like my account got suspended for fradulent issue or what. It's suspended because I didn't fill up some information on-time and they don't even allow me to temporarily reactivate my services or what. Especially when I had to wait for hours to get their team to verify my details before I can get my servers back.
You can't trust them with your business. Don't run any production stuffs with Google Cloud, ever.
4 hours ago
For privacy, B2B providers often won't even acknowledge that any given company has an account, let alone publish information about that account's standing.
3 hours ago
No warning, no email, nothing. Just you find out your business is down the same way your users do. Google needs to explain this.
4 hours ago
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
I'm interpreting that bit from Railway's blog to mean it wasn't just them that was impacted.
5 hours ago
It depends on what you mean by ‘need’. If you mean they should for PR purposes, I probably agree.
If you are saying they should be required to by law, then no I disagree.
5 hours ago
I don't suspect anyone here is demanding laws be written requiring every single player in any SaaS outage to make a public statement immediately following an event like this. That's an odd thing to state your preference on.
But, given that this incident unjustly caused real damages to another company, I am pretty certain that Google will be required to make some sort of response to this, and if it ends up in the courts, it will be public.
4 hours ago
I can't believe that readers of HN actually think that that is how it does or should work.
Google/GCP can only make very general statements and in this case we want more than that.
They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them.
Either way, we need to hear about this from Railway.
4 hours ago
It's entirely plausible Google won't tell Railway without an NDA to prevent them disclosing exactly what set it all off.
The bigger point though is Google really need to flag any business account as not subject to these suspensions until checked into by several humans. Back when I had a team that used a lot of App Engine they would even call us when we caused all their pagers to go off, and then conspire to keep the lights on while things got fixed. It's sad they have ended up like this.
3 hours ago
> It's entirely plausible Google won't tell Railway without an NDA to prevent them disclosing exactly what set it all off.
That case is called:
> or Railway can tell us that Google is refusing to tell them.
I'm not paying you (which let's face it, that's what an NDA is) just to find out why you messed up about as severely as one can imagine. In theory, there was a contract here: $ for cloud services, and the rug got pulled. One should get a very clear, and very apologetic explanation as to why, with no strings attached, or one should be voting with their wallet.
Now, whether Railway will do any of that, who knows.
4 hours ago
They did:
"We take full responsibility for the architectural decisions that allowed a single upstream provider action to cascade into a platform-wide outage, and detail below what happened, how we recovered, and the changes we are making to prevent this from happening again."
My guess is they will be switching away from Google Cloud. Because anything else would be nuts.
4 hours ago
I know hating GCP is hip, but why do only a minority entertains the possibility that Railway did really something off to trigger some alarms?
The calibration of the alarms might be off, and that's acceptable, but in the end if something can be held wrong, somebody will hold it wrong.
I did the same thing in the past, albeit in a much smaller scale. There's no shame in being wrong and admitting as long as it results in progress, so this stance of "we do nothing wrong" from both parties is getting a bit old now.
4 hours ago
Of course it doesn’t work this way but why shouldn’t it? It doesn’t because Google is a massive company and could kill dozens of Railways before they notice an issue to their bottom line. However in a world where companies care I’d expect them to make a statement.
4 hours ago
I took this a different way which was that to google railway is their customer and out of a variety of professional and security considerations want the communications to come from their customer and not them.
4 hours ago
I think there’s been far too many Google/GCP ‘suspensions with no human in the loop’ that Google does need to put out a statement about their practices.
4 hours ago
The only way is for PR to think they should open this
2 hours ago
Google fucked up. They should convince people that it won't happen again.
6 hours ago
Totally agreed the news like this makes me nervous too. For a high profile customer it feels like Google should give a clear explanation of what happened.
4 hours ago
> For a high profile customer
Are they really "high profile"?
I've never heard of them until this incident or maybe the other one with the database.
They just seem... Loud, publicly. And always about failure.
an hour ago
They publish some stats at https://railway.com/stats
72M deploys from 3M customers across 10M services in the last month.
4 hours ago
In this sort of community, fairly well-known yes.
https://hn.algolia.com/?dateEnd=1779148800&dateRange=custom&...
5 hours ago
The company I work for uses GCP and we preciously had intermittent CloudSQL connection errors for a few hours. We reached out and they resolved it after a day or so and said there was a minor incident but I don’t think it was ever publically reported.
5 hours ago
Yup, I think so. Makes one think about how dependent we are on cloud infra for core pieces versus supporting pieces of the architecture. They've probably negotiated some kind of private settlement.
5 hours ago
I've directly asked our account manager about it. It's pretty scary that we don't know what automated mechanisms could just cut us off.
5 hours ago
I really think Google underestimates the damage they've done to their reputation with these. These incidents are rare, but they're common enough where you can't trust them reliably anymore, and if that's the case, why would you pick them over other vendors?
it's not like they're the only provider, or even the #1
4 hours ago
> why would you pick them over other vendors?
Their security track record is pretty exceptional compared to the other two.
3 hours ago
> Railway's GCP account manager engaged directly
This would have been an amazing conversation to witness.
4 hours ago
Maybe if it was a US Gov site going dark, and Congress got involved. Maybe then. Maybe not.
3 hours ago
Stop calling Google Cloud, “Google”.
Completely different group of people, culture and incentives.
Google Cloud is another company under Alphabet.
2 hours ago
They brand it Google, trade it under the ticker GOOG, and are run by the same CEO.
4 hours ago
This makes me so mad, Google is not the first i've seen doing this (AWS, Upwork, Twilio) - and the cheek to say precisely nothing about why. If you're a buyer of cloud services right, i bet old-school on-prem is looking as tempting as an actress to a teenager rn.
Every regulated firm running on GCP is going to spend Monday explaining to their board how their resilience plan accounts for a hyperscaler that operates this opaquely. The compliance paperwork is the easy bit - the honest answer is we trusted that a hyperscaler would behave like a utility and they didn't.
So yes - they owe a statement. The whole point of paying hyperscaler prices is the assumption you won't wake up suspended with no explanation.
5 hours ago
This is exactly why people get nervous about platform risk
5 hours ago
Google has given a public statement about this category of incident (to wit: cloud provider imperils customer's operations by way of automated decision deliberately designed to withhold recourse).
That statement is the last 15 or so years.
3 hours ago
Should they give a statement? Maybe. Should they be required to? Not if they don't want to. Should they be expected to? Historically, nope.
4 hours ago
> Since this is a relatively high profile customer standards
This is a small unknown startup.
5 hours ago
Railway can simply move to other service. We all know Google in unreliable, so why should google give public statement?
Thanks.
5 hours ago
We already know the explanation. It's fundamental to their business model and continued existence to automate everything, false positives be damned, and they don't care about all the people who roll snake eyes on a given day. Because everyone just stays and keeps using them.
4 hours ago
Yep. Google doing Google things. Not everyone stays and just keeps using them though, we're actively planning to remove GCP from our primary workloads now and will cut our spend to about 1/10 of current as we keep them in the stack as a cold multi-cloud failover target only.
4 hours ago
I don't understand how you see what Google did to Railway and think you know what I'll build my business on GCP.
#1. you will most likely never be as big a customer as railway. if they did it them, you're fucked.
#2. if shit hits the fan, even a company as large as railway can get no human being on the phone. that's insane.
how is gcp still a thing after this event? if you're in charge of tech at your company, how do you stomach choosing this? how do you defend your choice to your CEO?
4 hours ago
Google is a lawnmower, just like Larry Ellison. You don't expect the lawnmower to give a public statement about why it chopped your hand off.
5 hours ago
Post your ask in the discussion: https://news.ycombinator.com/item?id=48204770