raggi
10 hours ago
> Look at the Railway GCP account ban situation. A literally billion dollar startup running on Google Cloud and Google just randomly snaps their fingers and deletes their account. Zero warning. No phone number to call. No account rep. Poof. Gone. It is actual insanity to me. A billion dollar customer gets the exact same automated middle finger as a low effort spam bot. Your B2B business is completely cooked if that is how you treat people. The enterprise cloud gravy train is right there and Google is standing on the tracks begging to get hit by the train.
If you've been around a while you know that at any business critical scale at all you establish a relationship with your cloud provider and get an account manager. When you do this, you have a number to call.
A billion dollar startup not doing this is a keen lesson for the CTO.
Yes, Google likely screwed up here, but being unprepared for account problems, having no established relationship with your provider is a critical mistake.
The article goes on to talk about Hetzner as an example: their pricing is great for individuals but they literally don't even offer account management relationships - even at scale they actively refuse them. There are equivalent stories of account terminations with Hetzner, which is also a key point: this isn't just a big business problem, at all.
sylens
10 hours ago
Our team at work uses all three major public clouds and our Google Account Team is by far the worst of the three. Nothing like having to explain the same problem I’m having from the start again on every monthly cadence call because they don’t note anything or try to help resolve it
mistic92
2 hours ago
I had the same experience with AWS. They even assigned principal someone to our case and guy sent us old blogposts and did nothing. I took a month to find "hidden" checkbox. On GCP I always escalate to pass indian support as they are super incompetent.
verst
9 hours ago
Who is the best? Just curious.
manquer
8 hours ago
Microsoft. It more than compensated for Azure not being the best product . They are incredibly more responsive, you have multiple points of contacts and escalation chain of actual humans you can meet
they will even come to your customer call or connect with their rep already working with the mutual customer and so on.
AWS has the best tech and but not as good as Microsoft service wise, they certainly improved a lot last few years and it shows but because they don’t have any enterprise apps like MS their footprint is more limited.
Google keeps talking about GCP being important but doesn’t feel anything has changed on ground
lokar
8 hours ago
My company also used all 3 (at a very large scale / spends). MS was nice, but useless / incompetent technically. Anything non-trivial took forever to get a straight answer or resolve. We rarely got to speak directly to anyone with real expertise.
AWS, we could speak directly to Sr engineers on the relevant team. Full transparency, highly responsive. They were clearly trying to understand our issues and suggest change for both us and themselves.
Google was mostly useless. There was one team I got to talk directly to, who were great. But that was the exception.
DanielHB
2 hours ago
My experience with AWS hasn't been good when we had major problems in redshift becoming unresponsive. Since it was an intermitent issue and not a full blown blackout they just shrugged and we kept having problems for months.
jvuygbbkuurx
8 hours ago
Microsoft of course
isaisabella
8 hours ago
Why not Oracle
smallmancontrov
8 hours ago
Because Oracle = Larry Ellison = The Lawnmower
> You need to think of Larry Ellison the way you think of a lawnmower. You don’t anthropomorphize your lawnmower, the lawnmower just mows the lawn - you stick your hand in there and it’ll chop it off, the end. You don’t think "oh, the lawnmower hates me" – lawnmower doesn’t give a shit about you, lawnmower can’t hate you. Don’t anthropomorphize the lawnmower. Don’t fall into that trap about Oracle. -Bryan Cantrill
A few years back someone at work stuck their hand in the lawnmower. I've seen this happen a few times, but that time it ended with Oracle fining us for every VirtualBox install and the company sent The IT Spanish Inquisition around to make sure we all deleted it off our computers. Fun times.
senkora
7 hours ago
You honestly just have to treat any Oracle product as malware, and proactively scan for it / block it from being installed on employee laptops in the first place.
mc32
8 hours ago
They also seem to keep revolving team-members so you're always explaining things all over to new people.
aurareturn
10 hours ago
I would think that Railway certainly had an account manager, right? Did they say they didn’t have one?
MrDarcy
10 hours ago
The article is incorrect and misleading. Railway did have an account manager and they did call them and they did pick up the phone and work with them to restore service.
sreekanth850
6 hours ago
What is the use of Account manager if a billion dollar company got screwedup for straight 5 + hours?
aurareturn
3 hours ago
Without an account manager, you submit a ticket and no one looks at it for a few days.
With an account manager, you call this person up immediately, and they hound the devs in person or call up the devs personally.
Forgeties79
9 hours ago
An account manager overseeing such a major client should’ve never let this happen. If they don’t, why the hell are they the account manager? What are they even doing to earn their keep? This was such a preventable situation.
I once worked at a company that had their domain lapse because of an internal error at the company that was paid for the domain. There was no alert, there was no attempt to rectify the situation, one day we woke up and we simply did not have control of our website for a full week. There was nothing wrong with our payment method, there was no reason the payment shouldn’t have happened, it was completely their fault. They found out because we called them in a panic. This was a major company. We left them a week later and our CEO talks about it constantly as a horror story to other companies and there is no way our situation was unique.
It’s not just about the value of the contract. This whole situation has been in the news for days now. It’s terrible PR and I guarantee you it is costing them business in the long run. All I have seen for days is people talking about how poor Google’s support is, and I’m not even somebody who makes those decisions.
I get it, “Google is too big to fail.” But eventually, that stops being true
esrauch
8 hours ago
I'm not sure what you think account managers do that they can prevent accidents/bugs like that?
cco
6 hours ago
Typically a strong account team builds processes with other teams (compliance, engineering, etc) that enshrines and insulates important accounts from accidents like this.
In this case, I'd expect major accounts (and maybe Railway isn't above this level?) to be in a protected tier that is immune from automated suspensions like this.
If suspicious traffic occurs that _would_ trigger a suspension like that, the account team would be paged. Because this may mean your important account was compromised, shipped a bug, has been hit by something and you should immediately start working _with_ them to figure it out.
Fairly basic for a company with any customer management motion at all.
thibaut_barrere
6 hours ago
See https://news.ycombinator.com/item?id=48231538, they appeared to have everything you can have, and it wasn’t enough to prevent this.
ralph84
7 hours ago
The account manager was almost surely just as blindsided as the customer.
sergiotapia
8 hours ago
You are completely misinformed here.
Railway had dedicated reps and everything. Even the number of the head. One of their railway company leads said so on X or their Discord during the outage.
They paid for everything you can pay for to get Google on the horn and ready because they are not a tiny company. Yet it was still terrible.
lern_too_spel
6 hours ago
Spot on. This is the wrong takeaway. The account manager can't do anything to prevent the problem. They can only escalate, and that's what they did.
The right takeaway is you should not use Google Cloud because Google itself doesn't use Google Cloud for anything critical. I don't know how many times I've said this, but it clearly hasn't sunk in. A Googler once responded, "Well ackshually, Google Domains is built on Google Cloud." Google Domains isn't mission critical and has since been sold to Squarespace.
xbar
7 hours ago
Victim-blaming is not appropriate here.
Google did not "likely" screw up.
To be sure, however, this is a keen lesson for every CTO.
raggi
7 hours ago
i was responding to the article, i hadn't bothered to read railways pm as it's not super interesting, anyway, i've read it now, here: https://blog.railway.com/p/incident-report-may-19-2026-gcp-a... and indeed they weren't at fault