It seems amazon itself is aware of this issue. The linked engadget article even mentions this:

> "The rate at which Amazon has burned through the American working-age populace led to another piece of internal research, obtained this summer by Recode, which cautioned that the company might “deplete the available labor supply in the US” in certain metro regions within a few years."

Seriously. I don’t know any half way decent engineer that would ever work there twice.

There seems to be some sort imputation that, just because someone is on an H-1b, that they are not a good engineer.

I used to be on an H-1b and gladly came back home to India. I run my own business now. And yes. I'm ex-Anazon. It was a tough place to work, but circa mid-nineties, the stock options made it worth working for them.

I'm willing to bet I'll outcode a significant fraction of the audience on this site. And I'm not even close to the best developer around. Some of the smartest people I've met have been on an H-1b visa. Please consider not letting prejudice affect your view. You'll do yourself a disservice by underestimating your competition.

Yes.

I would never work for AWS, given what I've heard, and consistently, of their internal culture.

Also, everything I've seen while working with internal staff makes me feel there's a culture of obfuscating all weaknesses from customers, practically to the point of deceit.

Are there really many illegal aliens working in the warehouses? I know that Amazon does verify employment eligibility and checks documents. There may be some committing identity theft but I doubt that it's a large proportion.

They pay very well, for some roles.

I have a friend who recently started there. He just brought a Mercedes, and a second house, and he’s still in his early thirties.

They keep him busy, though.

Not just firings. I sure got a deluge of Amazon recruiter emails right after they went full RTO.

> tens of millions of illegal immigrant warehouse workers

You have a source for that claim?

Given their self-imposed labor supply issues, notably the awful working conditions, I hope the workforce there figures out how to effectively organize against this cesspool of a corporation.

The article wasn't about the outage happening, it was about the amount of time it took to even discover what the problem was. Seems logical to assume that could be because there aren't many people left who know how all the systems connect.

Indeed. No disrespect to Justin (great person) or any of the engineers who were sacked but Corey's post here is basically "here's someone who was sacked, and here are several other layoff news". AWS is really big organization. Several orders of magnitude bigger than people who were remote/refused to RTO. Organizations like this survive these brain brains.

Seems like it worked fine. They laid off a quarter of their junior principal engineers, the stock went up. They had a massive outage a few months later, the stock went up again. Everything's working out fine for their strategy so far.

Nah, the decades old crop of "new" big tech companies are just entering their IBM phase.

I don't think there's ignorance of the fact that turnover is bad, I think the field is being designed to homogenize staff and favor uniform mediocrity so that employees truly do become interchangeable. We're so close to just plain talent being likened to cowboyism.

Still, by the time the failure start, that would have been the beginning of the day in Europe, so I suppose those teams got the page

Maybe it was still at the end of Indian day but together with the holiday I'd say that makes it more unlikely to be handled there

> It never made sense to me how such a ruthless and inhumane culture is sustainable in the long run.

It’s pretty simple, actually. Once such a dominant market position is achieved, you can get away with almost anything, whether with customers or employees. This is true of all the BigTech companies.

I think it comes down to demand and supply for jobs.

The only time Amazon was forced to change its ways was during Covid hiring boom where they couldn't compete in the talent market. They were forced to increases their salary bands and the culture was also a bit easy during that time. But starting mid 2022 it's been an employer's market and Amazon is making sure to juice every bit out of its employees while it can

I thought the recovery was early AM Seattle time (like 4am). Where I think start-of-day is like 9am. Maybe recovery started early (6am) New York time?

I noticed that too. I think tech culture has to change a bit. Silicon Valley is a great location if you're making hardware or prepackaged software. If you have to support a real economy that is mostly on the East Coast you need a presence there.

> Is 75 minutes really considered that long of a time? [...] When I worked on firmware we frequently spent _weeks_ trying to diagnose what part of the firmware was broken.

One might spend weeks diagnosing a problem if the problem only happens 0.01% of the time, correlated with nothing, goes away when retried, and nobody can reproduce it in a test environment.

But 0.01%-and-it-goes-away-when-retried does not make a high priority incident. High priority incidents tend to be repeatable problems that weren't there an hour ago.

Generally a well designed, properly resourced business critical system will be simple enough and well enough monitored that problems can be diagnosed in a good deal less than 75 minutes - even if rolling out a full fix takes longer.

Of course, I don't know how common well designed, properly resourced business critical systems are.

For a service like AWS, 75 mins is going to result in a LOT of COE's for people on way it wasn't mitigated quicker. A Sev 1 like this has an SLA of 20 mins to mitigate impact. Writing about these failures will consume a dozen peoples time for the next 6 weeks.

I have 10 years of experience at Amazon as an L6/L7 SDM, across 4 teams (Games, logistics, Alexa, Prime video). I have also been on a team that caused a sev 1 in the past.

>Is 75 minutes really considered that long of a time?

From my experience in setting up and running support services, not really. It's actually pretty darn quick.

First, the issue is reported to level 1 support, which is bunch of juniors/drones on call, often offshore (depending on time of the day) who'll run through their scripts and having determined that it's not in there, escalate to level 2.

Level 2 would be more experienced developer/support tech, who's seen a thing or two and dealt with serious issues. It will take time to get them online as they're on call but not online at 3am EST, as they have to get their cup of joe, turn on the laptop etc. Would take them a bit to realize that the fecal matter made contact with the rotating blades and escalate to level 3.

Which involves setting up the bridge, waking up the decisions makers (in my case it was director and VP level), and finally waking up the guy who either a) wrote all this or b) is one of 5 or 6 people on the planet capable of understanding and troubleshooting the tangled mess.

I do realize that AWS support might be structured quite a bit differently, but still... 75 minutes is pretty good.

Edit: That is not to say that AWS doesn't have a problem with turnover. I'm well aware of their policies and tendency to get rid of people in 2/3 years, partially due to compensation structures where there's a significant bump in compensation - and vesting - once you reach that timeframe.

But in this particular case I don't think support should take much of a blame. The overall architecture on the other hand...

Also it's pretty likely it took less time than that to get an idea, but generally for public updates you want to be very reserved, otherwise users get the wrong impressions.

Depends on what you're measuring.

Quite a few of AWS's more mature customers (including my company) were aware within 15 minutes of the incident that Dynamo was failing and hypothesized that it'd taken other services. Hopefully AWS engineers were at least fast.

75 minutes to make a decision about how to message that outage is not particularly slow though, and my guess is that this is where most of the latency actually came from.

75 minutes is damn good turnaround for any major problem, IMO

If you a regular company, no. If you are the biggest provider in the world and your change is breaking large parts of the Internet, yes.

Amazon is supposed to have the best infrastructure in the business because everyone else runs on it. They should have access to the sre talent that can quickly mitigate this kind of issue

I also love the humor and personality that authors are able to inject into their pieces

I nearly missed that this article was written by Corey Quinn, guy who writes a lot about AWS

They never shirk from calling anything a spade.

Whether it involves FAANG companies or not, a job is ultimately just a job. While it's nice to have such a company on a CV and to gain the experience, it is, in essence, similar to any other employment.

Eventually, you begin to consider the drawbacks, such as the monotony of the work or the exhausting nature of on-calls (which disrupt personal life). Then, an opportunity arises from a former colleague at another company, and the outcome is predictable.

Companies present numerous such inconveniences and actively introduce additional ones. Now, we are faced with mandatory RTOs, along with the continuous tightening of the screws and "cutting fat from the bone" (actual words of my company's CTO). Consequently, employees will depart, and it is often the high-performers who will seek opportunities elsewhere, as they are not afraid of the job market.

I think google is an excellent place to work, but the top AI companies do poach some of our good employees. If you're the cream of the crop, sure, take the Anthropic offer over G. But I do appreciate the massive amount of freedom to work on hard problems.

For internal addressing, why do we use DNS? In case of an address swap (hardware swap, load balancing or whatever reason), one could broadcast it to the network, and the relevant clients would acknowledge & update the address.

For super critical entries, we put it in /etc/hosts and also at the domain provider as fixed entries.

The combination never fails.

how many lines of code is it though? or how many tokens? we need something we can measure if we're gonna fire people!

I think you missed the point of the article. The layoffs are blamed for the poor response, not the outage itself.

Doesn’t an increasing LLM centric code base only compound this problem? Under the assumption that people are lazily screening LLM’s output when using them

cue in: programming as theory building [1] or building systems as theory building, ie, Mental causal models of how and why things work the way they do. Mental models live in people's heads and walk out of the door when they do. Management learns this the hard way [2].

[1] https://pages.cs.wisc.edu/~remzi/Naur.pdf

[2] https://x.com/elonmusk/status/1980221072512635117

But it is more important to keep the PIP, stack-ranking, ladder-climbing game running than keeping the people /s

Great words, but he lost any right to them when he made famous the "You're holding it wrong" workaround. IMO that was the defining moment when Apple started its decline on product innovation.

I’d add a corollary to Steve’s often quoted idea, that became even more relevant after Covid. Everyone competent that makes tons of money retired early. We are left with the dregs at all these companies, the newbies and those that didn’t budget well and plan for early retirement.

Any interaction you have with a company post-Covid you can feel it. Nothing works anymore and you can’t even tell anyone about it or why.

FWIW this has happens in consulting too, not just product companies. Just swap “product” for “delivery”.

That could be totally unrelated - I had the same thing happen last month. Eventually (~3 attempts over a week) managed to fight through the various chat options that tried to dead-end into "wait 3-5 business days and check again", and once I had a person on they took about 5 minutes reading the background and said "oh, I can see that this return was processed months ago, and we have it listed as received. I will return the amount you were charged immediately!"

Years ago I interviewed for Amazon. I found out that on a desired team size of 8, there were only 3 left, where the other 5 who were no longer there, as I was told, were poor performers. I didn't take the job.

Every good Amazon org has sacrificial bad employees. Managers know a team of all-stars creates risky team dynamics under PIP

What's PIP?

There's been a massive talent exodus, especially among the principal and senior principal engineering roles, across all Amazon orgs since the RTO policies have been enforced. Its demoralizing to lose key engineers that you look up to and want to continue to learn from all because a few people far removed from the day to day make a bad call.

RTO in combination with Amazon being last place in AI innovation have led to departures of anyone that can leave, leaving.

> So they can't suggest a fix even if they know 100% what it will be. Thats exactly what happened this time. EIGHT different staff members pointed to the underlying cause and were told (some literally) to "shut the f*ck up and get back to your job"

Jesus, if even an ounce of that is true... Yes, everyone on the internet is a cat clawing on a keyboard... but if a ton of people legitimately confirmed to be ex-AWS point to similar culture issues... probably it's AWS that's rotting.

I appreciate how you, with absolutely no evidence, blamed this outage on DEI, especially given Amazons complete backpedaling of DEI the first moment it became political convenient

The real story from this incident is that Amazon’s “aws” partition doesn’t actually have multiple regions - effectively, it’s all IAD in a trench-coat.

This is a big deal. My employer has already started to look at bringing back our old racks from storage and switching back to on-premises. Cannot imagine he’s alone in that.

Normally, you'd have drills and exercises to try to find and fix potential issues before they affect production. Saying that you can learn from failure is great comfort to the individual, but not an excuse for the institution.

Given today is Diwali, perhaps the reason everything went down is because the best and brightest from India were all on vacation and weren't there to babysit/roll back the deployment that broke everything?

It depends on how the resources are assigned, what projects they’re asked to focus on, and which strategic tech debt initiatives aren’t approved.

We’ve reached a point where I can no longer distinguish between people without experience and people repeating the talking points they’re told to repeat. That’s a major loss.

However, talent is a very small part of shipping a project. How that talent is resourced is far more important.

The Register is an opinionated tech tabloid filled with outrage bait. This article is not an exception, drawing far reaching conclusions from little evidence.

It's not your feeling about Amazon that would cast doubt on your take, it's that you've reduced it to one pet cause and decided a source is well written because it appeals to your dislike of RTO. Nowhere has any evidence of the relevance of that to this been presented, Amazon has had outages since before WFH even began, they've all always had their occasional outages and bad days.

As ex-amazonian, you should know that our average tenure for an SDE is just barely over 8 months. The old farts tool should have told you that during your day

From the article.

> At the end of 2023, Justin Garrison left AWS and roasted them on his way out the door. He stated that AWS had seen an increase in Large Scale Events (or LSEs), and predicted significant outages in 2024. It would seem that he discounted the power of inertia

Your comment is relying on that referenced inertia. Things will continue to function for a period of time, but there exists an inflection point at which they no longer function as previously.

I think this would make the squatting problem that we already have way worse. There would be bots buying every single remotely usable domain, and there would be no incentive for them to sell it unless they get an absurdly large offer.

I bought tombert.com in 2014 and forgot to renew it in 2015, and it was auctioned off by GoDaddy. For like six years, it was owned by squatters, and they wanted thousands of dollars for the domain [1]. I called offering the $100 for it, and they claimed that they can't go below $1400 because this domain is in "extremely high demand". I finally was able to buy it back in 2021, presumably because the squatter purged out domains that hadn't been purchased for N years and they wanted to save money.

Now, you could argue "see! You wouldn't have had to worry about it expiring if it were permanent on the blockchain", and that's true, but if someone else had gotten to that domain first, then I would also never get it. I think the only thing that keeps the internet even remotely fair in this regard is that domain names cost some amount of money to keep.

[1] https://web.archive.org/web/20160219161720/http://www.hugedo...

> People could host websites on the Blockchain from beyond the grave...

This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing (assuming they think about it at all)

Let’s solve rent-seeking by building a new system that’s even more rent-seeking!

At least currently death dissolves bonds.

> People could host websites on the Blockchain from beyond the grave...

This is precisely why something like this isn't a popular solution lots of people are working towards. Domains broadly speaking aren't a finite resource, but usable domains using common words definitely are. As time marches on human readable/typeable "permanent identifiers" are going to have to go away. Email address, usernames and the like are all going to get recycled, just like phone numbers are. Domains are currently recycled and most people probably think that's a good thing.

DNS is:

- simple

- battle hardened

- distributed

- affordable

blockchains are:

- essoteric, backwards, and not easily implemented

- new and unproven, frequently hacked

- effectively a ploy to centralize / redo Web 1.0 but owned by one blockchain

- ...waaaaaaay more about money and "owning something" than DNS is.

To bake an apple pie from scratch, you must first overthrow capitalism.

I am waiting for the author to lead the way.

Maybe its speculation, but I mean drawing conclusions becomes easy when 40% of devops staff being laid off by AWS was in the news three days ago.

https://blog.stackademic.com/aws-just-fired-40-of-its-devops...

Just because its speculation doesnt mean its worthless. But yeah it should be taken as speculation rather than a validated and tested hypithesis

I worked at AWS and still have friends who work there. I don’t know any L5s who wouldn’t jump at a chance to leave if they even got a slightly worse offer than what they are making now. I know a few L6s and L7s that would stick around out of momentum.

But I know very few people in the industry who know about Amazon’s reputation that have a life long dream of working there given a choice.

I was 46 when I was hired there for a “permanently remote [sic] field by design role” in ProServe and it was my 8th job out of college. I went in with my eyes wide open. I had a plan, stay for four years, sell my RSUs as soon as they vested, pay off debt, save some money, put it on my resume to open doors and make connections and leave.

I was never expecting to make more when I left. I used the time to downsize and reduce my expenses - including moving to state tax free Florida.

When I saw the writing on the wall, I played the game while I was on focus to get my next vest and wait for the “get 40k+ severance and leave immediately or try to work through the PIP”.

I took the latter and had three offers within 3 weeks. This was late 2023.