It isn't surprising at all. There's a reason why tech companies have insanely large engineering teams even though it feels to an outsider (and inept management) that nobody is doing anything. It takes a lot of manpower and hours to keep a complex system working and up to date. Who validates the backups? Who writes the wikis? Who trains new hires? Who staffs all the on-call rotations? Who organizes disaster recovery drills? Who runs red team exercises? After the company has had repeated layoffs and fired, outsourced or otherwise pushed out all this "overhead" eventually there's no one remaining who actually understands how the system works. One small outage later, this is exactly the situation you end up in.

Your comment suggests that you're not familiar with the diversity in M&S' operation.

Marks and Spencers started as a department store; they still have this operation. They sell clothes, beauty products, cookware, homeware and furniture. All these things are sold in physical shops and online. Most of this is straightforward for an e-commerce operation, but the furniture will involve separate warehousing and delivery systems.

They also offer financial services (bank accounts, credit cards and insurance). These are white labelled products, but they are closely linked to their loyalty programme (the Sparks card).

Finally, they have their food operation: M&S is also a high-end supermarket. You can't do your food shop on the M&S website (although their food products are available from online-only supermarket Ocado), but you can order some food products (sandwich platters and party food) and fresh flowers from the website.

So M&S is a mid-tier department store and a high-end supermarket. These are very different styles of retail operation: supermarkets require a lot of data processing to ensure the right things get to the right shops at the right time to ensure that food doesn't go to waste but also shoppers aren't annoyed by the unavailability of staples like bread and milk.

Finally, M&S is traditionally fairly strong in customer service; it's not exactly Harrod's or Fortnum and Mason's, but their bra-fitting service, for example, has a legendary reputation. The internet isn't their natural home.

So all-in-all, you have a business doing complicated things online because they think they have to, not because they want to: a pretty clear recipe for disaster.

How do you know it's safe to redeploy? If your entire operation may be compromised, how can you trust the code hasn't been modified, that some information the attackers have doesn't present a further threat, or that flaws that allowed the attack aren't still present in your services? It's a large company so likely has a mess of microservices and outsourced development where no-one really understands parts of it. Also, if they get compromised again it would be a PR disaster.

They're probably having to audit everything, invest a lot of effort in additional hardening, and re-architect things to try and minimise the impact of any future attack. And via some bureaucratic organisational structure/outsourcing contract.

HN posters love talking gangster shit when something goes offline but never walked a mile in their boots.

I most recently remember sifting through gloating that 4chan - a shoestring operation with basically no staff - was offline for a couple weeks after getting hacked.

I've worked at a shop that had DR procedures for EVERYTHING. The recovery time for non-critical infra was measured in months. There are only so many hands to go around, and stuff takes time to rebuild. And that's assuming you have procedures on file! Not to mention if there was a major compromise you need to perform forensics to make sure you kick the bad guys out and patch the hole so the same thing doesn't happen again a week after your magical recovery.

And if you don't know, you shut it down till it's deemed safe. How do you know the backups and failover sites aren't tainted? Nothing worse than running an e-commerce site processing customer payment card data when you know you're owned. That's a good way to get in deeper trouble.

I'm not that surprised, though 3-4 months does feel like a long time.

When I was at early Twilio (2011? 2012? ish), we would completely tear down our dev and staging environments every month (quarter? can't remember), and build them back up from scratch. That was everything, including databases (which would get restored from backup during the re-bring-up) and even the deployment infrastructure itself.

At that point we were still pretty small and didn't have a ton of services. Just bringing my product (Twilio Client) back up, plus some of the underlying voice services, took about 24 hours (spread across a few days). And the bits I handled were a) a small part of the whole, and b) some of the easier parts to bring up.

We stopped doing those teardowns sometime later in 2012, or perhaps 2013, because they started taking way too much time away from doing Actual Work. People can't get things done when the staging environment is down for more than a week. Over the following 10 years or so, Twilio's backend exploded in complexity, number of services, and the dependencies between those services.

I left Twilio in early 2022, and I wouldn't have been surprised if it would have taken several months to bring up Twilio (prod) from scratch at that point, though in their case it would be a situation where some products and features would be available earlier than others, so it's not really the same as an e-commerce site. And that was when I left; I'm sure complexity has increased further in the past 3 years.

Also consider that institutional knowledge matters too. I would guess that for all the services running at Twilio, the people who first brought up many (most?) of them are long gone. So I wouldn't be surprised if the people at M&S right now just have no idea how to bring up an e-commerce site like theirs from scratch, and have to learn as they go.

“If you have source code and release assets.” And a build process that works from a clean code base. And a deploy process that works on fresh servers.

All of which assumes you even know what services exist, which in any company of this age and size you probably don’t.

> with backups and failover sites

What a fun pair of assumptions!

The Co-Op (grocery store chain) was hacked around the same time in likely the same incident. It took three weeks for them to get food back on the shelves at my local store. I don’t understand how that’s even possible… what happened to all the meat and vegetables in the supply chain? They just stopped flowing? They rotted? Why couldn’t they use pen and paper? It’s unbelievable to me that a business would go three weeks without stocking inventory.

The British Library still aren't fully back up and running after their cyberattack in Oct 2023: https://www.bl.uk/cyber-incident/

So you haven’t dealt with ransomware gangs yet? Because they have gotten sophisticated enough to nuke source code repos and backups and replicated copies.

It’s part of the reason tape is literally never going to die for organizations with data that simply cannot be lost, regardless of rto.

For this particular audience, it's one of those things that could be rewritten in Rust over a weekend and then deployed on the cheap via Hetzner. At least then it'll be memory safe!

of course, if you redeployed everything from the source code, you could very well still have the same vulnerabilities that caused the problem in the first place..

There are no backups. There are no failovers. There is no git. There is no orchestration and deployment stratagies. Programmers ssh into the server and edit code there. Years and years of patchwork on top of patchwork with closely coupled code.

Such is a taste of what needs to be done if you wish to have a service that takes months to set back up after any disruption.

99% of "AI" talk in the public is for the sole purpose of making wall street happy to boost stock price and/or pump private valuations of AI startups. The reality on the ground is very different. CEOs are bragging about replacing senior software engineers with AI meanwhile their recruiters and hiring managers are desperately advertising $300-500K/yr jobs for these same engineers while still not being able to hire enough of them because of high demand.

Well we need to fix the business leadership problem asap. From the bio of the current M&S CEO. https://en.wikipedia.org/wiki/Stuart_Machin

>He resigned as managing director of Target in April 2016 because of accounting irregularities that he was unaware of but "happened on [his] watch".[4] He then became the chief executive of Steinhoff International.[4] (which seemed to have a lot of issues too https://en.wikipedia.org/wiki/Steinhoff_International#Debt_p...)

Foresight to mitigate potential major issues is exactly what CEOs are expected to do. I'm not sure how being unaware of major account irregularities is not seen as a career ending move here.

AI replacing CEOs seems straightforward as well. Accounting is such a data driven environment i think spotting account irregularities early would be straightforward. Likewise AI has the potential to think past short term thinking that leads to IT outsourcing (to the extent the store is not coming back online anytime soon!).

> How does one square those two realities?

People eat terrible food because they are bombarded with messages to do so. People can use terrible software for the same reasons. It doesn't matter that the food tastes worse than it used to–food companies are having record profits.

We just need one event of C*O of critical/big company bragging about firing engineering and replacing it with AI and then followed by huge cyberattack like that. Then see how AI balloon pops across news outlets.

AI replacing devs talk is about short term stock pumping and short term COGS reduction. The long tail is someone else's problem.

wait until they release AI for security and system orchestration

[dead]

With the case of M&S, and in many other cases in UK tech history that have gone poorly, it's mostly examples of the failure of hiring outside consultancies in India to do everything. Business executives continuously fall afoul of the fungibility myth. They believe that engineers are fungible, and that they should therefore simply pay for the cheapest engineers possible that meet the "requirements" on paper, usually set by someone who is not an engineer (HR, project manager, or a lower ranked middle-manager).

Time and time and time again we have seen major failures globally, and especially in the UK, that prove that there is no fungibility of engineers, and that outsourcing the critical technical infrastructure for your core systems and services is doomed to failure. They'd rather save a dollar today and lose ten million dollars tomorrow by damaging their national economy and sending more money to India. India's GDP is basically entirely propped up by tech services, and most of that is /failed service delivery/, hard to differentiate from frauds and scams at scale.

Interestingly the gov.uk website and everything around it is a prime example of software that just works. In terms of performance and accessibility. I work/volunteer for a non-profit design agency and we use the the uk.gov design system and I just love it: https://design-system.service.gov.uk/

The UK is wealthy enough to do it, if it wanted to. The thing is, the UK rewards grift way more than actual honest work.

So they saved £30 million a year and now losing estimated £40 a week (not to mention reputation and future opportunity loss due to lower customer confidence). They have almost wiped out all the savings.

If this is the reason, then this was a very bad deal for them.

M&S tried that, Amazon used to run the website:

https://www.theguardian.com/technology/2005/apr/19/business....

But they eventually took control back, so it clearly didn't work for them:

https://www.theguardian.com/business/2014/feb/18/marks-spenc...

M&S orders still use the same ###-#######-####### order number format as Amazon, so I'm not sure if it's still some sort of fork of whatever white-label Amazon technology they were using back then.

I'm not sure if getting Amazon to run your own ecomerce website is really the greatest idea in the long term (Amazon kinda want your customers to use Amazon, not your website), but M&S using them isn't as mad as that bit in the early 2000's where Waterstone's website was just a subsection of Amazon.co.uk.

This is the core thesis of a company like Shopify. Shopify will run everything else about being an e-commerce company (website, inventory, shipping, returns, ads, sales channels, etc) and then the merchant can focus on selling their product. But this is part of the larger thesis about running a business you hear in business school classes, to focus on your specialization and outsource your non-core expertise. Buy Workday/ADP/Paychex don't do payroll or HR. Don't build a data center, buy AWS/Azure/GCP. Don't build a sales database or marketing get Hubspot or Salesforce. Does your company take in a lot of mail? Outsource to a company that specializes in processing mail. Outsource your Technical Helpdesk. Outsource your customer support. This is why componentization is accelerating.

Err they were breached most likely through Tata Consultancy's helpdesk apparently which is literally the people they outsourced it to.

Their approach was to sell the UK operation to Tata in 2018 and piss everyone off until they leave and replace them with Indian staff to save costs over time.

You get what you pay for. They're now paying for it.

This incident has little to do with website or web store as such, and the only reason those are impacted is because pretty much all of M&S's IT systems have been impacted. Even if someone else would be running all of that, chances are that would still interface with the M&S computer systems to accurately get inventory information and the like.

They outsource as much as they can to the cheapest system integrators they can find, primarily TCS.

This take doesn't make sense. If one of your core businesses is selling clothes online, and you're a large enough entity, you should write your own software to sell clothes online.

Basically by your exact same logic you're asking Walmart and Target to outsource their websites, which is completely insane.

Bureaucracy is almost always the reason. They don't just need a website, they need -their- website back, because it was programmed with a million little business rules and pricing logic and regulatory requirements.

They already outsourced thier ecomerce offerings- to ocado - and that's up and running.

What remains is mostly logistics - this company runs farms and abertoires, food import and packaging and a network of warehouses and stores. The drop in cots product is SAP.

The whitelabel ecomerce site is kind of an insurance/ legacy thing while ocado is thier shopfront. Presumably they are prioritising what matters, the logistics, and might sunset this part of thier offer anyway.

You really think Shopify scales to a large department store?

The largest enterprise example of a Shopify customer on their marketing website has $500 million in sales.

M&S has an annual revenue of over £10 billion