Thanks for the summary for those of us who can't watch video right now.

There are so many layers of failures that it makes you wonder how many other operations on those ships are only working because those fallbacks, automatic switchovers, emergency supplies, and backup systems save the day. We only see the results when all of them fail and the failure happens to result in some external problem that means we all notice.

The problem is that there are a thousand merchant marine vessels operating right now that are all doing great - until the next loose wire. The problem is that nobody knows about that wire and it worked fine on the last trip. The other systems are all just as marginal as they were on the 'Dali' but that one shitty little wire is masking that.

Running a 'tight ship' is great when you have a budget to burn on excellent quality crew. But shipping is so incredibly cut-throat that the crew members make very little money, are effectively modern slaves and tend to carry responsibilities way above their pay grade. They did what they could, and more than that, and for their efforts they were rewarded with what effectively amounted to house arrest while the authorities did their thing. The NTSB of course will focus on the 'hard' causes. But you can see a lot of frustration shine through towards the owners who even in light of the preliminary findings had changed absolutely nothing on the rest of their fleet.

The recommendation to inspect the whole ship with an IR camera had me laughing out loud. We're talking about a couple of kilometers of poorly accessible duct work and cabinets. You can do that while in port, but while you're in port most systems are idle or near idle and so you won't ever find an issue like this until you are underway, when vibration goes up and power consumption shoots up compared to being in port.

There is no shipping company that is effectively going to do a sea trial after every minor repair, usually there is a technician from some supplier that boards the vessel (often while it is underway), makes some fix and then goes off-board again. Vessels that are not moving are money sinks so the goal is to keep turnaround time in port to an absolute minimum.

What should really amaze you is how few of these incidents there are. In spite of this being a regulated industry it is first and foremost an oversight failure, if the regulators would have more budget and more manpower there maybe would be a stronger drive to get things technically in good order (resist temptation: 'shipshape').

The fuel pump not automatically restarting on power loss may actually have been an intentional safety feature to prevent scenarios like pumping fuel into a fire in or around the generators. Still part of the Swiss cheese model, of course.

I have found that 99% of all network problems are bad wires.

I remember that the IT guys at my old company, used to immediately throw out every ethernet cable, and replace them with ones right out of the bag; first thing.

But these ships tend to be houses of cards. They are not taken care of properly, and run on a shoestring budget. Many of them look like floating wrecks.

Another case study to add to the maritime chapter of this timeless classic: https://www.amazon.com/Normal-Accidents-Living-High-Risk-Tec...

Like you said (and illustrated well in the book) it's never just 1 thing, these incidents happen when multiple systems interact and often reflect a the disinvestment in comprehensive safety schemes.

ive been in an environment like that.

"nuisance" issues like that are deferred bcz they are not really causing a problem, so maintenance spends time on problems with things that make money, rather than what some consider spit n polish on things that have no prior failures.

Just insane how much criminal negligence went on. Even boeing hardly comes close. What needs to change is obviously a major review of how ships are allowed to operate near bridges and other infrastructure. And far stricter safety standards like aircraft face.

Hopefully the lesson from this will be received by operators: it's way cheaper to invest in personnel, training, and maintenance than to let the shit hit the fan.

I watched Sal's video yesterday, great summary.

So much complexity, plenty of redundancy, but not enough adherence to important rules.

All you said is true - but these investigations are often used for the purpose of determining financial liability and often that comes down to figuring out that one, immediate, proximate thing that caused the accident.

A whole bunch of things might have gone wrong, but if only you hadn't done/not-done that one thing, we'd all be fine. So it's all your fault!

[dead]

> it's important to recognize times when you had a near-miss, and still fix those root causes as well.

I mentioned this principal to the traffic engineer when someone almost crashed into me because of a large sign that blocked their view. The engineer looked into it and said the sight lines were within spec, but just barely, so they weren't going to do anything about it. Technically the person who almost hit me could have pulled up to where they had a good view, and looked both ways as they were supposed to, but that is relying on one layer of the cheese to fix a hole in another, to use your analogy.

> Which is why the "retrospectives are useless" crowd spins me up so badly.

When I see complaints about retrospectives from software devs they're usually about agile or scrum retrospective meetings, which have evolved to be performative routines. They're done every sprint (or week, if you're unlucky) and even if nothing happens the whole team might have to sit for an hour and come up with things to say to fill the air.

In software, the analysis following a mishap is usually called a post-mortem. I haven't seen many complaints about those have no value. Those are usually highly appreciated. Thought some times the "blameless post-mortem" people take the term a little too literally and try to avoid exploring useful failures if they might cause uncomfortable conversations about individuals making mistakes or even dropping the ball.

this is essentially the gist of https://how.complexsystems.fail which has been circulating more with discussions of the recent AWS/Azure/Cloudflare outages.

> Swiss cheese model

I always thought that before the "Swiss cheese model" introduced in the 1990s that the term Swiss cheese was used to mean something that had oodles of security holes(flaws).

Perhaps I find the metaphor weird because pre-sliced cheese was introduced later in my life (processed slices were in my childhood, but not packets of pre-sliced cheese which is much more recent).

>Which is why the "retrospectives are useless" crowd spins me up so badly.

As Ops person, I've said that before when talking about software and it's mainly because most companies will refuse to listen to the lessons inside of them so why am I wasting time doing this?

To put it aviation terms, I'll write up something being like (Numbers made up) "Hey, V1 for Hornet loaded at 49000 pounds needs to be 160 knots so it needs 10000 feet for takeoff" Well, Sales team comes back and says NAS Norfolk is only 8700ft and customer demands 49000+ loads, we are not losing revenue so quiet Ops nerd!

Then 49000+ Hornet loses an engine, overruns the runway, the fireball I'd said would happen, happens and everyone is SHOCKED, SHOCKED I TELL YOU this is happening.

Except it's software and not aircraft and loss was just some money, maybe, so no one really cares.

> All the holes in the cheese line up...

I absolutely heard that in Hoover's voice.

Is there an equivalent to YouTube's Pilot Debrief or other similar channels but for ships?

https://www.youtube.com/@pilot-debrief

As I said elsewhere, the upshot is that you need to know which holes the bullet went through so you can fix them. Accidents like this happen when someone does not (care to) know the state of the system.

> Basically, the line of causation of the mishap has to pass through a metaphorical block of Swiss cheese, and a mishap only occurs if all the holes in the cheese line up.

The metaphor relies on you mixing and matching some different batches of presliced Swiss cheese. In a single block, the holes in the cheese are guaranteed to line up, because they are two-dimensional cross sections of three-dimensional gas bubbles. The odds of a hole in one slice of Swiss cheese lining up with another hole in the following slice are very similar to the odds of one step in a staircase being followed by another step.

Obviously this is the standard line any disaster prevention, and makes sense 99% of the time. But what's the standard line about where this whole protocols-to-catch-mistakes thing bottoms out? Obviously people executing the protocol can make mistakes, or fall victim to normalization of deviance. The same is true for the next level of safety protocol you layer on top of that. At some level, the only answer really is just "don't make mistakes", right? And you're mostly trying to make sure you can do that at a level where it's easier to not make mistakes, like simpler decisions not under time pressure.

Am I missing something? I feel like one of us is crazy when people are talking about improving process instead of assigning blame without addressing the base case.

It kind of is though. There's a lot less opportunity for failures at the limit and unforeseen scale. Mechanical things also mostly don't keel over or go haywire with no warning.

That was super helpful. I was assuming from skimming the text description that it was a failed crimp

A lot of people wildly under-crimp things, but marine vessels not only have nuanced wire requirements, but more stringent crimping requirements that the field at large frustratingly refuses to adhere to despite ABYC and other codes insisting on it

"Catastrophe requires multiple failures – single point failures are not enough. The array of defenses works. System operations are generally successful. Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident. Each of these small failures is necessary to cause catastrophe but only the combination is sufficient to permit failure. Put another way, there are many more failure opportunities than overt system accidents. Most initial failure trajectories are blocked by designed system safety components. Trajectories that reach the operational level are mostly blocked, usually by practitioners."

https://how.complexsystems.fail/#3

> In a well engineered control system, any single failure will not result in a loss of control over the system

That's true in this case, as well. There was a long cascade of failures including an automatic switchover that had been disabled and set to manual mode.

The headlines about a loose wire are the media's way of reducing it to an understandable headline.

Most cargo ships have a single main engine with plenty of backup-less failure points. They are sort of engineered so these failures can't happen suddenly but you can help yourself to a bunch of videos on how substandard fuel and parts shortages cause week-long poweroffs in a middle of the ocean.

The image brings to mind the Cisco ethernet boot infographic: https://www.cisco.com/c/en/us/support/docs/field-notices/636...

I guess this will still be bellow Therac-25 for CS and CE students, but above for EE, ME, and Civil Engineering.

There are.[1] Unfortunately they take longer to employ than the crew had time.

[1] As it happens I open with an anecdote about steering redundancy on ships in this post: https://www.gkogan.co/simple-systems/

Shipping is a low-margin business. That business structure does not incentivize paying for careful analysis of failure modes.

Seems to me the only effective and enforceable redundancy that can be easily be imposed by regulation would be mandatory tug boats.

No. Lots more : It's because they were abusing a non-redundant pump to supply fuel to the generators. Which then failed, which ....

From the report:

> The low-voltage bus powered the low-voltage switchboard, which supplied power to vessel lighting and other equipment, including steering gear pumps, the fuel oil flushing pump and the main engine cooling water pumps. We found that the loss of power to the low-voltage bus led to a loss of lighting and machinery (the initial underway blackout), including the main engine cooling water pump and the steering gear pumps, resulting in a loss of propulsion and steering.

...

> The second safety concern was the operation of the flushing pump as a service pump for supplying fuel to online diesel generators. The online diesel generators running before the initial underway blackout (diesel generators 3 and 4) depended on the vessel’s flushing pump for pressurized fuel to keep running. The flushing pump, which relied on the low-voltage switchboard for power, was a pump designed for flushing fuel out of fuel piping for maintenance purposes; however, the pump was being utilized as the pump to supply pressurized fuel to diesel generators 3 and 4\. Unlike the supply and booster pumps, which were designed for the purpose of supplying fuel to diesel generators, the flushing pump lacked redundancy. Essentially, there was no secondary pump to take over if the flushing pump turned off or failed. Furthermore, unlike the supply and booster pumps, the flushing pump was not designed to restart automatically after a loss of power. As a result, the flushing pump did not restart after the initial underway blackout and stopped supplying pressurized fuel to the diesel generators 3 and 4, thus causing the second underway blackout (lowvoltage and high-voltage).

The terminal blocks could also have been designed to aid visual inspection.

No, there was a larger failure: whoever designed the control system such that a single loose wire on a single terminal block (!) could take down the entire steering system for a 91,000 ton ship.

> Non redundant fuel pump that doesn't even restart on power failure

The crew weren't using the redundant fuel pumps. They were using the non-redundant fuel line flushing pump as a generator fuel pump, a task it was never designed for and which was not compliant.

That it doesn't restart on restoration of power is by design; you don't want to start flushing your fuel lines when the power returns because this could kill your generators and cause another blackout.

> Main engine shutting of (sic) when water pressure drops

Yeah, this is quite bad. There ought to be an override one can activate in an emergency in order to run the engines to the point of overheating, under the assumption that even destroying the engine will cause less catastrophic consequences than not having propulsion at the time.

> backup generator not even starting in time

There were 5 generators on board. Generators 1 through 4 are the main generators on the HV bus side, and the emergency backup generator is on the LV bus side.

When the incident occurred, the ship was being powered by generators 3 and 4, which were receiving their fuel via the non-redundant fuel line flushing pump. These generators powered the HV bus, which powered the LV bus via a transformer. The emergency backup generator was not running, so the LV bus was only receiving power from the HV bus via 1 transformer.

The incident tripped the circuit breaker for this transformer, disconnecting the HV bus from the LV bus, resulting in the first LV bus blackout. This resulted in main engine shutdown (coolant pump failure) and an automatic emergency backup generator startup.

There is an alternate (backup) set of circuit breakers and transformer that could have energised the LV bus, but the transformer switches were left in the manual position, so this failover did not happen automatically and immediately. There were no company procedures or regulations which required them to be left in the automatic position.

The LV bus also powered the fuel line flushing pump, so this pump failed. As a result, generators 3 and 4 started to fail (being supplied with fuel by a pump which was no longer operating). The electrical management system automatically commanded the start of generator 2 in response to the failing performance of generators 3 and 4.

Generator 1 and generator 2 were fed by the standard fuel pumps, which were available. One main generator is capable of powering the entire ship, so there was no need to start generator 1 as well; this would have just put more load on the HV bus (by having to run the fuel pump for generator 1 as well).

Instead of the automatic transformer failover (which was unavailable), the crew manually closed the same circuit breaker that had already tripped, 1 minute after the first LV bus blackout.

This restored power to the LV bus via the same transformer that was originally powering it, but did not restart the fuel line flushing pump supplying generators 3 and 4 (which were still running, but spinning down because they were being fed fuel via gravity only). This also restored full steering control, but this in itself was inadequate to control the vessel's course without the engine-driven propeller.

The main engine was still offline and takes upwards of half a minute to restart, assuming everyone is in place and ready to do so immediately, which was unlikely.

The emergency backup generator finally started 10 seconds later (25 seconds too late by requirements, 70 seconds after the first LV bus blackout).

Generator 2 had not yet gotten up to speed and connected to the HV bus before generators 3 and 4 disconnected (having exhausted the gravity-fed fuel in the line ahead of the inoperative fuel line flushing pump), resulting in an HV bus blackout and the second LV bus blackout. With only the emergency backup generator running on the LV side, only one-third of steering control was available, but again, this was inadequate without the engine.

3 seconds later, generator 2 connected to the HV bus. 26 seconds later, a crew member manually activated the alternate transformer, restoring power to the LV bus for the second time.

The collision was preventable:

- It is no longer a requirement that the engine automatically shuts down due to a loss of coolant pressure. It was at the time the vessel was constructed, but this was never re-evaluated. If it were, the system may have been tweaked to avoid losing the engine.

- If the transformer switches were left in the automatic position, the LV bus would have switched over to being powered by the second transformer automatically, and the engine coolant pumps and fuel line flushing pump would not have been lost.

- Leaving the emergency backup generator running (instead of in standby configuration) would have kept the LV bus energised after the first transformer tripped, and the engine coolant pumps and fuel line flushing pump would not have been lost.

- If the crew had opted to manually activate the second transformer within about half a minute (twice as fast as they reactivated the first one), and restarted the fuel line flushing pump, a second blackout would have been avoided, and the engine could have been restarted in time to steer away.

This shows the importance of leaving recovery systems armed and regularly training on power transfer procedures. It also illustrates why you shouldn't be running your main generators from a fuel pump which isn't designed for that task. This same pump setup was found on another ship they operated.

You're an optimist!

Yeah, when the word “allision” was right there!

Not really, because that's where that part of the investigation ends.

Pre-contact everything is about the ship and why it hit anything, post-contact everything is about the bridge and why it collapsed. The ship part of the investigation wouldn't look significantly different if the bridge had remained (mostly) intact, or if the ship had run aground inside the harbor instead.

Reminds me of "fetched up" describing what happened to the Exxon Valdez.

Thought the same, bridge is fallen on its entire length, sounds like a way to undersell it. Such an opportunity to pass on clickbait is interesting in this day and age.

Right? Like when I read that I thought we're talking a little paint-swapping.

No, we are not talking a little paint-swapping.

Clear plastic viewing windows on the spring terminals are the way to go. It allows for both instant feedback for the installer, and visual inspection or troubleshooting later by a third party.

The spring terminals should also be designed to have a secondary latch on this type on (what should be) rugged installation.

Finally, critical circuits should be designed to detect open connections, and act accordingly. A single hardware<->software design for this could be a module to apply across all such wiring inputs/outputs. This is simple and cheap enough to do these days.

A manual tug-test on the physical would be advisable when installing, to check the spring terminal has gripped the conductor when latched.

Does this comment apply to the current crop of American politicians? (Just curious.)

The WAGO connectors typically used in home wiring have a transparent plastic shell which lets you see whether the wire made it all the way through the spring clip. The ones shown in the NTSB video had an opaque shell around the spring clip.

I don't see anything in the report that suggests the connector failed. It sounds like the installer failed. Trust me, they can screw up twist connections too :)

The consensus seems to be skimming won’t occur. I’d encourage people to research the corruption of elected officials in the Baltimore area.