That is "a" solution.

Another solution that is very foreign to us in sweng, but is common practice in, say, aviation, is to have that fallback plan in a big thick book, and to have a light that says "Oh it's time to use the fallback plan", rather than require users to diagnose the issue and remember the fallback.

This was one of the key ideas in the design of critical systems*: Instead of automating the execution of a big branching plan, it is often preferable to automate just the detection of the next desirable state, then let the users execute the transition. This is because, if there is time, it allows all users to be fully cognizant of the inner state of the system and the reasons for that state, in case they need to take over.

The worst of both worlds is to automate yourself into a corner, gunk everything up, and then require the user to come in and do a back-breaking cleanup just to get to the point where they can diagnose this. My factorio experiences mirror this last case perfectly.

* "Joint Cognitive Systems" - Hollnagle&Woods

Which company deployed a chaos monkey deamon on their systems? Seemed to improve resiliency when I read about it.

>it shouldnt be

Software can (maybe) be perfect, or it can be relevant to a large user base. It cannot be both.

With an enormous budget and a strictly controlled scope (spacecraft) it may be possible to achieve defect-free software.

In most cases it is not. There are always finite resources, and almost always more ideas than it takes time to implement.

If you are trying to make money, is it worth chasing down issues that affect a miniscule fraction of users that take eng time which could be spent on architectural improvements, features, or bugs affecting more people?

If you are an open source or passion project, is it worth your contributors' limited hours, and will trying to insist people chase down everything drive your contributors away?

The reality in any sufficiently large project is that the bug database will only grow over time. If you leave open every old request and report at P3, users will grow just as disillusioned as if you were honest and closed them as "won't fix". Having thousands of open issues that will never be worked on pollutes the database and makes it harder to keep track of the issues which DO matter.

I don’t think it is exponential. In fact, one of the things that surprises me about software engineering is that it’s possible at all.

Bugs seem to scale log-linearly with code complexity. If it’s exponential you’re doing it wrong.

I learned navigation around 2005, and we used "nm" then. I honestly don't remember the moment when this abbreviation was granted to nanometer.

That's so adorable (for model railroads)

It says ‘metric’. Good.

"Hacker News failure caused by two units 12 orders of magnitude apart sharing 2-letter code"

I wouldn't have guessed until I read the comments. My assumption was somebody just mistyped km and somehow nobody cared to fix it.

Yeah, I went into the article thinking this because I expected someone had created waypoints right on top of each other and in the process also somehow generating the same code for them.

I was clarifying the post title, which uses "nm".

Yes. And, to quote the Wikipedia article: "Symbol: M, NM, or nmi". Not nm (as used in the title, but the article also uses it).

I could imagine conflict arising when switching between single and double precision causing inequality like this.

Sorry no. One interpretation, especially on this site, is that the problem was some kind of database bug, maybe where the same location was entered twice and a tiny location error ended up creating two locations.

I expect that out of any random sample 500 million literate and mentally healthy English speakers, more than 450 million of them are totally unaccustomed to thinking about nautical miles ever. Even people in science, or even who might have dealt with nanometers in school do not typically think about nautical miles unless they are sailors or airplane pilots.

It doesn't "imply" nanometers, it literally reads nanometers. The standard abbreviation for nautical miles is "NM."

Well, I certainly read nanometers, but I have to admit I may not be mentally well.

Well TIL I must be illiterate or mentally unwell.

I too first read it as nanometers. Perhaps it's just those with engineering/science backgrounds who have far more experience with nanometers than nautical miles?

I've never had cause to see the abbreviated form of nautical miles, but I know nanometers. Also, given purely the title I could see it being some kind of data collision due to precision errors between to locations that should be the same airport but perhaps two different sensors.

I assumed it probably wasn't nanometers but I have no idea what a nautical mile is.

Turns out 3600 NM = 6666 km

The question isn't really how many would think it meant nanonmeters, it's how many would recognise it as nautical miles.

Can’t claim to be “mentally well”, whatever that might mean, but I definitely read it as nanometers.

In aviation? You know how big an aircraft is right?

> obscuring this flight's waypoint problem have resulted in a potentially conflicting flight not being tracked among other flights?

Flights are tracked by radar and by transponder. The appropriate thing to do is just flag the flight with a discontinuity error but otherwise operate normally. This happens with other statuses like "radio failure" or "emergency aircraft."

It's not something you'd see on a commercial flight, but a private IFR flight (one with a flight plan), you can actually cancel your IFR plan mid flight and revert to VFR (visual flight rules) instead.

Some flights take off without an IFR clearance as a VFR flight, but once airborne, they call up ATC and request an IFR clearance already en route.

The system is vouchsafing where it does not need to.

Yes I agree. The reason the system crashed from what I understand wasn't because of the duplicate code, it was because it had the plane time travelling, which suggests very serious corruption.

> From the system's POV maybe this is the right way to resolve the problem. Could masking the failure by obscuring this flight's waypoint problem have resulted in a potentially conflicting flight not being tracked among other flights? If so, maybe it's truly urgent enough to bring down the system and force the humans to resolve the discrepancy.

Flagging the error is absolutely the right way to go. It should have rejected the flight plan, however. There could be issues if the flight was allowed to proceed and you now have an aircraft you didn't expect showing up.

Crashing is not the way to handle it.

It seems fundamentally unreasonable for the flight processing system to entirely shut itself down just because it detected that one flight plan had corrupt data. Some degree of robustness should be expected from this system IMO.

There is no need to shut down the whole system just because of one flight plan that the system was able to reject. Canceling (or forcing manual updates to) one flight plan is a lot better than canceling 1,500 flights.

Agreed. And most of the people with these attitudes have never written actual safety critical code where everything is written to a very detailed spec. Most likely the designers of the system thought of this edge case and required adding a runtime check and fatal assertion if it was ever encountered.

Spoken like a manager.

Look, when you're barking orders at the guys in the trenches who, understandably in fear for their jobs, do the stupid "business-smart" thing, then it is entirely the fault of management.

I can't tell you how many times just in the last year I've been blamed-by-proxy for doing something that was decreed upon me by some moron in a corner office. Everything is an emergency, everything needs to be done yesterday, everything is changing all the time because King Shit and his merry band of boot-licking middle managers decide it should be.

Software engineers, especially ones with significant experience, are almost surely more right than middle managers. "Shouldn't we consider this case?" is almost always met with some parable about "overengineering" and followed up by a healthy dose of "that's not AGILE". I have grown so tired of this and thanks to the massive crater in job mobility most of us just do as we are told.

It's the power imbalance. In this light, all blame should fall on the manager unless it can be explicitly shown to be developer problems. The addage "those who can, do, and those who can't, teach" applies equally to management.

When it's my f@#$U neck on the line and the only option to keep my job is do the stupid thing you can bet I'll do the stupid thing. Thank god there's no malpractice law in software.

Poor you - only one of our jobs is getting shipped overseas.

> C dev: "You are telling me that the three digit codes are not globally unique???

They are understood not to be. They are generally known to be regionally unique.

The "DVL" code is unique with-in FAA/Transport Canada control, and the "DVL" is unique with-in EASA space.

There are pre-defined three-letter codes:

* https://en.wikipedia.org/wiki/IATA_airport_code

And pre-defined four-letter codes:

* https://en.wikipedia.org/wiki/ICAO_airport_code

There are also five-letter names for major route points:

* https://data.icao.int/icads/Product/View/98

* https://ruk.ca/content/icao-icard-and-5lnc-how-those-5-lette...

If there are duplicates there is a resolution process:

* https://www.icao.int/WACAF/Documents/Meetings/2014/ICARD/ICA...

Did aviation software for 7 years. This is 100% the first assumption about waypoint / navaid when new devs come in.

And this is why you always use surrogate keys and not natural keys. No matter how much you convince yourself that your natural key is unique and will never change, if a human created the value then a human can change the value or create duplicates, and eventually will.

Or even more straightforward, just don’t believe anyone 100% knows what they are doing until they exhaustively list every assumption they are making.

or falsehoods programmers believe about global identifiers

I remembered this extensive article immediately (only that I've read it, not what and where to find). Thanks for saving me from endlessly searching it.

You can’t just reject or correct the flight plan, you’re a consumer of the data. The flight plan was valid, it was the interpretation applied by the UK system which was incorrect and led to the failure.

There are a bunch of ways FPRSA-R can already interpret data like this correctly, but there were a combination of 6 specific criteria that hadn’t been foreseen (e.g. the duplicate waypoints, the waypoints both being outside UK airspace, the exit from UK airspace being implicit on the plan as filed, etc).

Reject the plan surely should have come many places before shutdown the whole system!

Silently ignoring conditions where there are multiple or zero elements?

Race condition?

Too cheap... :o)

Or you configure your infrastructure to be remotely operated.

I'm not sure what you think is the insane failure mode?

The UK is part of the IFPS Zone, centrally managed by EUROCONTROL using AFTM. IFPS can accept/reject IFR flight plans, but the software at NATS can't. By the time NATS gets the flight plan, it has already been accepted. All their software can do is work out which parts enter the UK's airspace. If it's a long route, the plane has already taken off.

NATS aren't even thinking of a mixed-mode approach (for IFR flight plans) where they have both automated processing and manual processing of things the automated processing can't handle. They don't have a system or processes capable of that. And until this one flight, they'd never had a flight plan the automated system couldn't handle.

The failures here were:

1) a very unlikely edge case whose processing was specified, but wasn't implemented correctly, in the vendor's processing software

2) no test case for the unlikely edge case because it was really _that_ unlikely, all experts involved in designing the spec did not imagine this could happen

3) they had the same vendor's software on both primary and secondary systems, so failover failed too; a second implementation might have succeeded where the first failed, but no guarantees

4) they had a series of incident management failures that meant they failed to fix the broken system within 4 hours, meaning NATS had to switch to manual processing of flight plans

> That's a lot harder.

I wonder what 1,500 cancelled flights and 700,000 disrupted passengers adds up to in cost? And that’s just this one incident.

Similarly, no pilot in the Devil’s Lake region is using DVL to mean Deauville, and vice versa. :)

D'oh - the flight number was different and the lack of year on the post had me thinking it was just the same again.

My primary fear was the compliance issues re-coding the real s/w would drive this into the ground, where the compliance for a front fix might be lower. You could hash-map the collisions seen before and drive them to a "don't do that" and reduce the risk to new ones.

"we fall over far less often now"

Thank you for neg votes kind strangers. Remember ATC is rife with historical kludges, including using whiteout on the giant green screens to make the phosphor get ignored by the light gun. This is an industry addicted to backwards compatibility to the point that you can buy dongle adapters for dot-matrix printers at every gate, such that they don't have to replace the printer but can back-end the faster network into it.

C/F rebooting a 787 inside the maximum-days-without-a-reboot

What did they mean, if not 'nanometers'?

It's like doing the Kessel run in less than twelve parsecs.

I bet not everybody on here does, so picking the unambiguous unit sign would definitely avoid some double-takes.

Still, either "nmi" or "NM" would be better than the current and less correct "nm", even if "nm" is what's used in the article.

Unfortunately shutting down air traffic generally does not result in zero excess deaths: https://pmc.ncbi.nlm.nih.gov/articles/PMC3233376/

It's true, not saying they did a bad job here, just that even minor problems in your code can exacerbated into giant net effects without you even considering it.

[dead]