reminds me of https://www.cbc.ca/news/canada/saskatchewan/stranded-man-cut...

>A man stranded in the bush in northern Saskatchewan was rescued last week after chopping down four power poles — knocking out electricity to surrounding communities. [...]

>But he had an axe and he knew SaskPower would have to check the downed line, so he went to work.

Nothing like summoning the North American Fiber-Seeking Backhoe! https://imgur.com/a/DZNhdIG

I remember a story from an acquaintance working in construction. In one case they had to dug up right where a fiber optic ran, so they had cleared it with its operator and waited for their go-ahead. And waited. And waited, until the boss said something to the effect of 'start digging, we have a schedule to keep'...

I think it happened many times to electrical lines due to lost campers.

All these cable/internet service companies are incompetent as it gets.

I am paying, $1000, $1800 & $1900 for the same service at 3 different location (20 mins from each other).

Two locations, I also have old coax lines that are still active, but not paying for it.

When I bought two businesses, I learned that they were paying for a dedicated fiber but using coax service.

At one of the location, we had fiber, and paying for backup coax and wireless. But if you turn off fiber box, it wont fail over to either one.

I wouldn't surprise it was down for weeks and no one bothered about it.

Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.

Could this be sabotage?

A lot of people don't check their backups until they need to restore.

A lot of people are incompetent.

Besides what the others have said, speaking from a neteng perspective, sometimes backup lines (and the core infrastructure in general) are engineered in such a way that it can't easily be tested properly without taking other things down, or manually rolling a truck specifically to test it in isolation with extra equipment.

Not saying that's what is going on here, just that it's possible.

That's very uncharacteristic, especially given all those contingency requirements (backup policies, failovers, testing schedules etc.) imposed on corporations/companies in the wake of 9/11.

[dead]

They had two diverse paths, but apparently did not have a (reliable?) process for ensuring the backup path was functional during fail-over. I imagine there will be some soul-searching over that.

Nothing I've ever seen or experienced with FAA indicates a lack of care; the parsimonious explanation is almost always that people who care a great deal are working within complex systems that don't always have a consistent or externally legible set of priorities. Or more intuitively, the failures we see are the "acceptable" ones versus the unacceptable ones (like planes falling out of the sky).

I've told this story so many times I've worn the corners off it. BT was giving a presentation to MSN WAN OPS about the new datacenter buildout in London for us. They kept going on and on about physical security, man traps, and that our fiber left the building on each side and didn't get close to each other for so many km away.

BT guy ends that part of the presentation with "the IRA will really have to get shit together to take you off the net"

This was during the troubles. Same planet, different world.

I’m pretty sure that type of project is contracted to the private sector and isn’t caused by government technologists in any way…

There’s been “a new ATC system” rolling out for the last 25 years.

Over the weekend, I got linked to https://admiralcloudberg.medium.com/reaping-the-whirlwind-in..., which is an exhaustive look at the causes of the first airspace collision in the US in several decades. After reading that, my conclusion is that, if the software being rolled out is what I think it is, it would be a borderline impeachable offense to not first trying it out in the DC airspace.

The fundamental problem with DCA, and one of the two main causes of the crash [1], is that they are required by political pressure to operate at a higher operational tempo than they can safely operate at. DCA has essentially 1½ usable runaways--large planes can only use the larger runway, and given the capacity restrictions, airlines have been pushing to use fewer small planes at the airport. This shift in plane size means the effective safe slot capacity has gone down, but people still keep citing the same number as justification for safe numbers, and the politicization of the issue has shut down everyone who complained that the actual traffic just couldn't be safely handled.

If DCA's theoretical slot capacity (36 landings and takeoffs each per hour) is to be reached one just one runway, you have about 100 seconds to go from plane 1 touchdown; exit the runway, letting plane 2 on to take off; plane 2's wheel leaving the runway, clearing plane 3 touchdown. That's doable, but with essentially 0 margin for error. Alternating the runways used for each landing would give you closer to 30s of margin, but if only 10-20% of the planes can use the alternate runway, you can't divert enough planes to use the alternate runway.

Now DCA and the FAA aren't stupid enough to actually schedule 36 landing slots every hour, but the problem is that because of a few various factors, what was scheduled as, say, 30 slots for an hour (giving 120 seconds between landings, probably sufficient margin) ends up being 12 slots used in the first half-hour and 18 slots used in the second half-hour, which means a lot of the actual operation ends up having no safety margin even though on paper you have sufficient margin.

One of the ways you can rectify that is to assign slots further in advance so that you don't get the bunching. Effectively saying "oh, if you leave right now, you'll arrive at 3:30 with three other planes, but if I hold you for 10 minutes, I can push you into a less busy arrival time." Doing this requires good, accurate prediction of the actual flight travel times, and my understanding is that this is what the new software is meant to provide.

[1] The other main cause is essentially that the DoD's aviation practices in the area is a giant clusterfuck that endangers lives, and unfortunately that sentence is not relegated to the past tense.

That system wouldn't have replaced the one that went down. It's good to see investment in this area but that system is not going to fix much TBH.

I suspect it is vibe coded. Contract won in June. Deploying in September.

Good luck to all of us.

* Yes, TRACON have their own dedicated links. You can look into ASTERIX and STARS to learn some of the cursed ways the data processing and dataflow work.

* There is supposed to be a primary and a secondary link, in this case the primary failed and the fail-over also failed. It's unclear from the reporting if they were damaged in the same incident or if the failover was not tested or monitored adequately.

The Philadelphia TRACON site has been notoriously unreliable and was supposedly improved in 2025, it's also unclear if these issue actually could stem from that implementation.

Do you want air traffic control to be on the general internet?

That seems like an extremely foolhardy thing to do.

It is, but you need to set it up correctly.

Fiber was all installed long after we knew locating it was important. I suspect it all had a tracer wire installed with it. Of course sometimes the tracer wire can break.

Before fiber a lot of things were put into the ground with no thought how you would find it again.

Always bring a length of fiber optic cable when you're in the wilderness. If you get lost or stranded, bury the cable. Within a few hours, a backhoe will stop by to dig it up, and the crew will rescue you.

Local people know where they are because once upon a time a D8 drove through pulling a big orange spool of fiber conduit and passers by chatted about how another fiber was going in. Also around here the contractors seeded the disturbed ground with a wildflower mix (presumably to discourage weeds) so you can also look for straps of wildflowers.

And of course there are marker posts that say "buried fiber optic cable do not dig".

Put two for redundancy

Ah yes let’s make our entire infrastructure depend on a company controlled by a single person.

I can't tell if you're being downvoted because of the sarcasm, or because other users can't tell it's sarcasm.

Wasn't the UK ATC system the one where the whole thing core dumped, followed by the backup ATC system core dumping, when someone filed a flight plan with confusing same-name waypoints?

With low-enough engineering quality, you don't actually need saboteurs.

You'd hope Putin is not so stupid as to start committing acts of war on US territory. Helping the Iranians against the US seems to have gone mostly unnoticed but I'm pretty sure that if such an act was traced back to russia there would be a different tune played soon after.

[dead]

god, fk this