andix
4 days ago
> "Just as the FAA mandates manual flying time even when the autopilot is perfectly capable, SRE teams can designate certain incidents, even the ones the AI could handle, as human-practice opportunities"
This is a great example. Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes. While complexity increases and automation takes away more and more tasks.
Muromec
4 days ago
Aviation is a good example if you want to have zero fatal mistakes. There is a way and there is a cost to do so.
The same way erlang is good example of how to run a highly available system with hot reload and observability and stuff. There is cost for that too.
Maybe not all of the projects need the same properties. I'm fine if some of the stuff is having scheduled or unscheduled maintenance. Some kind of government portal that I use once a year had my permission to be offline between 0300 and 0500 and then again after 1200.
jeremyjh
4 days ago
The problem is that people go on Twitter and see the posts from "thought leaders" and "influencers" who say things like "reading the code is not needed". Those people are not working on systems that have paying customers who would cancel their subscription - or worse - if the software became too defective or unreliable.
You only need a handful of occasions of creating massive billing errors for your customer's customers before they will slag you to all their CTO friends and pull the plug. Imagine Stripe began double-charging every third transaction once a week, and couldn't fix it after two months of thrashing because no one there has any understanding of how anything works anymore. They are simply done at that point, all the value of their brand lost.
conartist6
3 days ago
The amount of trust AI has set on fire is incredible.
Business and people that I once though competent I trust no longer.
I'm watching leaders I thought had vision throw the future away for the shortest-term gains.
What have we become?
vkgcr
3 days ago
It's been one of the most eye-opening things for me as well. People who I considered serious and intelligent completely falling prey to "no need to read the code" AI psychosis.
Perhaps it reveals something deeper about about differing personality traits, about people who like programming as a discipline and like understanding code, vs people who view at as a means to an end.
nickstinemates
4 days ago
> Those people are not working on systems that have paying customers who would cancel their subscription - or worse - if the software became too defective or unreliable.
That's just not true.
Spartan-S63
4 days ago
I think the parent take is valid if you disqualify people managers/leaders rather than ICs. I think that's also valid to disqualify those takes because they aren't in the trenches every day. They may have points strategically for what's best for the company, but not what's best for the workforce or the industry.
jeremyjh
4 days ago
Name one. Point me to one developer that actually works - themselves, not just in the same company - on a system with serious customers that demand correctness and is spreading that message on Twitter.
nickstinemates
4 days ago
[flagged]
qlte
4 days ago
You're promoting your own company here, which as far as I can tell from the vibecoded website is Yet Another Agentic Platform of some sort. And prominently features a leaderboard with high scores for some reason.
It hardly meets the bar of a mission critical product like an ERP or payment processor layer where introducing random billing errors in customer data could be fatal to a B2B company's viability.
nickstinemates
4 days ago
Ok. Keep moving the goal posts.
I guess we are not serious engineers making a serious product for serious business. I'll make sure to let our customers know.
qlte
4 days ago
If you scroll up to the message you originally replied to, OP's goal posts from the start were companies like Stripe (which was specifically named).
Most businesses aren't in that category, that doesn't mean they aren't able to provide useful services and make money. But your company does not seem like it meets that bar.
Expanding the group to "any money making SaaS" would be moving the goalposts for the point OP was clearly making from the beginning.
Sorry for being a bit dismissive in my previous comment but it was fairly jarring clicking your link expecting a more responsive counter example to their argument, and then left a further bad taste after realizing from your bio that it was self-promotion.
disgruntledphd2
3 days ago
And just to pile on, payments is an incredibly competitive space, and operational excellence is non negotiable for someone like Stripe, as they are a higher priced provider.
icedchai
4 days ago
Your site does not give me “serious” vibes. Swamp club? More like slop club.
imtringued
3 days ago
His top customer is skunk-ape.
jeremyjh
4 days ago
Oh my that is convincing. Yes, I've seen lots of this sort of thing. I believe you, a Twitter influencer is behind it. AI building AI systems for AI to build AI systems with.
nickstinemates
4 days ago
Thanks!
charitablechari
3 days ago
Unable to understand sarcasm, AI truly allows a certain type of mouth breather to shine.
user
3 days ago
otterley
4 days ago
As if engineers, especially those writing software to handle financial transactions (which are usually in regulated businesses), don’t know how to write conformance tests with invariant cases.
jeremyjh
4 days ago
Who are the people who are responsibly writing all this test code but declare loudly on Twitter that no one needs to read the production code?
charcircuit
4 days ago
The test code is written and checked by AI without humans needing to be the ones looking at it.
majormajor
4 days ago
soooo wouldn't that be exactly the "deskilling" being discussed here, and isn't that deskilling concerning for money-handling firms?
having seen both (a) mission-critical human-built systems become nightmares for maintainability and extensibility and (b) bleeding-edge ai coding agents confidently and quickly fix a wrong problem in the face of an issue (or fix it in a slightly wrong way), I remain skeptical that this great de-skilling will all be perfectly fine.
of course, if you aren't looking at the code you won't fucking know until someone yells...
charcircuit
4 days ago
I disagree with several premises of the article. There is not going to be human led investigations of outages. People do not need to train on easy outages to be able to handle hard outages. Even if you wanted to train people on easy outages AI makes it easier than ever to setup up real world like issues to replace the need for real world experience. I also disagree in labeling it deskilling when it is a reallocation of skills to what is actually needed.
>concerning for money-handling firms
The real world is robust. It's okay for money handling code to have bugs. It's not the end of the world and AI will be able to have a lower bug rate than humans.
discreteevent
3 days ago
> The real world is robust. It's okay for money handling code to have bugs.
This is true. The world will go on. But your business will lose more money than it gained by not paying attention to the code. It will lose money because it will have to compensate customers or lose customers. That's how the world maintains it's robustness.
littlecranky67
3 days ago
Your business competes against other businesses. If hand written code is superior, that business will win.
frankzinger
4 days ago
I've seen some terrible AI-written tests. You have to check that code too. You have to check all the code.
user
4 days ago
user
4 days ago
keeganpoppen
4 days ago
I don't see what the giant social cost of AI hucksters is supposed to be... who cares if some random influencers claim you can vibe code your entire existence? the thing that bothers you is that the grifters are basically right, just for the wrong reasons (and a bit too early). Comments like this imply that there is some cavalcade of situations like this that have caused actual harm to actual people, but it is actually just an emotional response to people who are personally distasteful, and borne out of a desire to make sure that the "assholes aren't proven right". stuff like "Imagine Stripe began double-charging every third transaction once a week" is just so silly... what about AI makes this more likely in any way whatsoever? If anything, AI would have such a subtle scheme for robbing us blind that it would defy characterization into pithy quotation marks. this feels like a complaint about humanity dressed up as a complaint about AI. for all we know, AI could actually improve the situation-- it's not like humanity has set some unimpeachable precedent.
jeremyjh
4 days ago
AI can absolutely improve the situation. It has helped find and fix bugs I have despaired of ever fixing. But it cannot learn, and it cannot understand things for you. And people outsourcing all their understanding to it absolutely could lead to existential disaster for a system at the heart of the platform. Coding agents write too much code and ignore existing patterns. This has been studied quite seriously and they generally write 2.5x the amount of code a human does when refactoring or enhancing code in a real project. When a very large system slips past a critical point of decoherence you really cannot put any certain boundaries on what kind of problems become possible or how long it may take to fix them.
Yes, there are responsible engineers using AI in innovate ways and making things both more productive and better. But there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place.
Paradigma11
3 days ago
For practical purposes an agent is learning if you put information in its context. So you can create a comprehensive architectural analysis and put it into context and suddenly the agent is familiar with the project. You can also delete information from context, which is a rather hard and imprecise process with humans.
keeganpoppen
4 days ago
> it cannot learn, and it cannot understand things for you
the former is demonstrably untrue (and wildly so), and the latter is just an egocentric/anthropocentric "the genie in a bottle only answered my wishes, it didn't tell me what to wish for". buried in that statement is the presupposition that humans being able to understand that something is actually Important; short of religious/faith-based reasons I don't see the justification for the primacy of human understanding when it comes to the determination of Truth. if anything, being "understandable" to humans strikes me as an anti-goal. are we really to believe that human comprehension is the limiting factor keeping society from degrading into "decoherence"? how well is that going for us? the reason humanity places such a premium on identifying "The Truth" is precisely because of how illusory it is for our fallible minds. LLMs struggle with many things, but coherence is least among them. that is why "slop" works so well-- if it wasn't at least mildly coherent, it would be easy to discriminate.
> there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place and there are even more lazy humans who are happy to ascribe this to a shortcoming of the technology and not of humanity itself.
insofar as we are descending further into post-truth, it isn't because of AI slop blurring the lines between fiction and reality; it is because we are all collectively realizing that Truth isn't all it's cracked up to be. LLMs are forcing us to take a long hard look into the mirror and we don't necessarily like what we see.
jeremyjh
4 days ago
Please point me to the paper about the coding model that can continuously learn.
imtringued
3 days ago
I can choose a weak LLM to disprove your entire comment. Next time come up with something that can't be debunked in two seconds.
jasonvorhe
3 days ago
I mean, have you looked at Microsoft products since their inception? They still print money. Or Atlassian.
icantevenhold
3 days ago
Software quality was never as big of an issue as it’s made out to be.
Even before AI most software that ran “mission-critical” systems was kind of bad. I mean huge enterprises got routinely hacked bc of some owasp top 10 shit. And this happened many many times across all industries.
It’s not like before AI everyone was writing well-tested and reviewed secure super brain code
andix
4 days ago
Not losing production data and not getting breached is also something we try to make zero mistakes at.
toomuchtodo
4 days ago
Those of us who care. Those optimizing for economic returns or out of ideology are not. More carnage will be required for incentives to swing back I think.
"Security is for losers"
> Andy Yen, the CEO of Proton, had a message for an employee in the company’s security Slack channel who urged the company to at least take a few days to pentest newly written code before pushing it live. “That’s bullshit and loser thinking. Of course it’s possible to be fast and secure,” Yen wrote in the channel at the end of August. “Stop thinking always of how to be slower, and focus more on how to be faster.” In a follow-up message, Yen reiterated his point: “only losers think like that.” His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”
Proton CEO Andy Yen: "This will be the last year that we hire engineers." - https://news.ycombinator.com/item?id=50295112 - October 2026
Manifestos, Fear, and 'Loser Thinking': Inside Proton's AI Reckoning - https://news.ycombinator.com/item?id=50289692 - October 2026
Good luck.
Edit: There appears to be some consequences for acting like this and those actions being made public.
Proton's CEO Cancels AI 'Re-Interviews' - https://news.ycombinator.com/item?id=50307183 - October 2026
thereader12
4 days ago
That's the sort of thing you really don't want to hear from your encrypted mail company.
toomuchtodo
4 days ago
Vote with your feet or prepare for your eventual breach notification and data loss event. You control who you choose to do business with, and who custodians and processes your data, it is a choice. Make good choices.
lenerdenator
4 days ago
The problem is, ultimately, most companies have this mindset. Certainly the ones you've heard of do.
Charlie Munger once said, "Show me the incentives, and I'll show you the outcomes." There are very, very, very few actual incentives to operate a real infosec effort at most companies. Think about it: how many companies have you heard of going out-of-business because of breaches? If you're like most Americans, you've probably received at least one letter from some company you did business with telling you that their systems were compromised and your data may or may not have been stolen as a result. They'll usually close out the letter with an offer of free credit monitoring for some finite length of time. Since that's useless and a lawsuit is a pain in the ass with no guarantee of benefit, the vast majority of people simply sigh and toss the letter in the trash.
There's no real way to pin identity theft or other consequences on a given data breach, so it's hard to make any sort of claim that way.
At the end of the day, the company has saved money by not instituting best InfoSec practices and red-teaming their software on a routine basis. There are no civil or criminal penalties for them not doing so.
So they don't.
pjmlp
3 days ago
Which is why the recent movements for cybersecurity liabilities are so relevant, they are the incentives.
Now in some countries those with data security office role have more to worry about than only losing their job, if they sign off on something that isn't qualified to be out there in first place.
https://germancomplianceinstitute.com/blogs/news/personal-li...
mcv
4 days ago
I can imagine people are quickly looking for an alternative now. I wouldn't want to hear this from any company I'm a customer of.
chrisweekly
4 days ago
FWIW, Fastmail is awesome.
mcv
4 days ago
I've been doing a bit of research, and mailbox.org looks like a winner. It's in Europe and it supports IMAP. Those are pretty much my main requirements, I think.
encrypted_bird
4 days ago
Do they have a free tier?
tancop
4 days ago
Moving faster is good. But only if it means sending more agents (and humans, that part is important) to do the same amount of testing in less time. Dropping critical work is not that.
coldtea
4 days ago
> Moving faster is good.
Says who? For what purpose? To whose benefit?
I want my encrypted email company to move slow, deliberately, and very carefully.
I'd like most software in general to move slower. Some, I'd like to not move at all. I hate churn for churn's sake, or piling useless features just so you pretend you did something to justify your subscription pricing, both of which were already the norm before AI and are worse now.
To speak in Apple terms, we don't miss more and faster crap novelties. We miss the Snow Leopard releases.
jjav
3 days ago
> Moving faster is good.
No, not necessarily. Sometimes, yes. For anything important, the last thing I want is a company that's moving too fast.
An email company? The base protocol RFC was written in the 70s. It's as mature as computer technology gets. There is zero value in moving fast here, all the value is in building an absolutely rock solid and secure solution.
The very opposite of move fast and break things.
linksnapzz
4 days ago
Oblig. Simpsons:
Homer: “There’s three ways to do things. The right way, the wrong way, and the Max Power way!
Lisa: "Isn’t that also the wrong way?"
Homer: "Yeah, but... faster.”
devmor
4 days ago
Why is moving faster good? I don’t see much benefit to it, unless I am a shareholder.
disgruntledphd2
3 days ago
Even if you're a shareholder, the long term gains of an investment are much higher than the short term ones.
The trouble is that executives are bonused on much shorter time scales, and they run the company.
jjav
3 days ago
> That's the sort of thing you really don't want to hear from your encrypted mail company.
Tone deaf award of the decade surely goes to that guy.
If it were some social game app or something, I guess it wouldn't be as tragic, but for a company whose sole reason to exist is to provide security... wow. Goodbye any trust ever again.
jerf
4 days ago
"His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”"
Mmmmm... I think I may have to score this particular point for the CEO. Lovingly handcrafted code has been many things over the past 30 years, but "secure" is not one of the adjectives I'd apply to it. Anyone who thinks that the lovingly handcrafted code has always just been wonderfully secure needs to check the tint on their glasses.
A proper AI setup probably can produce more secure code more quickly than conventional development processes... however one of the keys will be to understand that yes, that will still be slower than writing code that is not rather carefully analyzed for security and a company like Proton had better have a strong commitment to never, ever skipping that step because it's slowing them down or it got expensive. "Proper" in "proper AI setup" is the most important word.
But that's not to say that I'm endorsing everything else he said. Even if it is true that companies will be able to stop hiring engineers, which at the moment I seriously doubt [1], a company like Proton should expect to be one of the last companies who can do that, not be getting publicly excited about it first. Hey we dumped all our engineers because our managers can just prompt our AIs just as well here in October 2026 would definitely be a "instantly cancel any Proton services I had" signal for me.
[1]: One of the things I don't expect to change is that there are plenty of people I would describe my actual day-to-day work to and they would consider it a living hell if they had to do it. I still quite enjoy it overall. And there are plenty of jobs that other people love and I would consider a living hell. Even in 25 years that's going to be the case. There will be people willing to put up with the process of working with the system to create this stuff... but the people who are willing to be professional about this, do the QA, do the work beyond the first fun prompts, is going to be a minority, and an even smaller minority who take the time to get good at it. There will always be people who are better and people who are worse.
Yoric
4 days ago
While I agree that handcrafting code has never been any kind of guarantee of secure code, I'd argue that it remains a condition: if all you have is a set of specs (or even worse, specs written by AI and not read carefully by anyone) and no mean to ensure that the implementation actually matches the spec, you essentially have no understanding of what the code is doing, which is security suicide.
Of course, AI reviewers and security tools will catch some security errors, but this should come as an addition to a proper security design, not instead of it.
throwaway270925
4 days ago
God am I happy to not have switched to Proton right now!
boredatoms
4 days ago
Wow this makes me hesitate to renew with them
imtringued
3 days ago
I am wondering if these managers and CEOs even understand what their company's product is and that this lack of understanding is what causes them to become obsessed with things only a manager does. Which is ironic in the face of the accusations that developers are against AI because they just want to keep their jobs. It's the managers who want to live out their management fantasy.
crnkofe
4 days ago
Just learned about his position on security today. Been a Protonmail user for a while now. Initially I switched to detach myself from the US corp monopolies. Its just a matter of time now before I shutdown my Protonmail account. CEOs should understand that company ethics and security are not just black pixels on a white background.
Sinidir
4 days ago
Wow. I'm literally reconsidering having any mails at proton anymore. This is beyond abysmal attitude.
fcarraldo
4 days ago
The irony here with Yen (and to be clear, fuck him) is that Proton has been _slow as molasses_ since the beginning. It took them literal years to launch a calendar. A calendar!
Either he's been like this the entire time, trying to crack the whip and failing, and thus a bad leader who has repeatedly failed to get results, or he's (as his employees have said) very deep in AI Psychosis and completely abandoned his principles.
It doesn't matter which it is, they're both extremely damning for the company and his leadership.
coldtea
4 days ago
>The irony here with Yen (and to be clear, fuck him) is that Proton has been _slow as molasses_ since the beginning. It took them literal years to launch a calendar. A calendar!
As if that's a problem? I want them to be my fucking encrypted email. Not a calendar. Not a backup. Not a VPN. And they've already stretched to all kinds of shit.
fcarraldo
4 days ago
They are that. Is something fundamental missing from Protonmail?
My point is they announced they’d ship a calendar in 2019 and it took 2 years to ship a full release and 3 to ship an iOS app.
It also sort of sucks, and in 2024 (year 5) they announced they’re rebuilding it. The rebuild partially shipped in 2026.
For a guy suddenly obsessed with speed these timelines are even more absurd.
Muromec
4 days ago
I would rather have protonmail without all the stuff they are adding there
MDTHLN
4 days ago
What is it with weirdo tech executives and calling people losers? First DHH, now this guy.
toomuchtodo
4 days ago
Weak man's idea of a strong man. Strong leaders protect and support those who work for them, and accept legitimate criticism and being challenged on their ideas, but I digress.
"Any man who must say 'I am the king' is no true king." The measure is not who you can bully and denigrate successfully, but who will follow you to hell.
toomuchtodo
2 days ago
Related:
Patrick Stewart taught us what good men do with power - https://news.ycombinator.com/item?id=50327650 - November 2026
ishouldnotbutk8
4 days ago
>Strong leaders protect and support those who work for them, and accept legitimate criticism and being challenged on their ideas
And whose idea of a strong man is this? I don't think many people would follow Putin to hell willingly, or that he accepts any criticism, or accept being challenged on his ideas. But he could be called a "strong man", at least the kind of "strong man" some people aspire to be.
user
4 days ago
jerf
4 days ago
According to my friendly neighborhood AI there is no commonly accepted term for this effect, but when you have a community of people that is heavily selected for a particular characteristic you will end up with people that generally are outliers in some other significant ways too. In this case, caring deeply about email security is weird... unfortunately... and you can expect further weirdness from such people.
You can also read about people talking about this in the super-early computer user communities. When you had to be a bit crazy to even be interested in computers you ended up with people who were outliers in a lot of other ways too. As more people came in to the space it would regress to the mean... this is one of the things that "regression to the mean" can be regressing from.
vkou
4 days ago
It's not weirdness, it's just the combination of being an asshole and nobody in his life being able to tell him 'No.'
1659447091
3 days ago
> According to my friendly neighborhood AI there is no commonly accepted term for this effect, but when you have a community of people
Social contagion [0] (not to be confused for memetics [1]) involves behaviour, emotions, or conditions spreading spontaneously through a group or network. [...] Throughout history all social contagions have had a single thing in common: the people who follow them genuinely believe their group is only doing positive effects, and cannot rationalize negative effects from within their group.
Tech executives are a group (and a network), its not some "regression to the mean" its common human social behaviour within group dynamics. It's not just those two there has been a progressive slide of executives joining the 'tech-bro' subgroup.
linksnapzz
4 days ago
Anybody who is as responsible for popularizing Ruby on Rails & YAML as DHH has, has no standing to call anyone else a loser.
mort96
4 days ago
I mean if a company doesn't care about protecting its users' data then any conversation about improving security is gonna be inapplicable to them.
I'd also suggest that companies being so blasé is a direct result of lacking regulation. If there was no laws about aviation safety I bet we'd have more airlines with lax safety standards and more accidents.
imtringued
3 days ago
This is something I noticed as well.
Because humans were, loosely speaking, careful about not making egregious security mistakes, most security vulnerabilities turned out to be either harmless or the result of a programming language allowing the type of exploit to begin with.
But now we have code from non-people ordered by people who both don't care about security at all.
That means you will get maximally bad security outcomes which in turn will make governments consider regulating the software development industry. They are speed running their own demise.
QuiEgo
4 days ago
Eww. I have now lost all interest in their products.
red_candle_wax
4 days ago
> "Security is for losers"
Lol, brave words in an age of $6B heists and major CVEs every week.
tybit
4 days ago
There’s a limit to how much society can spend on aircraft safety, but it’s quite a lot because it’s so important. Cyber Security teams have to work with large constraints because spending an ever increasing amount of money directly, or losing money indirectly (like not allowing product teams to move quickly) is explicitly not a goal of most companies. So while we try to make zero mistakes, most companies are willing to take a non zero amount of extra risk of these things happening in exchange for other benefits.
andix
4 days ago
Aviation is working with the same constraints. Development of new systems is very slow, that's true. But day-to-day operation is extremely fast paced, and still impressively safe. They are able to change an engine of an aircraft within hours, with a very high probability of not making any relevant mistakes in the process.
simonra
4 days ago
Sure, but at what cost? How quickly improvements are delivered is one thing, but the materials and chosen process also come at a premium. And while it probably is worth it in aviation due to life and health being highly irreplaceable, the value of for instance my chat history or photos is certainly finite and quite low. Should they be corrupted, leaked, or worst case gone, it would somewhat suck. But at the same time I've effectively concluded that safeguards are worth about 0.4% of my monthly take home after tax, and when the yearly bill comes due I'm each time increasingly questioning whether I actually do value them even that highly.
Brian_K_White
4 days ago
"Sure, but at what cost?"
At the appropriate and necessary cost.
A less costly new engine design or less costly engine swap procedure that fails is worse than no airplane at all. If it's cheaper and fails, we don't want it. It wasn't cheaper, it was even worse than expensive.
It's not hyperbolic all or nothing. Sure infinite cost is just as useless as infinite failure, but there is no reason to talk about infinite.
At what cost? Some. At some cost.
NegativeK
4 days ago
> At what cost? Some. At some cost.
This doesn't address how the cost of safety related procedures versus the cost of risks mitigated is calculated (which is way harder than saying "some") so that the buyer can actually know what "some" cost is.
It's also not addressing the parent comment's point: the bad outcomes in aviation tend to be very impactful (both in human safety and financially), so the tolerance for mistakes is way lower.
Compared to cybersecurity for a brochureware website, where it takes a far larger stretch to go from "security incident" to "someone's dead". (Also, people in cybersecurity aren't going to accept "it has some cost, and you might buy it if it's not too expensive".)
reverius42
4 days ago
For the commercial flight industry this is very easy to calculate: zero deaths are acceptable. This is not like car manufacturer recall calculations (famously, referenced in Fight Club: if the lawsuit costs the manufacturer less than the recall, they don't recall). The airline industry is built different thanks to heavy regulation.
You could ask: is it _worth_ making it that safe? But we don't ask that. We just make it that safe.
I think there is a good point to be made that this level of safety (it costs what it costs) is probably not worth it for cybersecurity, in general, even if it's largely considered worth it for commercial aviation.
NegativeK
2 days ago
Zero acceptable deaths is equivalent to zero acceptable breaches -- the only solution is to offline the system entirely. You simply can not get rid of all risk or even know if you've gotten rid of all risk.
https://assets.performance.gov/APG/files/2023/june/FY2023_Ju... states on page 5 that they're aiming to cut fatalities in half from 8.9 per 100M passengers in FY 2009 to 4.4 fatalities per 100M passengers in FY 2025. (It looks like they were doing way better than that goal.) "As low as reasonably practical" (ALARP) seems to be the term for aviation, which is very clearly not a zero fatalities goal.
> You could ask: is it _worth_ making it that safe? But we don't ask that. We just make it that safe.
They... Don't? Aviation accidents continue to happen and they will continue to happen. They've decided that a positive number of fatalities is worth it not because they want anyone to die, but because they know that perfection is impossible. You can't achieve it, and you can't know that you've achieved it.
disgruntledphd2
3 days ago
As software intermediates more and more of the physical world, we're gonna end up at least as regulated as finance, and potentially as much as aviation.
The only real question is how much damage will be caused before that happens.
simonra
a day ago
I think it's highly doubtful we're going to see software generally regulated as finance, which like aviation can facilitate damage to individuals on a scale they'll never be able to recover from by themselves (in terms of lifetime needed to (re)acquire the resources). Most of the world is quite insignificant and unimpactful, and already regulated as tightly as it is based on that. Software being a larger part of the services provided or goods transacted is not suddenly going to inspire tighter regulation of the quality of services and goods offered beyond what they already are, like for instance some minimums around certain things not catching fire and killing people, or some services being rendered in a mostly food safe way. If humanity desired to go beyond lazy minimums we'd see universal effective elimination of planned obsolescence, not just best cases of minimum warranty periods measured in a handful of years, and then being fine with products mysteriously failing within days of the sellers support obligations ending.
disgruntledphd2
9 hours ago
> I think it's highly doubtful we're going to see software generally regulated as finance,
Probably not all of software, but basically all of our financial infrastructure, operational infrastructure and communications infrastructure is either already critically dependent on software, or heading that way.
I think that the EU's DORA act is probably the way things will head. We'll regulate the outcomes and ensure that these services can be maintained.
Honestly, the only reason that this hasn't already happened is that the US government is wildly against basically all regulation of software. I figure that if we end up in some kind of cybersecurity crash (likely, given the current trajectory) that we'll get lots and lots of regulation.
And to be honest, this will be for the better in the long term, but it'll probably suck for all of us in the short term (but at least there'll be jobs).
Brian_K_White
3 days ago
We also need some way for a regular person to recognize cheap vs quality software on sight the way they can for a frying pan.
It's fine for cheap pans to exist because any user of one can see what it is and make an informed choice to use the junk one or the good one.
justincormack
4 days ago
You can also reduce blast radius from breaches, and make systems more robust, rather than just trying to make zero mistakes. How bad breaches are and how well you can recover is also part of the outcome, just framing it as zero breaches or nothing is not that helpful, even though reducing breaches is important.
jeremyjh
4 days ago
This isn't a different thing - it is just one of techniques used, and it still requires humans to understand the systems they are building. Teams that outsource their systems understanding to AI will fail, no matter how they frame it.
andix
4 days ago
Isn't that the same thing as "not making mistakes"? If you can reduce the blast radius and didn't, wasn't that a major mistake in the first place?
charlieyu1
4 days ago
A lot of code is certainly not written careful enough. Plenty examples of plaintext passwords storage in the wild.
phil21
4 days ago
No, it's recognizing mistakes will be made and no one is perfect. You will also make mistakes regarding blast radius - sometimes the blast doors don't close or function like you thought they would during an actual explosion.
Defense in depth is the idea. Swiss cheese model of failure.
disgruntledphd2
3 days ago
Yeah, but this is bottlenecked by both your ability to observe and understand the system, which is precisely the thing that's being reduced by careless use of AI tools.
znpy
4 days ago
But why stop there? Aim higher!! End of all world’s wars, no more hunger anywhere anymore, free abundant clean electric power for everyone everywhere!
Jokes aside, you can’t get anywhere near zero fault unless to impose strict operating regulations , requirements and procedures and regular checks… just like in aviation.
And even there, boeing manages to fuck it up somehow.
nimih
4 days ago
Maybe it's different in your neck of the woods, but all the government portals I use tend to have access to rather sensitive PII, and I would really prefer their ops teams be up to snuff as a result.
hengheng
4 days ago
A better metaphor would perhaps be machining. You will not find a shop that has no CNC anywhere, but the machinists who can also use manual machines will always have an edge for certain things. Not that they're working manually much in their day to day, and not as all as a failsafe, but you can just tell who has that kind of background.
(And you can also tell whose mind is stuck in the old world, too.)
mikepurvis
4 days ago
It's first-principles thinking. You have to have some sense of how you would do it in order to reasonably engage with a machine's idea of how to do it. Otherwise you're just nodding along, like yeah sure bro uhhhh lgtm.
sobellian
4 days ago
Aviation is an example of how there's no way to have zero mistakes, or even zero fatal mistakes.
SoftTalker
4 days ago
People always find new ways to make mistakes. The approach in aviation is to investigate, identify root causes, and do what is possible to prevent the same mistake happening again.
There is no guarantee we'll ever reach perfection.
andix
4 days ago
It's more about managing mistakes. Pilots and other people in aviation still make a lot of mistakes all the time. But aviation figured out how to deal with that fact. Design procedures in a way, that mistakes self-correct most of the time. Add sanity checks exactly where they matter, and many other strategies.
Retric
4 days ago
The lesson from Aviation is how to constantly have mistakes without failure.
Sometimes they close down airports, that’s risk management not complete risk avoidance.
yallpendantools
4 days ago
No coincidence it is the textbook example of a complex system.
geraneum
4 days ago
> I'm fine if some of the stuff is having scheduled or unscheduled maintenance.
You might be, and your customers may not! And your competitor could be more than happy to offer a more reliable service.
coldtea
4 days ago
Then we all get shit service based on interrupted operation without due maintenance and more stress.
brightball
4 days ago
I’m using Elixir for every (web) project now specifically because the cost is negligible. I can't think of a reason to use anything else and the agents are _really_ good at writing it, particularly given the excellent structural guardrails that are in place.
Being able to ensure all of the test run freakishly fast async is a huge win too.
Muromec
4 days ago
I don't get elixir, maybe I'm not ruby brained enough. Too much syntax sugar for me
brightball
4 days ago
Elixir is great from a syntax perspective. It gets compared to ruby a lot for some reason but they're nothing alike. Elixir syntax is very deliberate about everything.
The BEAM runtime is the real win. Your options are Erlang, Elixir and Gleam to get the BEAM.
Out of the 3, I prefer Elixir and Jose is a brilliant leader for it.
bpbp-mango
4 days ago
How often do you need the HA features BEAM can give you in a webapp though?
brightball
3 days ago
It’s not the HA features that are the selling point for me.
The combination of lightweight processes with their own HEAP and the preemptive scheduler means that millions of processes can run on the same machine with consistent execution time, no GC pause, and no worry that a single resource heavy process is going to slow down everything else.
The balance tradeoffs allows you to do a lot of things that would otherwise be poor architectural choices.
LiveView is just one example, where you’re able to completely forgo the need for a frontend framework but get all of the same benefits with lightning fast execution times.
There’s a lot of wins that come from it.
From a AI standpoint, functional code is simpler to trace and maintain because the AI doesn’t have to worry about side effects.
The HA perks are a bonus.
runjake
4 days ago
The OP didn't present an absolute. They said "Aviation is a perfect industry to learn from" and I 100% agree with them.
jjav
3 days ago
> Maybe not all of the projects need the same properties.
Definitely not all projects need the same level of quality.
But per the thread, if you allow the skills of all the engineers in your company to rot, what happens when there is a system that needs the higher level of quality?
nrmitchi
4 days ago
A (very) key difference is that this behaviour is mandated by the FAA, and consistent across all providers. There isn't a (legal) option of just not-doing-it because it's "cheaper".
Modern companies generally do not have the same incentive to keep their SRE (or engineers in general) upskilled. Cutting SRE because "we don't really have incidents, why do we keep paying them? If we need to do layoffs I would rather keep a product engineer" will always* (okay, _generally_) be the direction that organizations go without external influence or requirements.
Now I am sure there are some companies or industries where this is required (for example, I would not be surprised if certain military software/services contracts had requirements for support staff with X hours or training/etc), but that is not consistent everywhere.
If we want something that is "money-losing" in the short/medium term to be consistently applied, it unfortunately needs to be (uniformly) regulated.
ygouzerh
a day ago
+1. The issue is not that us SRE doesn't want to upskill, on the contrary. It's mostly that we doesn't have time, between the backlog always full during the day, and the incident at night. We are considered cost center, and companies try to reduce it a maximum by over-optimizing headcount/tasks
andix
4 days ago
You're right, there is no mandate for learning and doing better in SW engineering. Many companies just don't care, also kind of fine. This doesn't mean that everyone doesn't care and nobody wants to improve.
GlobalChubby
4 days ago
This was just the default where I worked. We called the exercise Wheel of Misfortune: facilitator sets up an outage / bug in a non-prod environment (inspired by real events or not), spins a wheel, picks a victim, and the problem is solved collectively, manually. That's the standard format, but it could be much more freeform than that: the point is to educate each other regarding "incident management."
I will say, I'd always been skeptical that AI could be better at incident management than an experienced engineer with deep domain / institutional knowledge, but I was recently proven wrong: Claude root-caused and helped resolve a Postgres lock queue pileup that I believe would have taken me much longer to fix.
Re: aviation being a perfect industry to learn from, I think https://m.youtube.com/watch?v=5ESJH1NLMLs might be a good watch. I was introduced to this at a Wheel of Misfortune around 5 years ago, so it's interesting watching it today when AI is encroaching upon everything.
leecoursey
4 days ago
Just like pilots, SREs will have to spend a minimum number of annual hours in the simulator to keep their skills sharp.
oblio
4 days ago
SREs will be de-materialized and converted to a Docker image that will be run on Kubernetes, in the prod env.
sunrunner
4 days ago
Exactly as foretold: https://www.youtube.com/watch?v=0otdTJa9IUc.
pmarreck
4 days ago
It's not a great example.
The plane lands in the same amount of time whether an autopilot lands it or a human, so there's no incidental cost.
If you let a human SRE handle a problem completely, not only will it likely take longer than an AI, but since that SRE is now deskilled, it may take FAR longer. And that's a cost. Who's going to pay it?
ludamad
3 days ago
Is that true? Wouldn't a pilot mistake in theory cause a plane to take longer?
renehsz
3 days ago
I'm not a pilot, so take everything I say with a grain of salt.
Yes, a mistake can lead to a go-around, which at a busy airport can easily add an extra 10 to 20 minutes of flight time. But as long as no such mistake is made, a hand-flown visual approach can actually be quite a lot faster.
But first, we have to clarify the terminology. On most instrument approaches, the autopilot only flies the beginning of the approach, until the pilot flying decides to disconnect and lands the plane manually (somewhere between the final approach fix and minimums, depends on visibility and crew preference). So the most common way of flying an approach is neither fully automated nor fully manual.
Autolands are only done rarely, like in low visibility or periodically for pilots and aircraft to stay current. Visual approaches, on the other hand, are more common (though it depends on the region, controllers in the US hand out more visual approach clearances than controllers in Europe for instance). But even on a visual approach, crews often keep the autopilot engaged until very late in the approach or at least use the ILS as an aid while flying manually. That makes it tricky to do a comparison.
All external conditions being equal, I'd assume that an autoland is indeed less likely to lead to a go-around than a hand-flown visual approach. This is mainly because autolands, like other instrument approaches, are typically set up with a long final and earlier stable gate (typically 1000ft). A lot of go-arounds happen due to being unstable at the gate (too high, too fast, not configured, etc.) - that's less likely with an autoland which forces you to be stable much further out. Additionally, an autoland has a very low decision height leading to fewer go-arounds in marginal weather (though in such cases a visual approach most likely wouldn't be attempted in the first place). On the other hand, an autoland has stricter crosswind and maybe also tailwind limits, could be subject to ILS signal interference, and usually requires two or three autopilots to be fully functional. Still, the first argument has more weight to it I think.
But assuming no go-around, a hand-flown visual approach can be quite a lot faster than an autoland! As I already mentioned, autolands need a long final to establish early to give the airplane time to stabilize (often crews are looking to establish well prior to the FAF). A visual approach, on the other hand, allows tight turns onto final, slowing down and configuring the airplane much later, etc. The aircraft needs to be stable at the gate, but when flying visually that's typically 500ft AGL, so short final basically. Visual approaches also give controllers more freedom to give vectors with fewer track miles, and visual separation allows for reduced spacing.
Overall, I think that hand-flown visual approaches often save time compared to autolands, and possibly over other instrument approaches as well. And even if the probability of a go-around is slightly higher (assuming equal weather conditions), a go-around is still so rare that despite being a significant outlier in flight time, I wouldn't think it moves the average up much.
But maybe the comparison just doesn't make much sense to begin with. An airplane is flown by pilots and computer systems together. Pilots can choose between different levels of automation, but both extremes of the automation spectrum (on the one hand: fully hand-flown visual approaches without even the ILS frequency or any navaids tuned in; on the other hand: full CAT III autoland) are quite rare for airliners. Most approaches fall somewhere in the middle on that automation spectrum. Is that the safest and quickest way to do it? I don't know. But I assume the industry has converged on it for good reasons.
lenerdenator
4 days ago
Aviation is a perfect industry to learn from if you are willing to hold IT to the same standard of regulatory scrutiny. You can bet your entire net worth that anyone running anything in the industry would fight tooth and nail to prevent that sort of regime from coming into force.
digitallogic
4 days ago
The thing is, without the FAA mandating something like this for SRE’s, no one will do it. In the moment, getting the broken thing back up and running is seen as far more important than the learning opportunity.
I noticed this constraint a few years ago when the SRE/DevOps hiring market was really tight, but no one was investing in non-senior talent. I ended up founding a startup that tried to address this by supplying training via simulated outages. Maybe we were too early.
teirce
4 days ago
Isn't this the whole point of something like Chaos Monkey? Has the industry at large just lost the plot here?
mschuster91
4 days ago
> Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes.
Aviation is used to paying through the nose for stuff. For developers, employers want to pay as little money as possible.
budman1
4 days ago
But, at this point in history, they are paying through the nose. As they say.
Cut dev salaries by a factor of 3, and then it would be normal.
keeganpoppen
4 days ago
no offense, but I think aviation is a horrible example. if we treated every problem like we treated aviation, we would still be living in the 1950s. honestly, probably the 1850s. hell, we sent a guy to the moon LONG before it was actually safe to do so. (is it safe even now?). do we really think LLMs would be more blasé about human life than humanity itself up until... iono, since the Baby Boomer generation? the "issue", such as it is, with AI is not "we need to make so many guardrails in order for it to be useful", it's that unlike aviation, where the ground truth is "did the plane crash or not?", AI is an open-ended experiment in what would happen where we don't have any feedback from anything other than... how humans feel about it. humans are the antithesis of paperclip maximizers.
I don't see any reason whatsoever why "AI regulations" would become antifragile from their having been "written in blood" in the way that aviation regulations have been; if anything, our inevitable present and future AI crises will each be exponentially more devastating than the last. Modern aviation is a triumph because of how fallible humans are-- that's why we delegate control of aircraft mostly to machines. The shape of AI disruption isn't "the computers aren't following the rules", it's that even by our own rules of rationality our world is terribly underspecified-- we have computer security that isn't secure, subjective rules of ethics/fairness/judgment that defy straightforward explication, "knowledge work" that is actually neither "knowledge" nor "work"... the whole situation is much closer to the human domestication of the dog (letting a potential threat into one's home) than any traditional story of humans coming to terms with their own technological innovations. human behavior is FAR easier to predict than un-aligned AI behavior-- that is why it is such a singular moment in human history.
user
4 days ago
ryanthedev
4 days ago
lol wait until self driving becomes popular.
Those will be fun times.
eric_cc
4 days ago
I wish it was already the case. Tesla FSD is already better than almost every human driver out there.
Onavo
4 days ago
Fuck aviation, stop using it as a dead horse metaphor for safety through human protocols. I am private instrument rating holder and there's very much a top down push for single pilot ops. Your newer jets (even the jetliners) are all designed with that in mind even though they may not be certified as such.
andix
4 days ago
Okay, just because aviation is now also developing in the wrong direction (maybe), all the things they did right in the past are suddenly pointless? I suggest to learn from the things they did right, not the things they did wrong.
bee_rider
4 days ago
This is kind of an interesting thought (underneath the grumpiness, which I guess is where the downvotes came from). Programmers seem to hold aviation processes in very high regard, but their industry is also under cost pressures. The “UX” of riding in a plane does kind of suck, I wonder if some of that comes from the costs involved.
Not too surprised to see pressure to reduce the number of pilots.
Onavo
4 days ago
Oh yeah, and you will find very quickly that most people are a bunch of flaming hypocrites when it comes to flying. Lots of wannabe aerospace engineers in tech. Half the people I have met who put aviation on a pedestal hate flying but love to trumpet the statistics of how safe commercial aviation is. Try asking them to get into a Cessna and all of a sudden the excuses come out. It's oh noes I have a fear of heights, oh noes general aviation is less safe and only slightly safer than a motorcycle, oh noes the FAA is corrupt and allowed the helicopter crash in DC to happen, oh noes I have a family etc.
Aviation safety discussions on hacker news have barely more more depth than biotech and healthcare discussions here. It's extremely ironic because a tremendous number of pilots are engineers, and especially software engineers. It's not exactly a cheap hobby. But you don't see massive numbers of English majors come crawling out of the woodwork to join a discussion on the safety of C++ vs Rust, but tech bros flock to worshipping aviation as a platonic safety idol like moths to a flame.
yallpendantools
4 days ago
Err...have you actually asked someone to get into a Cessna?
Also, "fear of heights" is a totally valid excuse. Fear is a gut reaction and may have nothing to do with one's justified true beliefs. Eg., I know spiders on probability won't harm me but there simply won't be any amicable co-existence between me and any arachnid.
Onavo
4 days ago
> Err...have you actually asked someone to get into a Cessna?
Yes? In practice once you convince them to get into the plane the only real issue in the air for those new to flying is air sickness.
Also fun fact a good number of pilots claim to have a fear of heights (never met one irl except online in aviation forums).
If you are in aviation/aerospace, don't be afraid to share the joy of flying. There's nothing quite like cruising at 90-100 knots @ 3000-4000ft AGL on a nice clear day.
shimman
4 days ago
Not aware of the concerns you have, but they seem just as valid about other similar concerns across industries.
Any books you'd recommend or journalists to follow that cover this beat?
Onavo
4 days ago
Half of aviation safety (especially for commercial aviation) is done on the ground and has nothing to do with aerospace engineering. Stuff like shift times for pilots and ATC, mandated rest periods, proper ground equipment in zero visibility conditions etc. Many developing countries cut corners on maintenance and spare parts. Mandated use of safety technologies (the DC helicopter incident was entirely preventable). Airports that are designed with safety in mind. If you have a dual engine failure immediately after take-off on an airliner in the middle of a built-up city you are generally fucked. Midwestern airports are the best if you have to do a forced approach. Transport category aircraft (basically anything bigger than a 12 seater plane) don't really have many options, they can't easily land on the runway like smaller planes. That's why the Hudson was a miracle. Your airliners stay safe through constant maintenance and inspections. Dual engine failures are very rare (unless through deliberate pilot action like the recent case in India).
On the engineering side you can look up ETOPS and the various ICAO and FAA and EASA standards for airliner class aerospace engineering https://en.wikipedia.org/wiki/ETOPS
Trust me, y'all think airplanes are "safe" because of some sort of magical protocol. No single protocol actually works. It's mostly lots of layers that are stacked on top of each. It's why in aviation, catastrophic failure modeling uses something called the Swiss Cheese model; you need every single hole (error or failure) to line up to generally trigger a complete breakdown in safety. It may sound counter intuitive, but by aviation safety philosophy, a canvas and wooden frame plane operated under commercial standards and kept to a strict inspection schedule is "safer" than e.g. a privately operated 737 that doesn't follow manufacturer maintenance standards.
NASA has plenty of guidelines for writing "safe" aviation software. I doubt any software engineer will be willing to entertain the level of effort and certification needed to comply with those for e.g. their Django side project.
shimman
2 days ago
hmm I'm now curious if you saw the second season of The Rehearsal on HBO. The second season why about pilots and in particular how copilots can be disempowered to speak up when things go wrong. It sounds serious, and it is at times, but also a comedy.
losvedir
4 days ago
I don't understand the perceived hypocrisy. The chance of death in a 1hr Cessna ride is something like 100x a domestic commercial jet flight, so it's reasonable to have different views of them? Maybe I'm missing your point.
RetardedBastard
3 days ago
[flagged]
budman1
4 days ago
Try again. Commercial aviation, with people on board going between point A and point B.
That 2nd pilot makes up for lot's of impossible branches in the fault tree.
Onavo
4 days ago
I believe this is when the term "okay boomer" is warranted.
RetardedBastard
3 days ago
[flagged]
caaqil
4 days ago
Aviation is different because their rent-seeking is enforced by the required legal hedging. I'm not sure the same incentive structure exists in the software side of things.
In other words, we might need to find some hobbies.
goncharow
4 days ago
In what way is requiring someone holding the title pilot to know how to manually fly a plane in case of disaster “rent-seeking”?
caaqil
4 days ago
Rent seeking has a specific definition and it fits here. When Autopilot does almost all the critical work needed (besides announcing frivolous stuff in carbon-based voice), and you have mandatory seats for humans to babysit the Autopilot mainly to satisfy some law, the term fits.
I am not convinced humans can do much better in "disasters" than Autopilot, despite the emotional stories to the contrary (survivorship bias is one hell of a drug).
renehsz
4 days ago
This take is so ignorant and provably false. I can't tell if you're trying to ragebait or if you actually believe this.
The autopilot does not do "almost all critical work", not even close. It only does some of the routine work (definitely not all of it). But the real critical work is when stuff goes wrong, dealing with emergencies.
The inherent complexity of aviation means that there are so many different failure modes and combinations thereof that no traditional system could ever handle all possibilities and edge cases.
I'm not going to claim that an AI could never do it. But as of right now, with current technology, well-trained pilots are absolutely critical to safety in even the most advanced airliners.
Also, the survivorship bias argument runs against you. Most events where pilots save the day are completely invisible. Like catching an incorrect clearance, a misconfigured approach, dealing with a technical malfunction, etc.
And even when disasters happen, it is very rare that simply engaging the autopilot would have saved the day. In fact, there are plenty of examples where the opposite is the case, and dropping to a lower level of automation would, in hindsight, have been better.
caaqil
4 days ago
Your first sentence would lead one to believe that you have some substantial evidence to demonstrate the ignorance and falsehood you identified. It's impressive that you are regurgitating the same formulaic-template set of cliches that boil down to "X is so complex bro, how can you even?".
Note that I never claimed aviation relies on AI right now, most of the Autopilot systems are deterministic automation systems, but the original point stands regardless.
renehsz
3 days ago
Your claim was that "the autopilot does almost all the critical work", and that couldn't be further from the truth in any airliners you can buy today.
I didn't say that an autonomous airliner could never be built. With more technological progress it might become feasible at some point despite the immense complexity.
cyberax
4 days ago
As someone studying for a pilot license, your take is literally ridiculous.
You should perhaps look at all the crashes where the autopilot stalled the aircraft on approach due to incorrect settings.
inoffensivename
4 days ago
As an airline captain, your take is completely ridiculous. The autopilot takes care of the mechanical act of guiding the plane along the desired path, but it does nothing else. There are hundreds of other tasks we need to perform in order to effect a safe flight, and the autopilot does nothing to help us.
ikety
4 days ago
Really? Do autopilots diagnose threatening situations in the main cabin? Understand something unexpected happening on the ground? Visually diagnose situations? Properly identify threat levels?
It's genuinely worrying to me to see statements like this. As engineers, we of all people should know that there's 1 billion variables that could go wrong and another 1 billion we aren't even aware of yet...
Toutouxc
3 days ago
You might want to fire up a flight sim or at least read up on what “autopilot” actually means. It’s pretty much 3D cruise control and that’s it.
marcosdumay
4 days ago
Pilots are nothing near powerful enough to seek any rent on that situation.
Instead, they are the weakest part on that market.
andix
4 days ago
It is a good industry to learn from. I'm not suggesting SW should become exactly like aviation.
conqrr
4 days ago
It indeed is and I like this analogy a lot. A lot of critical systems can definitely learn from aviation. It's one of the finest balances of humans and automation.
budman1
4 days ago
But there is a lot of software in aviation. And it is slow to develop, and expensive. And it has defects.
In general, it is of high quality.