It might remind you of Crowdstrike because of the scale.

Outages are in a large majority of cases caused by change, either deployments of new versions or configuration changes.

This document by Dr. Cook remains _the standard_ for systems failure. Thank you for bringing it into the discussion.

We incorrectly thought at the time it was attack traffic coming in via WARP into LHR. In reality it was just that the failures started showing up there first because of how the bad file propagated and where it was working hours in the world.

Probably because it was the London team that was actively investigating the incident and initially came to the conclusion that it may be a DDoS while being unable to authenticate to their own systems.

I think the reasoning behind this is because of the nature of the file being pushed - from the post mortem:

"This feature file is refreshed every few minutes and published to our entire network and allows us to react to variations in traffic flows across the Internet. It allows us to react to new types of bots and new bot attacks. So it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly."

The computer science equivalent of choosing between the red, green and blue wires when disarming a nuke with 15 seconds left on the clock

> I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did

Damn corporate karma farming is ruthless, only a couple minute SLA before taking ownership of the karma. I guess I'm not built for this big business SLA.

> I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate.

I'm so jealous. I've written postmortems for major incidents at a previous job: a few hours to write, a week of bikeshedding by marketing and communication and tech writers and ... over any single detail in my writing. Sanitizing (hide a part), simplifying (our customers are too dumb to understand), etc, so that the final writing was "true" in the sense that it "was not false", but definitely not what I would call "true and accurate" as an engineer.

You call this transparency, but fail to answer the most important questions: what was in the burrito? Was it good? Would you recommend?

A very human and authentic response. Love to see it.

Fantastic for recruiting, too.

Appreciate the extra transparency on the process.

A postmortem postmortem, I love it. Transparency to the power of 2.

I really appreciate this level of transparency. Thank you for being a good person in such a powerful position in the world.

I'm not sure I've ever read something from someone so high up in a company that gave me such a strong feeling for "I'd like to work for these people". If job posts could be so informal and open ended, this post could serve as one in the form of a personality fit litmus test.

How do you guys handle redaction? I'm sure even when trusted individuals are in charge of authoring, there's still a potential of accidental leakage which would probably be best mitigated by a team specifically looking for any slip ups.

Thanks for the insight.

that's very cool, thanks

I'm sure he wanted traffic to flow for those few hours but it didn't :p

There’s lots of things we did while we were trying to track down and debug the root cause that didn’t make it into the post. Sorry the WARP takedown impacted you. As I said in a comment above, it was the result of us (wrongly) believing that this was an attack targeting WARP endpoints in our UK data centers. That turned out to be wrong but based on where errors initially spiked it was a reasonable hypothesis we wanted to rule out.

I feel like your username really brings something extra to the party. Now go home.

Mainly to make others aware of what’s happening in the context of this Cloudflare outage. Sure I can avoid giving it visibility/reach but it’s growing and proliferating on its own, and I think ignoring it isn’t going to stop it so I am hoping awareness will help. I’ve noticed a huge rise in open racism against Chinese and Indian and workers of other origin, even when they’re here on a legal visa that we have chosen as a nation to grant for our own benefit.

The legislation that MTG (Marjorie Taylor Green) just proposed a few days ago to ban H1B entirely, and the calls to ban other visa types, is going to have a big negative impact on the tech industry and American innovation in general. The social media stupidity is online but it gives momentum to the actual real life legislation and other actions the administration might take. Many congress people are seeing the online sentiment and changing their positions in response, unfortunately.

> They failed on so many levels here.

That's often the case with human error as especially aviation safety experts know: https://en.wikipedia.org/wiki/Swiss_cheese_model

> Why did they think this was a cyberattack

Isn’t getting cyberattacked their core business?

> Why do they do non-critical changes in production before testing in a stage environment?

I guess the noncritical change here was the change to the database? My experience has been a lot of teams do a poor job having a faithful replica of databases in stage environments to expose this type of issue.

It’s easy to pick on logic that failed and for which you have a very detailed and great post mortem write-up.

Yet you omit to acknowledge that the remaining 99.99999% logic written that powers Cloudflare works flawlessly.

Also, hindsight is 20/20

I know its easy to criticize what happened after the fact and having a clear(er) picture of all the moving parts and the timeline of events, but I think that while most of the people in the thread are pointing out either Rust-related or lack of configuration validation, what really grinds my gears is something that - in my opinion - is bad engineering.

Having an unprivileged application querying system.columns to infer the table layout is just bad; Not having a proper, well-defined table structure indicates sloppiness in the overall schema design, specially if it changes quickly. Considering specifically clickhouse, and even if this approach would be a good idea, the unprivileged way of doing it would be "DESCRIBE TABLE <name>", NOT iterating system.columns. The gist of it - sloppy design not even well implemented.

Having a critical application issuing ad-hoc commands to system.* tablespace instead of using a well-tested library is just amateurism, and again - bad engineering; IMO it is good practice to consider all system.* privileged applications and ensure their querying is completely separate from your application logic; Sometimes some system tables change, and fields are added and/or removed - not planning for this will basically make future compatibility a nightmare.

Not only the problematic query itself, but the whole context of this screams "lack of proper application design" and devs not knowing how to use the product and/or read the documentation. Granted, this is a bit "close to home" for me, because I use ClickHouse extensively (at a scale - I'm assuming - several orders of magnitude smaller than CloudFlare) and I have spent a lot of time designing specifically to avoid at least some of these kind of mistakes. But, if I can do it at my scale, why aren't they doing it?

I don't think these are realistic requirements for any engineered system to be honest. Realistic is to have contingencies for such cases, which are simply errors.

But the case for Cloudflare here is complicated. Every engineer is very free to make a better system though.

> Why did they think this was a cyberattack and only after two hours realize it was the config file?

They explain that at some length in TFA.

> I'm migrating my customers off Cloudflare.

Is that an overreaction?

Name me global, redundant systems that have not (yet) failed.

And if you used cloudflare to protect against botnet and now go off cloudflare... you are vulnerable and may experience more downtime if you cannot swallow the traffic.

I mean no service have 100% uptime - just that some have more nines than others.

I agree. I think the comments about how "it is fine, because so many things had to fail" do not apply in this case.

It's not that many things had to fail, it's that many things that are obvious haven't been done. It would be a valid excuse if many "exotic" scenarios would have to align, not when it's obvious error cases that weren't handled and changes have not been tested.

While having wrong first assumptions is just how things work when you try to analyze the issue[1], not testing changes before production is just stupidity and nothing else.

The story would be different if eg. multiple unlikely, hard to track things happened at once without anyone making a clearly linkable event, something that would also happen in staging. Most of the things mentioned could essentially statically checked. This is the prime example of what you want as any tech person, because it's not hard to prevent compared to a lot of scenarios where you deal with balancing likelihoods of scenarios, timings, etc.

You don't think someone is a great plumber, because they forgot their tools and missed that big hole in the pipe and also rang at the wrong door, because all these things failed. You think someone is a good plumber if they said they would have to go back to fetch a bulky specialized tool, because this is the rare case in which they need it, but they could also do this other thing in this spcific case. They are great plumbers if they tell you how this happened in first place and how to fix it. They are great plumbers if they manage to fix something outside of their usual scope.

Here pretty much all of the things that you pay them for failed. At a large scale.

I am sure this has there are reasons which we don't now about, and I hope that CloudFlare can fix them. Be it management focusing on the wrong things, be it developers not being in the wrong position or annoyed enough to care or something else entirely. However, not doing these things is (likely) a sign that currently they are not in the state of creating reliable systems - at least none reliable enough for what they are doing. It would be perfectly fine if they ran a web shop or something, but if as experienced many other companies rely on you being up or their stuff fails, then maybe you should not run a company with products like "Always Online".

[1] And should make you adapt the process of analyzing issues. Eg. making sure config changes are "very loud" in monitoring. It's one of the most easily tracked thing that can go wrong, and can relatively easily be mapped to a point in time compared to many other things.

So where are you migrating to?

The configuration file is updated every five minutes, so clearly they have some past experience where they’ve decided an hour is too long. That said, even a roll out over five minutes can be helpful.

I think defence against a DDOS against your network is the best reason for a quick rollout

> "To make error is human. To propagate error to all server in automatic way is #devops"

This saying dates back to 1969: To err is human but to really foul things up requires a computer.

* https://quoteinvestigator.com/2010/12/07/foul-computer/

Also: I know there’s a proverb which says ‘To err is human,’ but a human error is nothing to what a computer can do if it tries.

* https://quoteinvestigator.com/2017/05/26/computer-error/

I miss him. It must be more than 10 years now

>Once every 5m is not "rapidly".

Isn't rapidly more of how long it takes to get from A to Z rather than how often it is performed? You can push out a configuration update every fortnight but if it goes through all of your global servers in three seconds, I'd call it quite rapid.

By rapid I mean a rapid rollout of changes to 100% of the fleet, not how often changes are made.

Thanks for sharing that AWS doc

Same as for almost every bug I think: the dev in question hadn't considered that the input could be bad in the way that it turned out to be. Maybe they were new, or maybe they hadn't slept much because of a newborn baby, or maybe they thought it was a reasonable assumption that there would never be more than 200 ML features in the array in question. I don't think this developer will ever make the same mistake again at least.

Let those who have never written a bug before cast the first stone.

> I don't understand why they didn't validate and sanitize the new config file revision.

The new config file was not (AIUI) invalid (syntax-wise) but rather too big:

> […] That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.

> The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.

Thank you. I am sympathetic to CF’s need to deploy these configs globally fast and don’t think slowing down their DDoS mitigation is necessarily a good trade off. What I am saying is this presents a bigger reliability risk and needs correspondingly fine crafted observability around such config changes and a rollback runbook. Greater risk -> greater attention.

This isn't what they do, though. This is a data/config push - original article says _a “feature file” used by our Bot Management system_

Yes the lack of observability is really the disturbing bit here. You have panics in a bunch of your core infrastructure, you would expect there to be a big red banner on the dashboard that people look at when they first start troubleshooting an incident.

This is also a pretty good example why having stack traces by default is great. That error could have been immediately understood just from a stack trace and a basic exception message.

  [lints.clippy]
  dbg_macro = "deny"
  unwrap_used = "deny"
  expect_used = "deny"

What would the difference be if they had enforced no unwraps/expects/slicing and instead logged the error and returned a 500?

As the user, I can't tell the difference, but it might have sped up their recovery a bit.

I would love to go further and explicitely forbid unwrap and similar calls using a `no_panic` attribute.

Back when it meant Site Reliability Engineer and not Sysadmin Really Expensive

Yes? Funnily enough, I don't often use indexed access in Rust. Either I'm looping over elements of a data structure (in which case I use iterators), or I'm using an untrusted index value (in which case I explicitly handle the error case). In the rare case where I'm using an index value that I can guarantee is never invalid (e.g. graph traversal where the indices are never exposed outside the scope of the traversal), then I create a safe wrapper around the unsafe access and document the invariant.

Usually you'd want to write almost all your slice or other container iterations with iterators, in a functional style.

For the 5% of cases that are too complex for standard iterators? I never bother justifying why my indexes are correct, but I don't see why not.

You very rarely need SAFETY comments in Rust because almost all the code you write is safe in the first place. The language also gives you the tool to avoid manual iteration (not just for safety, but because it lets the compiler eliminate bounds checks), so it would actually be quite viable to write these comments, since you only need them when you're doing something unusual.

I mean... yeah, in general. That's what iterators are for.

> Yes, I always thought it was wrong to use unwrap in examples.

And because it gets picked up by LLMs. It would be interesting to know if this particular .unwrap() was written by a human.

Dunno, I think the alternatives have their own pretty significant downsides. All would require front loading more in-depth understanding of error handling and some would just be quite a bit more verbose.

IMO making unwrap a clippy lint (or perhaps a warning) would be a decent start. Or maybe renaming unwrap.

> at least do `expect("damn it, how did this happen")`

That gives you the same behavior as unwrap with a less useful error message though. In theory you can write useful messages, but in practice (and your example) expect is rarely better than unwrap in modern rust

I have to disagree that unwrap is ever OK. If you have to use unwrap, your types do not match your problem. Fix them. You have encoded invariants in your types that do not match reality.

Change your API boundary, surface the discrepancy between your requirements and the potential failing case at the edges where it can be handled.

If you need the value, you need to handle the case that it’s not available explicitly. You need to define your error path(s)

Anything else leads to, well, this.

"Checked Exceptions Are Actually Good" gang, rise up! :p

I think adoption would have played out very different if there had only been some more syntactic-sugar. For example, an easy syntax for saying: "In this method, any (checked) DeepException e that bubbles up should immediately be replaced by a new (checked) MylayerException(e) that contains the original one as a cause.

We might still get lazy programmers making systems where every damn thing goes into a generic MylayerException, but that mess would still be way easier to fix later than a hundred scattered RuntimeExceptions.

It's a lot lighter: a stack trace takes a lot of overhead to generate; a result has no overhead for a failure. The overhead (panic) only comes once the failure can't be handled. (Most books on Java/C# don't explain that throwing exceptions has high performance overhead.)

Exceptions force a panic on all errors, which is why they're supposed to be used in "exceptional" situations. To avoid exceptions when an error is expected, (eof, broken socket, file not found,) you either have to use an unnatural return type or accept the performance penalty of the panic that happens when you "throw."

In Rust, the stack trace happens at panic (unwrap), which is when the error isn't handled. IE, it's not when the file isn't found, it's when the error isn't handled.

> it's essentially a checked exception without a stack trace

In theory, theory and practice are the same. In practice...

You can't throw a checked exception in a stream, this fact actually underlines the key difference between an exception and a Result: Result is in return position and exceptions are a sort of side effect that has its own control flow. Because of that, once your method throws an Exception or you are writing code in a try block that catches an exception, you become blind to further exceptions of that type, even if you might be able to or required to fix those errors. Results are required to be handled individually and you get syntactic sugar to easily back propagate.

It is trivial to include a stack trace, but stack traces are really only useful for identifying where something occurred, and generally what is superior is attaching context as you back propagate which trivially occurs with judicious use of custom error types with From impls. Doing this means that the error message uniquely defines the origin and paths it passed through without intermediate unimportant stack noise. With exceptions you would always need to catch each exception and rethrow a new exception containing the old to add contextual information, then to avoid catching to much you need variables that will be initialized inside the try block defined outside of the try block. So stack traces are basically only useful when you are doing Pokemon exception handling.

> When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.

It's not a checked exception without a stack trace.

Rust doesn't have Java's checked or unchecked exception semantics at the moment. Panics are more like Java's Errors (e.g. OOM error). Results are just error codes on steroids.

checked exceptions failed because when used properly they fossilize method signatures. they're fine if your code will never be changed and they're fine when you control 100% of users of the throwing code. if you're distributing a library... no bueno.

That's kind of what I'm saying with the blind spot comment. The words "unwrap" and "expect" should be just as much a scary red flag as the word "panic", but for some reason it seems a lot of people don't see them that way.

Even in lowly Java, they later added to Optional the orElseThrow() method since the name of the get() method did not connote the impact of unwrapping an empty Optional.

A lot of stuff should be done about the awful unwrap family of methods.

A few ideas:

- It should not compile in production Rust code

- It should only be usable within unsafe blocks

- It should require explicit "safe" annotation from the engineer. Though this is subject to drift and become erroneous.

- It should be possible to ban the use of unsafe in dependencies and transitive dependencies within Cargo.

Really not! This is a huge faceplant for writing things in Rust. If they had been writing their code in Java/Kotlin instead of Rust, this outage either wouldn't have happened at all (a failure to load a new config would have been caught by a defensive exception handler), or would have been resolved in minutes instead of hours.

The most useful thing exceptions give you is not static compile time checking, it's the stack trace, error message, causal chain and ability to catch errors at the right level of abstraction. Rust's panics give you none of that.

Look at the error message Cloudflare's engineers were faced with:

     thread fl2_worker_thread panicked: called Result::unwrap() on an Err value

A proxy stack written in a managed language with exceptions would have given an error message like this:

    com.cloudflare.proxy.botfeatures.TooManyFeaturesException: 200 > 60
        at com.cloudflare.proxy.botfeatures.FeatureLoader(FeatureLoader.java:123)
        at ...

In the past I've been able to diagnose production problems based on stack traces so many times I was been expecting an outage like this ever since the trend away from providing exceptions in new languages in the 2010s. A decade ago I wrote a defense of the feature and I hope we can now have a proper discussion about adding exceptions back to languages that need them (primarily Go and Rust):

https://blog.plan99.net/what-s-wrong-with-exceptions-nothing...

Alternatively you can look at actually innovative programming languages to peek at the next 20 years of innovation.

I am not sure that watching the trendy forefront successfully reach the 1990s and discuss how unwrapping Option is potentially dangerous really warm my heart. I can’t wait for the complete meltdown when they discover effect systems in 2040.

To be more serious, this kind of incident is yet another reminder that software development remains miles away from proper engineering and even key providers like Cloudfare utterly fail at proper risk management.

Celebrating because there is now one popular language using static analysis for memory safety feels to me like being happy we now teach people to swim before a transatlantic boat crossing while we refuse to actually install life boats.

To me the situation has barely changed. The industry has been refusing to put in place strong reliability practices for decades, keeps significantly under investing in tools mitigating errors outside of a few fields where safety was already taken seriously before software was a thing and keeps hiding behind the excuse that we need to move fast and safety is too complex and costly while regulation remains extremely lenient.

I mean this Cloudfare outage probably cost millions of dollars of damage in aggregate between lost revenue and lost productivity. How much of that will they actually have to pay?

I would say, sure, if you feel the same way about panic calls making to production. In other words, review all of them the same way. Because writing unwrap/expect is exactly the same as writing “if error, panic”.

> If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.

It wouldn't, not meaningfully. The outage was caused by change in how they processed the queries. They had no way to observe the changes, nor canaries to see that change is killing them. Plus, they would still need to manually feed and restart services that ingested bad configs.

`expect` would shave a few minutes; you would still spend hours figuring out and fixing it.

Granted, using expect is better, but it's not a silver bullet.

If the `.unwrap()` was replaced with

1:

  ?

  map_err, or, or_else, etc.

  match ... { 
    Ok(..) => {}, 
    Err(..) => {},
  }

  if let ... {
  } 

The function signature returned a `Result<(), (ErrorFlags, i32)>`

Seems like it should have returned an Err((ErrorFlags, i32)) here. Case 2 or 3 above would have done nicely.

Removing unwrap() from Rust would have forced the proper handling of the function call and would have prevented this.

Unwrap() is Rust's original sin.

In general for unexpected errors like these the internal function should log the error, and I assume it was either logged or they can quickly deduce reason based on the line number.

I'm not sure if this is serious or not, but to take it at face value: the value of this sort of thing in Rust is not that it prevents crashes altogether but rather that it prevents _implicit_ failures. It forces a programmer to make the explicit choice of whether to crash.

There's lots of useful code where `unwrap()` makes sense. On my team, we first try to avoid it (and there are many patterns where you can do this). But when you can't, we leave a comment explaining why it's safe.

blaming the language is not the way to approach this. if an engineer writes bad code that’s the engineers fault, not the languages.

this was bad code that should have never hit production, it is not a rust language issue.

> Instead, the routine likely should have worked within `Result`.

But it's a fatal error. It doesn't matter whether it's implicit or explicit, the result is the same.

Maybe you're saying "it's better to be explicit", as a broad generalization I don't disagree with that.

But that has nothing to do with the actual bug here, which was that the invariant failed. How they choose to implement checking and failing the invariant in the semantics of the chosen language is irrelevant.

That's exactly the point (ie just prior to distribution) where a simple sanity check should have been run and the config replacement/update pipeline stopped on failure. When they introduced the 200 entry limit memory optimised feature loader it should have been a no-brainer to insert that sanity check in the config production pipeline.

Or even truncating the features to their limit and alerting through logs that there is likely performance degradation in their Bot Management.

I'm really confused how so many people are finding it acceptable to bring down your entire reverse-proxy because the length of feature sets for the ML model in one of your components was longer than expected.

Would expect with a message meet that criteria of exiting with a more helpful error message? From the postmortem it seems to me like they just didn’t know it even was panicing

> However Nginx doesn’t panic on malformed config. It exits with hopefully a helpful error code and message.

The thing I dislike most about Nginx is that if you are using it as a reverse proxy for like 20 containers and one of them is up, the whole web server will refuse to start up:

  nginx: [emerg] host not found in upstream "my-app"

  location / {
      resolver 127.0.0.11 valid=30s; # Docker DNS
      set $proxy_server my-app;
      proxy_pass http://$proxy_server:8080/;
      proxy_redirect default;
  }

  nginx: [emerg] "proxy_redirect default" cannot be used with "proxy_pass" directive with variables

That's to say that all software out there has some really annoying failure modes, even is Nginx is pretty cool otherwise.

In this case it definitely wasn’t the least bad option though.

Unsafe blocks have nothing to do with it. Yes - they maintain all the same invariants as safe blocks or those unsafe blocks are unsound regardless of panics. But there’s millions of way to architect this (eg a supervisor process that notices which layer in FL2 is crashing and just completely disables that layer when it starts up the proxy again. There’s challenges here because then you have to figure out what constitutes a perma crashing (eg what if it’s just 20% of all sites? Do you disable?). And in the general case you have the fail open/fail close decision anyway which you should just annotate individual layers with.

But the bigger change is to make sure that config changes roll out gradually instead of all at once. That’s the source of 99% of all widespread outages

I think the parent is implying that the panic should be "caught" via a supervisor process, Erlang-style, rather than implying the literal use of `catch_unwind` to resume within the same process.

The unwrap should be replaced by code that creates enough alerting to make a P0 incidident from their canary deployment immediately.

OR even, the bot code crashing should itself be generating alerts.

Canary deployment would be automatically rolled back until P0 incident resolved.

All of this could probably have happened and contained at their scale in less than a minute as they would likely generate enough "omg the proxy cannot handle its config" alerts off of a deployment of 0.001% near immediately.

They would also tell you that .unwrap() has no place in production code, and should receive as much scrutiny as an `unsafe` block in code review :)

The point of option is the crash path is more verbose and explicit than the crash-free path. It takes more code to check for NULL in C or nil in Go; it takes more code in Rust to not check for Err.

Out of interest... apart from the em dash, how else can you tell it's an LLM response? What are the telltales signs?

It’s kinda funny that the “not …, but…” + em dash slop signifiers translate directly to Chinese “不是。。。而是。。。” and double-width em dash

Yeah @IBOutlets are generally the one thing that are allowed to be implicitly-unwrapped optionals. They go along with using storyboards & xibs files with Interface Builder. I agree that you really should just crash if you are attempting to access one and it is nil. Either you have done something completely incorrect with regards to initializing and accessing parts of your UI and want to catch that in development, or something has gone horribly, horribly, horribly with UIKit/AppKit and storyboard/xib files are not being loaded properly by the system.

> Silent failure is ultimately terrible for users.

Agreed.

Unfortunately, crashes in iOS are “silent failures,” and are a loss of control.

What this practice does, is give me the option to handle the failure “noisily,” and in a controlled manner; even if just emitting a log entry, before calling a system failure. That can be quite helpful, in threading. Also, it gives me the option to have a valid value applied, if there’s a structural failure.

But the main reason that I do that with @IBOutlets, is that it forces me to acknowledge, throughout the rest of the code, that it’s an optional. I could always treat implicit optionals as if they were explicit, anyway. This just forces me to.

I have a bunch of practices that folks can laugh at, but my stuff works pretty effectively, and I sleep well.

> This is terrible.

No it's not. Read my other comments.

ToMAYto, ToMAHto.

I have learned that it's a bad idea to trash other folks' methodologies without taking the time to understand why they do things, the way they do.

I have found dogma to be an impediment, in my own work. As I've gotten older, the sharp edges have been sanded off.

Have a great day!

It looks like changing the permissions triggered creation of a new feature file, and it was ingestion of that file leading to blowing a size limit that crashed the systems.

The file should be versioned and rollout of new versions should be staged.

(There is definitely a trade-off; often times in the security critical path, you want to go as fast as possible because changes may be blocking a malicious actor. But if you move too fast, you break things. Here, they had a potential poison input in the pathway for synchronizing this state and Murphy's Law suggests it was going to break eventually, so the question becomes "How much damage can we tolerate when it does?")

Right, but the point isn't to make errors impossible; the point is to have them be 1) less likely to write, and 2) easier to spot on review.

People's biggest complaints about golang's errors:

1. You have to _TYPE_OUT_ what to do on EVERY.SINGLE.ERROR. SOO BOORING!

2. They clutter up the code and make it look ugly.

Rust is so much cleaner and more convenient (they say)! Just add ?, or .unwrap()!

Well, with ".unwrap()", you can type it fast enough that you're on to the next problem before it occurs to your brain to think about what to do if there is an error. Whereas, in golang, by the time you type in, "if err != nil {", you've broken the flow enough that now you're much more likely to be thinking, "Hmm, could this ever fail? What should we do if it does?" That break in flow is annoying, but necessary.

And ".unwrap()" looks so unassuming, it's easy to overlook on review; that "panic()" looks a lot more dangerous, and again, would be more likely to trigger a reviewer into thinking, "Wait, is it OK if this thing panics? Is this really so unlikely to happen?"

Renaming it `.unwrap_or_panic()` would probably help with both.

In C++, functions are error-colored by default. You write "noexcept" if you want your function to be infallible-colored instead.

(You usually want to make a function infallible if you're using your noexcept function as part of a cleanup path or part of a container interface that allows for more optimizations of it knows certain container operations are infallible.)

Rust makes infallibility the syntactic default and makes you write Result to indicate fallibility. People often don't want to color their functions this way. Guess what happens when a programmer is six levels deep in infallible-colored function calls and does something that can fail.

.unwrap()

Guess what, in Rust, is fallible?

Mutex acquire.

Guess what you need to do often on infallible cleanup paths?

Mutex acquire.

Yeah I know but isn't unwrap basically a trivial way to: (1) give up catching the exception/error (the E part in `Result<T, E>`) that the callee throws; and also (2) escalate it to the point that nothing can catch it. It has such a benign name.

Rust's Result is the same thing as C++'s std::expected. How is calling std::expected::value undefined behaviour?

Linting is not good enough. The compiler should refuse to compile the code without it marked with an explicit annotation. Too much Rust code is panic happy since using casual use of `unwrap` is perma-etched into everyone's minds by the amount of demo code out there using unwrap.

and people make mistake

at some point machine would be better in coding because well machine code is machine instruction task

same like chess, engine is better than human grandmaster because its solvable math field

coding is no different

Continuing only makes sense for cases you know you can handle.

_In theory_ they could have used the old config, but maybe there are reasons that’s not possible in Cloudflare’s setup. Whether or not that’s an invariant violation or just an error that can be handled and recovered from is a matter of opinion in system design.

And crashing on an invariant violation is exactly the right thing to do rather than proceed in an undefined state.

Idiomatically, it gives you `err` and you do `if err != nil {return err}`. While in rust you mostly do `.unwrap` and panic.

It's not in the type system, but it's idiomatic

which mean an unexpected behavior could go unnoticed for a long time.

I'd prefer a loud crash over that.

> make them look like a GIANT EYESORE

React.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED comes to mind. I did have to reach to this before, but it certainly works for keeping this out of example code and other things like reading other implementations without the danger being very apparent.

At some point it was renamed to __CLIENT_INTERNALS_DO_NOT_USE_OR_WARN_USERS_THEY_CANNOT_UPGRADE which is much less fun.

right and if the language designers named it UNWRAP_OR_PANIC() then people would rightfully be asking why on earth we can't just use a try-catch around code and have an easier life

It also makes it very obvious in the code, something very dangerous is happening here. As a code reviewer you should see an unwrap() and have alarm bells going off. While in other languages, critical errors are a lot more hidden.

> What was wrong was the developer assumption, not the use of unwrap.

How many times can you truly prove that an `unwrap()` is correct and that you also need that performance edge?

Ignoring the performance aspect that often comes from a hat-trick, to prove such a thing you need to be wary of the inner workings of a call giving you a `Return`. That knowledge is only valid at the time of writing your `unwrap()`, but won't necessarily hold later.

Also, aren't you implicitly forcing whoever changes the function to check for every smartass dev that decided to `unwrap` at their callsite? That's bonkers.

> when you do unwrap() you're saying the Result should NEVER fail

Returning a Result by definition means the method can fail.

It's easy to cause this kind of failure in Haskell also.

And Gleam

Works when you have the Erlang system that does graceful handing for you: reporting, restarting.

> Do you think that some people claim that Rust's memory safety guarantees mean that a Rust program is incapable of crashing or having a bug?

Undoubtedly yes.

> ...but what does this have to do with match blocks?

You tell me. You're the one advocating for placing one after every single function call.

> If that’s the case, the old config is already in memory when the new config is being parsed

I think that's explicitly a non-goal. My understanding is that Cloudflare prefers fail safe (blocking legitimate traffic) over fail open (allowing harmful traffic).

Even if you want it to crash, you almost never unwrap. At the very least you would use .expect() so you get a reasonable error message -- or even better you handle the potential error.

> Crashing is not an outage.

Are you in the right thread?

The type system is for asserting assumptions like "this cannot fail". You don't crash at all.

You're the nth person on this thread to say that and it doesn't make sense. Events that happen multiple times per second change data that you would call "configuration" in systems like these. This isn't `sendmail.cf`.

If you want to say that systems that light up hundreds of customers, or propagate new reactive bot rules, or notify a routing system that a service has gone down are intrinsically too complicated, that's one thing. By all means: "don't build modern systems! computers are garbage!". I have that sticker on my laptop already.

But like: handling these problems is basically the premise of large-scale cloud services. You can't just define it away.

Is it actually consul-template? (I have post-consul-template stress disorder).

I think they need to strongly evaluate if they need this level of rollout speed. Even spending a few minutes with an automated canary gives you a ton of safety.

Even if the servers weren't crashing it is possible that a bet set of parameters results in far too many false positives which may as well be complete failure.

CloudFront has quotas[0] and they likely just hit those quota limits. To request higher quotas requires a service ticket. If they have access logs enabled in CloudFront they could see what the exact error was.

And since it seems this is hosted by Atlassian, this would be up to Atlassian.

[0] https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...

Probably non zero number of companies use cloudfront and other cdns as fallback for cloudflare or running a blended cdn so not surprising to see other cdns hit with a thundering herd when cloudflare went down

ahah oops. yeah, it's a problem. i've got two projects ongoing that each rely on one of them, and i can never keep it straight.

I think the idea is to ship the sqlite database around.

It’s not a terrible idea, in that you can test the exact database engine binary in CI, and it’s (by definition) not a single point of failure.

This isn't really "configuration" so much as it is "durable state" within the context of this system.

In the early 2000s when Google explained how they achieved their (already back then) awesome reliability, ie assuming that any software and hardware will eventually fail, and that they designed everything with the idea that everything was faulty, there were some people who couldn't get it, who would still bring the argument that "yeah but today with modern raid..."

People here chatting about unwrap remind me of them :)

Propagating upwards a valid way of handling it and often the correct answer.

There needs to be something at the top level that can handle a crashing process.

Only formal proof languages are immune to such properties. Therefore all languages are poorly designed by your metric.

Consider that the set of possible failures enabled by language design should be as small as possible.

Rust's set is small enough while also being productive. Until another breakthrough in language design as impactful as the borrow checker is invented, I don't imagine more programmers will be able to write such a large amount of safe code.

No, the "if it compiles, it works" is genuinely about the program being correct rather than just free of memory errors, but it's more of a hyperbolic statement than a statement of fact.

It's a common thing I've experienced and seen a lot of others say that the stricter the language is in what it accepts the more likely it is to be correct by the time you get it to run. It's not just a Rust thing (although I think Rust is _stricter_ and therefore this does hold true more of the time), it's something I've also experienced with C++ and Haskell.

So no, it's not a guarantee, but that quote was never about Rust's guarantees.

Well let me avoid those that don’t understand it. It’s literally Rust 101.

I've worked on commercial codebases that did better, shrug.

> What happens to it up the callstack?

as they say in the post, these files get generated every 5 minutes and rolled out across their fleet.

so in this case, the thing farther up the callstack is a "watch for updated files and ingest them" component.

that component, when it receives the error, can simply continue using the existing file it loaded 5 minutes earlier.

and then it can increment a Prometheus metric (or similar) representing "count of errors from attempting to load the definition file". that metric should be zero in normal conditions, so it's easy to write an alert rule to notify the appropriate team that the definitions are broken in some way.

that's not a complete solution - in particular it doesn't necessarily solve the problem of needing to scale up the fleet, because freshly-started instances won't have a "previous good" definition file loaded. but it does allow for the existing instances to fail gracefully into a degraded state.

in my experience, on a large enough system, "this could never happen, so if it does it's fine to just crash" is almost always better served by a metric for "count of how many times a thing that could never happen has happened" and a corresponding "that should happen zero times" alert rule.

The way I’ve seen this on a few older systems was that they always keep the previous configuration around so it can switch back. The logic is something like this:

1. At startup, load the last known good config.

2. When signaled, load the new config.

3. When that passes validation, update the last-known-good pointer to the new version.

That way something like this makes the crash recoverable on the theory that stale config is better than the service staying down. One variant also recorded the last tried config version so it wouldn’t even attempt to parse the latest one until it was changed again.

For Cloudflare, it’d be tempting to have step #3 be after 5 minutes or so to catch stuff which crashes soon but not instantly.

Presumably you kick up the error to a level that says “if parsing new config fails, keep the old config”

Yeah, see, that's what I mean.

Unwrap is controversial. The problem is that if you remove it, it makes the bar even higher for newcomers to Rust. One solution is to make it unsafe (along with panic).

> the "no true Rustacean would have put unwrap in production"

The "no unwrap" rule is common in most production codebases. Chill.

> Costs to architect systems that serve millions of request daily have gone down. Not up.

I never said serving millions of requests is more expensive. Protecting your servers is more expensive.

> Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client.

HN uses Cloudflare. You're making my point for me. If you included the fixed costs that Cloudflare's CDN/proxy is giving to HN incredibly cheaply, then running HN at the edge with good performance (and protecting it from botnets) would costs hundreds of millions of dollars.

> People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)

Again, attacking your own straw men here.

Writing high-throughput web applications is easier than ever. Hosting them on the open web is harder than ever.

Cloudflare and us-east-1 are single points of failures for millions of their customers, many of which overlap.

"Single" means "you only need one," not that there is only one.

not necessarily, by default `Result` does not even carry a stack trace

> but Rust's type system did catch this error - and then author decided it's fine to panic if this error happens

The issue is that it wasn't fine to panic, thus Rust did not catch this error.

After The Great If-Let Outage Of 2024, we audited all our code for that if-let/rwlock problem, changed a bunch of code, and immediately added a watchdog for deadlocks. The audit had ~no payoff; the watchdog very definitely did.

I don't know enough about Cloudflare's situation to confidently recommend anything (and I certainly don't know enough to dunk on them, unlike the many Rust experts of this thread) but if I was in their shoes, I'd be a lot less interested in eradicating `unwrap` everywhere and more in making sure than an errant `unwrap` wouldn't produce stable failure modes.

But like, the `unwrap` thing is all programmers here have to latch on to, and there's a psychological self-soothing instinct we all have to seize onto some root cause with a clear fix (or, better yet for dopaminergia, an opportunity to dunk).

A thing I really feel in threads like this is that I'd instinctively have avoided including the detail about an `unwrap` call --- I'd have worded that part more ambiguously --- knowing (because I have a pathological affinity for this community) that this is exactly how HN would react. Maybe ironically, Prince's writing is a little better for not having dodged that bullet.

I am one of those old grey beards (or at least, I got started shipping C code in the 1990s), and I'd leave asserts in prod serverside code given the choice; better that than a totally unpredictable error path.

> You are saying this would not have happened in a C release build where asserts define to nothing?

Afaik, Go and Java are the only languages that make you pause and explicitly deal with these exceptions.

To be fair, if you’re not “this tall” you really shouldn’t consider using goto in a c program. Most people aren’t that tall.

Yes it's meant to be used in test code. If you're sure it can't fail do then use .expect() that way it shows you made a choice and it wasn't just a dev oversight.

panicans should be using .expect() in production

There's certainly a discipline involved here, but it's usually something like guaranteeing all threads are unwind safe (via AssertUnwindSafe) and logging stack traces when your process keeps dying/can't be started after a fixed number of retries. Which would lead you to the culprit immediately.

I'm just pushing back a bit on the idea that unwrap() is unsafe - it's not, and I wouldn't even call it a foot gun. The code did what it was written to do, when it saw the input was garbage it crashed because it couldn't make sense of what to do next. That's a desirable property in reliable systems (of course monitoring that and testing it is what makes it reliable/fixable in the first place).

But now consider how much extra data Cloudflare at its size would have to have just for staging, doubling or more their costs to have stage exactly as production. They would have to simulate similar amount of requests on top of themselves constantly since presumably they have 100s or 1000s of deployments per day.

In this case it seems the database table in question seemed modest in size (the features for ML) so naively thinking they could have kept stage features always in sync with prod at the very least, but could be they didn't consider that 55 rows vs 60 rows or similar could be a breaking point given a certain specific bug.

It is much easier to test with 20x data if you don't have the amount of data cloudflare probably handles.

The irony of a rust rewrite taking down the internet is not lost on me.

Yes, that was missing (along with the London WARP thing). Other comments mentioned that their status page is an Atlassian Statuspage solution, hosted on AWS CloudFront.

Unclear to me if it's an Atlassian-managed deployment they have, or if it's self-managed, I'm not familiar with Statuspage and their website isn't helping. Though if it's managed, I'm not sure how they can know for sure there's no interdependence. (Though I guess we could technically keep that rabbit hole going indefinitely.)