Unless using your own dogfood prevents you from fixing it if it breaks

https://www.theguardian.com/technology/2021/oct/05/facebook-...

I have a memory that Slack fell into this trap too (I could be wrong)

same here, and I just got started, Hm..

When I was at Big Corp, I loved reading the internal postmortems. They were usually very interesting and I learned a lot. It's one of the things I miss about leaving.

A tech company that publishes the postmortems when possible always get a +1 in my eyes, I think it's a sign of good company culture. Cloudflare's are great and I would love to see more from others in the industry.

A big reason for that is it comes from the CEO. Other providers have a team and then at least 2 to 3 layers of management above them and a dotted line legal counsel. So the goal posts randomly shift from "more information" to "no information" over time based on the relationships of that entire chain, the customer heat of the moment, and personality.

Underneath a public statement they all have extremely detailed post-mortems. But how much goes public is 100% random from the customer's perspective. There's no Monday Morning QB'ing the CEO, but there absolutely is "Day-Shift SRE Leader Phil"

Cloudflare deploys stuff on Fridays, and it directly affected shopify, one of their major ecommerce customers. Until they fix their internal processes all writeups should be seen as purely marketing material.

Depending on how long someone's been in the industry it's more a question of if, not when, an outage will occur due to someone deciding to push code haphazardly.

At my first job one of my more senior team members would throw caution to the wind and deploy at 3pm or later on Fridays because he believed in shipping ASAP.

There were a couple times that those changes caused weekend incidents.

In these times, it could be "the AI did it".

It says network routing issue.

Network routes consist of a network (a range of IPs) and a next hop to send traffic for that range to.

These can overlap. Sometimes that’s desirable, sometimes it is not. When routers have two routes that are exactly the same they often load balance (in some fairly dumb, stateless fashion) between possible next hops, when one of the routes is more specific, it wins.

Routes get injected by routers saying “I am responsible for this range” and setting themselves as the next hop, others routers that connect to them receive this advertisement and propagate it to their own router peers further downstream.

An example would be advertising 192.168.0.0/23, which is the range of 192.168.0.0-192.168.1.255.

Let’s say that’s your inference backend in some rows in a data center.

Then, through some misconfiguration, some other router starts announcing 192.168.1.0/24 (192.168.1.0-192.168.1.255). This is more specific, that traffic gets sent there, and half of the original inference pod is now unreachable.

it means their servers were unreachable due to network misconfig.

I already love the product, and I think it would be great to see. Even if its not as "quickly" as CloudFlares (they post ASAP its insane) I would still be happy to see postmortem threads. We all learn industry wide from them.

They turned it off and on again.

Agreed. When cloudflare (ugh, aka the internet) goes down, we can't access information to think and work through. ("the fuel" in some metaphor)

But what about when LLMs go down and a good chunk of a whole generation won't even know how to think, when the remote system goes down? (Is the ability to think "the engine" of self and agency in this metaphor?)

We are building a wildly irresponsible context to exist in.

it is much worse, I forgot how to push to remote so deploys are delayed :)

Nah, I've now resolved to never use it, it'd a total time waster. When it works it's decent, but it does not work like half of time for me.

Is there any utility in thinking about LLM provision in terms of the electricity grid?

I've often thought that local power generation (via solar or wind) could be (or could have been) a viable alternative to national grid supply.

All we need is one research group somewhere in the world releasing good open models.

Ah, you need to buy into this dystopia wholesale. The internet is also down because the LLMs fucked up the BGP routing table, which congress agreed (at the time) should run through the LLM interface.

Imagination, either the first or last thing to die in 2075.

I’m cheating your budget a bit, but for $5600 you can get an M3 Ultra with 256GB of RAM.

https://www.reddit.com/r/LocalLLaMA/

You can run quite powerful models at home on a maxed out Mac Studio. The difference between those and SoTA is more like 2x.

My poorly informed hope is that that we can have mixture of experts with highly tuned models on areas of focus. If I'm coding in language Foo, I only care about a model that understands Foo and its ecosystem. I imagine that should be self-hostable now.

I can stop any time I want, and in fact I am going to stop. Just one more (bug)fix.

Just a warning for those not on the max plan; if you pay by the token or have the lower tier plans you can easily blow through $100s or cap your plan in under an hour. The rates for paying by the token are insane and the scaling from pro to max is also pretty crazy.

They made pro have many times more value than paying per token and then they made max again have 25x more tokens than pro on the $200 plan.

It’s a bit like being offered rice at $1 per grain (pay per token) or a tiny bag of rice for $20 (pro) or a truck load for $200. That’s the pricing structure right now.

So while i agree you can’t easily exceed the quota on the big plans it’s a little crazy how they’ve tiered pricing. I hope no one out there’s paying per token!

Exactly I feel like my brain burns out after a few days. Like Im the limit already (yet im the maximizer also) its a very weird feeling