I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.

> Exactly this.

It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.

That was the final straw that stopped me using it.

I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.

They also charge ahead to prod with known defects and then change their story when they're caught out https://news.ycombinator.com/item?id=48032310

Ironically wouldn't someone who solves real problems at scale actually be a 'keyboard warrior'?

At least they aren't creating new problems at scale either.

I solve real problems at scale and the engineering practices I see on a daily basis are a clown show.

There's little to no basic understanding of networking, distributed systems, databases, etc. 99% of our engineers were hired from their college internships and never worked anywhere else. Industry hires to improve systems rarely last more than a year and it is almost never their fault.

We're in the next tier down from the biggest tech companies and what we do is hardly uncommon among our peers.

I should be shocked that 99% of engineers I deal with treat all resources as infinite bandwidth, 100% uptime, but I'm not. They NIH super hard and write tons of code for things that a docker container running nginx (or similar) would solve in 5 minutes. There's almost no useful testing and worse documentation.

Welcome to corporate life.

Is GitHub a tech company? Because that's who they're criticising.

It's less about "can't afford" and more about "don't want to spend".

Microsoft is very much not FAANG. They’ve been trying to eliminate dedicated QA for years and underpay contractors.

Microsoft can afford to do a lot of things, but why when you can squeeze a bit more profit out.

> It's Microsoft

That means nothing, especially with all the layoffs.

One thing that I learned over the years is never mystify "code quality". Maybe you can say a certain team/project produces high quality code, and maybe Google's overall quality is better than my company's (considering their bar of hiring), but you want to avoid generalizing that, especially at scale.

The average software engineer at a large company ships just enough code at good enough quality to earn a salary, nothing fancy about it.

Answer is, you do comprehensive unit testing irrespective of Microsoft or any other company doing it. Also, as the OP of your comment posted, you are over estimating the software quality of these big name corporate companies. It can’t be the bar against which you would want measure your own company standards.

Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.

We used to be able to afford two! Acronym letters cost as much as houses nowadays!

and all of high quality

https://github.com/oven-sh/bun/issues/39797

And 5K PRs. I'm crying.

Strange to think they are probably triaged by LLMs at this point

That would just drive users into the arms of the alternatives, which would love to own the world's code... like Cursor/Musk.

Microsoft and GitHub's only option is to suck it up, absorb this growth, and lower failure rates. They have the money, so that's not the issue.

As someone on the sidelines, this is really interesting to watch unfold.

so... when you reach your monthly limit in the middle of the month, what is the recourse?

Or, is the idea just to drive everyone away from your platform?

I still remember when they decided to limit the number of private repos you could have as a free user. Kind of silly to me at the time, and even more so now!

Which will just increase the cries of "enshittification" and hasten the mass migration to the next free platform that surely, this time, won't ever go down.

In what way? If I have multiple local commits pushed once I expect to see CI type actions to run once for the push rather than for ever commit in the push.

But they can't process the commit until I push it?

Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push.

You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.

One has to wonder what the point of a collaborative version control system even is if the software we’re writing is one-shotted by a call to an API ?

Serious question: do you think your 6 year old is learning coding from prompting an AI?

I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. This is proven on a nearly daily basis at my job, where, with identical agents and prompts, I see designs being pushed with objectively incorrect facts, sub-optimal code in PRs, and a general explosion of slop. That tells me that it still very much matters if you know how to do the job without the help of robots.

And what would be the expected result (other than spending $20 a month)?

Is it impressive? All it's doing is decaying the services. 15 years ago never have imagined I would go to the lengths to host a github alternative on a VPS but after doing just this (also being the last one in the my professional group to do so), GitHub is giving a master class in destroying their reputation in pursuit of advocating for hostile entities.

Not all growth is good, especially growth that is actively hurting the company.

There is an equilibrium in both nature and software. Purposely designing systems that mimic the effects of cancer is going to benefit who exactly?

How is it impressive if we all know it's autogenerated? There's no more people there than there were before. Heck, at ~2x growth that's possibly a decrease in real humans there since bots generate loads of them per person

Should use on premises.

They do. Talk to your account representative for the alternate domain name to hit.

AWS saw this growth every year for two decades, hyper growth tech sees it all the time

cloud providers and hyper growth tech deal with these growth rates all the time

this is common at AWS/cloud providers

I am afraid thats now how infrastructure works from what I have seen. The number that really matters is QPS. For any system the QPS varies through out the day and across the week and months. Most design considerations easily absorb any 2X increase. Pick up any company and the chances are that the servers are over provisioned, no one takes chances specially with critical components.

What you have going on with Github is mix of multiple things. Traffic alone is not the cause from what little I know, it does adds to the problem for sure

1. Infrastructure is being moved to use Azure, and overall all the cloud providers are struggling with hardware at the moment (same is going on for linkedin too)

2. The core teams, the people who knew the existing systems have either been laid off or moved from Github

3. Microsoft veterans are brought in to fill the gap across the board, they are trying their best but its a lot of unknown for them

Sure, but the new patterns are several years old at this point.

Not adapting is a choice

No doubt. ...You can have non-zero revenue and still be loss-making though.

Jittered exponential backoff. You don't want the whole herd to come back at the same time, you have to add timing jitter to the clients.

Don’t forget jitter!

Exponential is also overkill (even with jitter as others have mentioned).

I seem to remember there was a "you failed 5 PIN entries in a row, please wait 500000 seconds before you retry" on Apple phones. So, you probably also want a sensible max... which makes exponential a bit pointless. Just do a basic fixed delay + (large, e.g. 0.5 x the delay) jitter and you'll be fine for most things. You can add a bit of cumulative delay if it's really costly to do retries.

Don't forget jitter as well. Otherwise your clients will just coordinate their retries.

This is one of the scenarios where I feel we as engineers should have been more diligent in publicly writing down what techniques have worked in which scenarios and what haven't, and the AI could have gathered decades of operational knowledge and provide proper guidance to architects designing new systems. It's also true that we're witnessing unprecedented levels of scale.

Google “thundering herd” and you’ll understand why uncontrolled retries can be / are bad.

Exponential back off retries could hide a real issue, but I would estimate something like 99.99999999% of network retries are resolved within the first 2 attempts.

Not using retries is optimizing for the astronomically rare case, which is better mitigated by other means

The Github outage was about internal clients. Phone apps are a reasonable place to say things are known to be unreliable and can't be fixed. Your IP address changes when you leave the house.

Btw, PWAs added offline capabilities to websites. I hate how the only thing that got used for was these stupid pages that look like you were able to reach the site but it's actually just saying you have no internet, like YouTube.

I wasn’t being completely serious with my suggestion, but I was thinking of a totally PAYG model. Monthly plans could come with credits, like they already do for Actions.

You’re torturing your own logic to make Azure the villain here. And it also sounds like you lack experience with capacity exhaustion. Things fail slowly, then suddenly.

I’m firmly in the camp of “something stinks at GitHub” but

> It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage

In my experience, there are hard thresholds that get passed that expose hidden bottlenecks like this. A previous system I worked on we had absolutely loads of headroom by all of our measured metrics, but one day we filled a cache because the value hadn’t been tweaked in recent memory. Plenty of space on disk and in memory, but all of a sudden we went from a very high cache hit rate to a very low cache hit rate, and everything ground to a halt.

There's always a cliff, this part is fine. You sometimes know the cliff but often do not.

This absolutely can happen in large systems. If some part of the system is at capacity, then slightly increasing the load can cause it to fall behind and start accumulating a backlog.

These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.

>"While accelerating our migration to Azure," meaning, they will only solve problems if it helps them also use Azure more.

Look, I hate Microslop as much as anyone but you'd have to purposely misinterpret TFA in order to arrive at this interpretation. C'mon.

that is more manageable but c'mon I can't even keep Google One 100GB up on a consistent basis, that's how poor I am. self-hosting would be a far better option because apparently I find enough people to provide free Hetzner VPSes and stuff as long as I can sell this as mutually beneficial.

for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets you integrate it with a status bar (or anything in nvim, really). that would be a decent measure against drive-by slop contributions, and I'd accept contribs over private github mirror from those who I know but can't bother setting up git mail

EDIT: TIL that one can host SourceHut themselves. discoverability may still be a problem (sr.ht just ranks higher in search engines) but 1) fixable with github mirror that points to sourcehut instance as a canonical development platform, 2) it's moderately easy to sync contributions between tangled and sourcehut, so tangled is also an option

EDIT 2: the email part would be PITA, so $4/mo is attractive on that background

> We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.

That can't be cheap.

Microsoft don't release the costs as you know, but

Compute and Storage for Free Tiers: Hosting code for over 150 million developers and processing over 2 billion GitHub Actions (CI/CD) workflows a month requires astronomical server power and data storage. The "Free" tier is a massive cost sink that Microsoft treats as a loss-leader marketing expense

Let me know when you understand how that's not free.

I never found GitHub's systems for issue tracking to be all that great. Cross-repository issues and development plans are hard to track within a git host. I've always used an external panning and issue tracking tool, mostly Linear, and it works really well. GitLab's Linear integration is excellent, FWIW.

I've actually worked with a couple of companies who do use GitHub for their code, and they all use Linear in addition to GitHub.

I understand the concern you're talking about wrt. Codeberg, but I wouldn't view it as a significantly bigger risk than anything else. Any platform can suddenly decide that your project is against ToS (GitHub will absolutely not accept just anyone's use of their platform either) and Codeberg introducing some rules recently doesn't, in my mind, drastically increase the risk of a dramatic ToS change in the future. But we all have to make our own risk evaluations and I won't judge yours. Luckily, moving between Git hosts isn't that difficult; setting up CI again and losing merge request history does suck but it's not the end of the world, unlike something like, say, losing your AWS/GCP/whatever account.

"sophistication" seems like a strange way to describe GitHub to me - I've found in every individual aspect (code browsing, issue tracking, code review, package management, etc), it's the worst out of all the systems I use regularly... But it's good _enough_ for most people, and it has all those features in one place, which is more convenient than wrangling 10-15 high quality but disconnected systems

Gitlab has the benefit of having very little traffic, both free and paid. Their limits are still way above the current usage so less likely to be an issue

Dude no one uses github tracking for anything serious, come on. The only thing github has over gitea/forgejo is discussions. That's the only real social "innovation" github has contributed to open source development and it's just a shitty tacked on forum.

Also for accuracy, Codeberg has a pro-human and anti-corporation stance. Both of which are definitely en vogue at the moment.

Much better than GitHub's pro slop sentiment, which is doing nothing but destroy their reputation.

“Software forge” is an established[1] term by this point: a piece of software providing a full set of features for collaborative development, usually at least code hosting with a web interface, tickets, and webpage hosting, and these days often also code review and integrated CI or at least the possibility of integrating CI. It’s admittedly squishy but in the same way “IDE” is squishy, which is to say it’s still well-defined enough to be useful.

[1] https://en.wikipedia.org/wiki/Forge_(software)

> a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.

You're in luck, GP comment described it for you.

Forgejo > The name of the software

self-hosted > You install it on your server

lightweight > It does not consume a lot of resources (cpu, disk, ram)

software forge > offers tools that help with creating software collaboratively (repository hosting, change request management, wiki for docs,…)

The exciting thing about AI is precisely that it'll let software move beyond Yet Another B2B SaaS and into doing useful things in the real world. I regularly ride in driverless cars! That was the stuff of science fiction when I was a kid.

If you're worried about food prices, you should be happy that robots will make agriculture less labor-intensive and bring prices down.

load balancer failure? rate limit woudl address concurrency limits? throttle or queue up messages. auto-scale failed cause was misconfiguration policy, which i admit cannot be handled by my suggestions. The cascade? it's downstream service degradation, which I mentione should have had been prevented with queues. One of the jobs that queues/kafka solve is to prevent these downstream outages.

you have to opt in to private contributions being visible like that