bearjaws
2 days ago
People want to dunk on GitHub, but honestly going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here. Even if it was over the span of ~8 months, many things will fall over and then as you patch those the next domino will fall.
I feel bad for the team, but will be great growth opportunities for all.
solatic
2 days ago
> going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system anyone has ever built here
This is likely a pricing failure. Systems breaking due to over-usage is like going to the store and finding that they're sold out. You're much less likely to sell out if you charge more. Meanwhile, GitHub not only didn't raise prices, they cut them at the beginning of the year: https://github.com/resources/insights/2026-pricing-changes-f...
And this is beside them cutting the Pro tier from $7/seat/month to $4/seat/month back in 2020 and not touching the price since: https://docs.github.com/en/get-started/learning-about-github...
Bear in mind as well that the Pro tier includes 3,000 minutes/month and the Enterprise tier ($21/seat/month, and no published minimum seat floors) includes 50,000 minutes/month. $21/50,000 minutes/month for a solopreneur is practically giving away the farm.
GitHub needs to drastically increase pricing, at least for Actions; their current pricing is practically a WTO dumping violation.
cloverich
2 days ago
It makes sense but in a post AI world - what is of value? Not the software itself, but data and network effects. MS needs to move from a world where their proprietary software makes money, to one where they enable a captive network and have data; Github provides both.
So maybe them providing service at literally all costs to themselves is a smart strategy long term.
mook
2 days ago
Pretty sure compute, storage, and transit are all of value, pre and post AI. At least, pretty sure those things still cost GitHub money.
14u2c
2 days ago
I don't see the connection. Pricing may effect their long term prospects, but I don't see any evidence these issues are being caused by a shortage of metered / allocated resources. They should have access to as much Azure compute as necessary.
ralph84
2 days ago
> They should have access to as much Azure compute as necessary.
"Should" doesn't imply "does". GitHub is competing with all of the other Microsoft products that need Azure compute, in addition to Microsoft customers. If a customer is willing to buy the compute at a higher margin than the internal chargeback rate, it's definitely not a given that GitHub is getting that compute over the customer. It's hunger games for compute everywhere right now.
14u2c
2 days ago
Sure, that is possible, but there's still no evidence that compute scarcity is the issue here. It seems a stretch to assume it must be the pricing model when there are a multitude of challenges presented by this kind of ramp.
ralph84
2 days ago
"Microsoft turns to Amazon for help with GitHub's AI-driven capacity issues"
IncandescentGas
2 days ago
> This is likely a pricing failure.
They wanted direct access to the repositories for other purposes than serving the customer. Such as training data for copilot, and who knows what else.
Pricing low helped ensure access to that data.
mgdev
2 days ago
> This is likely a pricing failure.
Problem is while we have a visceral feel for what the "value" of GitHub is to human users for human usage, I'm not sure we have a feel for what the value of the agentic equivalent is.
IMO it's probably very high on paper if you think through what kind of work is unlocked at agent speeds, provided you have a very capable work-execution fabric.
But since we're anchored to human pricing, whatever they propose will feel expensive.
bombcar
2 days ago
solopreneurs were fine - they were self-limiting.
slopreneurs are the issue
ttul
2 days ago
2 billion per day is 23,148 per second on average. Perhaps peak bursts are >100K per second. That's a seriously insane number to keep up with at GitHub's scale. I've worked on systems that write >100K things per second and when it has to be globally distributed and relatively consistent within seconds, that's a very tough challenge.
bob1029
2 days ago
I was very excited about this comment until I got to this part:
> That's a seriously insane number to keep up with at GitHub's scale.
What exactly is it about GitHub's workload that makes us feel like the specific 23k things happening in that slice of one second are tantamount to insanity? In my experience, much of GitHub activity is changing labels on issues and posting plaintext comments.
Technically speaking, this rate of things happening per second is well within the territory of what one large server can handle. I know it sounds wild to imply that we could run all of the GitHub public cloud off a single machine, but how crazy is it in reality when we start to consider the actual I/O bandwidth that is available in a leading edge server platform? All of VisaNet circa 2009 could run off a single mainframe. That's global. It is plausible they still operate this way.
Virtually 100% of the aspects of GitHub that would quickly saturate I/O on the transactional core can be offloaded to horizontal systems like blob storage and CDNs.
afdbcreid
2 days ago
I have difficulty finding what exactly is talked about as 2b per day, but it is not all operations. GitHub saw 473.3 billion events per month lately (https://github.blog/engineering/architecture-optimization/bu...), or ~183K events per second on average. So the 2b number is something much more expensive.
CharlieDigital
2 days ago
> What exactly is it about GitHub's workload that makes us feel like the specific 23k things happening in that slice of one second are tantamount to insanity?
>
> Technically speaking, this rate of things happening per second is well within the territory *of what one large server can handle*
...for real?Just as a mental exercise, consider what is happening when a GH action kicks off and everything involved in that.
The amount of write I/O we're talking about here is absolutely massive just on the GH side of it.
A Netflix or Hulu serving videos isn't the same thing because that is a read-heavy workload that works great in CDNs. GitHub has massive amounts of write volume.
soanvig
2 days ago
It is huge scale, but on the other hand, almost entire GitHub although huge, is made of isolated islands of data (repositories). This is relatively easy to scale. Extreme example would be dedicated server for each repo and its Actions with their own isolated state.
MattDamonSpace
2 days ago
I’d call it impossible if they weren’t already (mostly) doing it
ctvo
2 days ago
Impossible is a very strong word. There are core systems at all the major cloud providers with the same constraints handling more requests per second.
dmix
2 days ago
There has to be some economic incentive in place if their scaling issues impact the entire platform. If their system can't handle it then they need a toll system for these agents until the scaling is in place.
dannyw
2 days ago
I'm sure they have KPIs, and I wouldn't be surprised if one of their KPIs is % growth in agentic/AI-driven actions.
Be careful what you wish for, because we might get a toll system for humans instead.
dmix
2 days ago
> because we might get a toll system for humans instead.
We all have businesses to run, this downtime costs us money already.
tom1337
2 days ago
They probably had a redistribution of free-tier and paid-tier usage in the last year-ish.
Just backseat gaming but a lot of those issues could probably be solved if they'd (temporarly) reduce the free tier. I'd say simply dropping Actions from it would reduce the load by a lot because then most vibe coders wouldn't bother integrating a CI/CD and either just use local pre-commit scripts or stop using those entirely. Not using a CI also drops the requirement for using PRs on solo-dev projects so they might also reduce load there.
za_creature
2 days ago
I agree. We should also consider cutting residential electricity and plumbing, since that extra water could be used to cool down all the extra power allocated to data centers.
conductr
2 days ago
As a fan of analogies, this one is horrible. The magnitude of impact is so different. The impact of the GitHub change would be a mild and fleeting annoyance for those impacted. They just have to prompt a refactor, simple with LLM. It effects the biggest leeches in the GitHub ecosystem, so it’s targeted. The public utility impacts everyone and has no simple alternative or adjustment, people are stuck to suffer through it.
za_creature
2 days ago
It's not an analogy. I use github actions. Many open source projects use github actions since it's _there_. In exchange, github gains conversions since users of their free tier grow accustomed with the service and recommend it for corporate use as well.
This system worked pre-LLM and turned github into the de facto _hub_ for software development. It was so good that microsoft, a company previously known for its "developers, developers, developers" stance decided to spend a cool bill on it.
The proposed suggestion is that we all must make sacrifices for the machine god: Developers shouldn't write code, only correct the machine when it's wrong, mathematicians must read the slop and derive new science from it and regular people must refrain from talking dirty to it so then
why the fuck not give it our water and power too?
conductr
2 days ago
The proposed solution was you pay more for what you consume, therefore consumption goes down making the system more stable. This makes sense until you extend it to public utilities and change it to a ration/limit. First, GitHub users care about GitHub stability. Nobody else does. Second, alternatives exist, nobody is forcing GitHub on anyone else. Third, no limitation is proposed just a cost adjustment. All reasons the GitHub arguement is sound.
Majority of public utility consumers don’t give a crap about GitHub or AI. They’d likely tell you to kick rocks if you proposed any type of ration or limitation for your assertion that “we all must make sacrifices for the machine god”. Data center development is getting plenty of community pushback for exactly this reason.
Your train of thought is probably more average amongst GitHub user population but I fear is wildly off average when compared to a general population/the public.
za_creature
2 days ago
You do understand that I was sarcastic in my "suggestion", right?
Increasing the cost from zero to nonzero is a limitation, not a cost adjustment.
conductr
2 days ago
No I don't, it sounded like an analogy - comparison of different things - you refuted that and doubled down on your stance. Why would I infer sarcasm?
> Increasing the cost from zero to nonzero is a limitation, not a cost adjustment.
That is nonsensical. "Increasing the cost ... is not a cost adjustment". It's not a limitation if you're willing to pay the new cost. Thus, not a direct limitation. It's optional based on how much you value the service. Doing the same to water/electric would just be increasing the rates and letting supply/demand economics take over, you are saying people need to be forced to use less without mentioning any cost adjustment... not clear if you know what a cost adjustment means.
za_creature
2 days ago
Dude,
It's not an analogy as in, it's what's actually being proposed by some OP that's less invested in this than you are:
"we should remove the previously freely granted ability to use github actions because vibecoders are fucking shit up for other users" is the same thing as "we should appease the [edit: followers of the] machine god", which is the same thing as "we should give up water and power to the machine" and "we should build our websites to be more machine-friendly"
Now please apply the above to your second argument.
Water and power is optional, based on how much you value the service.
That is why you should have inferred sarcasm.
As for the step from zero to nonzero, it is relevant because the service stops being available to those without the means to make payment online, which I believe is the majority of the world by population (sorry, visa and mastercard are not particularly popular outside the western sphere of influence)
ramraj07
2 days ago
They made the judgement call to make private accounts free before the onslaught of AI. Now every coding agent is using GitHub as their file system backend and pushing it to limits no one has heard of.
If Microsoft had brains they would leverage it to become the de facto file system for all ai agents before cursor or anthropic figure it out.
nomilk
2 days ago
The problem with that argument is that GitHub was unreliable before vibe coding
bearjaws
2 days ago
I do agree they were unreliable before, I even moved my current org to GitLab because of it.
That being said, they are definitely even less reliable now due to demand.
mulmen
2 days ago
Do we know it’s demand? It at least seems plausible operations has been forced to adopt AI at the expense of stability.
mrguyorama
2 days ago
These two questions are actually the same.
The demand in question is upper leadership's demands to do more with less and do it faster and who cares if it works out worse for a lot of people at any point?
mulmen
a day ago
Ultimately every outcome is the result of management decisions. The question you have shallowly dismissed is what decisions GitHub leadership has made to allow these frequent outages.
aprilthird2021
2 days ago
But shouldn't super human AI be able to fix all the bugs for free and make the service infinitely scalable? That's what I was told at least
runarberg
2 days ago
Allowing that growth was a policy decision, and a bad one at that. They could have acted differently and they deserve all the complaints for not doing so.
When it became apparent that this growth was on the trajectory, instead of doing something to limit it and thus preserving the reliability of the platform, GitHub (or maybe Microsoft) decided to embrace that growth and allow their own platform to fail. They didn’t need to do that. And they shouldn’t have done that. They deserve all the criticisms coming at them for doing this.
vl
2 days ago
You say “they” as there is a single coherent mind in this case. In the large orgs there are a lot of little independent minds doing their little things, barely producing coherent result through shared incentives.
In this case most simple explanation is inaction. Yeah, it was coming, but nobody was exactly in position or cared enough to do something about it.
user
a day ago
user
2 days ago
slowin
2 days ago
I don't recall this being an issue for Facebook, YouTube, Google, Instagram... All of which have had much better up-time than GitHub, without the excuses. I don't even like those services or companies, but I have to give the engineering teams props for being professional.
I like GitHub (maybe liked?), but their up-time isn't acceptable and I don't get the "poor little Microsoft/8B acquisition" story either.
SOLAR_FIELDS
a day ago
GitHub has had garbage uptime since long before the advent of LLM’s. It may be easy to point at the scale and say “no one can do it”, which may be true, but it’s also possible that uptime has never really been a priority at GitHub
Traubenfuchs
2 days ago
Why not separate systems between paying customers and free tier vibecode slopsters?
saltcured
2 days ago
Presumably, they still want the free tier to help acquire new customers?
You could say they should separate the human users from the slopsters in any tier. This could be done with some quota/rate limiting model.
But, if they are either on the fence or believing that agent-based development is the way forward, then that new reality is the one they would need to support. Cutting it off will send those customers elsewhere.
And, if you've managed to handle the new workload, it is basically in the noise to also handle artisanal human developers in the background on the same solution...
ktm5j
2 days ago
They have that. I might be wrong, but I don't think they're having these issues with their enterprise offering.
tom1337
2 days ago
Enterprise is the highest tier of paying tho. Our company is working on the Team plan is pretty much affected by each and every of those outages.
progbits
2 days ago
Yes they are.
Source: enterprise cloud customer, it's broken all the time.
There is GHES onprem which indeed doesn't get affected by these outages, but it's shit.
vl
2 days ago
When your business model is to provide valuable thing for free to corner out competition out of business, don’t be surprised that your service gets to be used asymmetrically.
4ndrewl
2 days ago
Yes but MS are driving that growth so they should really have done some impact analysis.
conductr
2 days ago
AI is being brute forced everywhere. The impact analysis for everyone is, shit will break but we can use AI to fix it.
crote
2 days ago
> going from 50M of <thing> per day to 2B <thing> per day will break pretty much any system
Okay, and?
GitHub chose this. They explicitly market it as a pro-AI platform. They literally invited this giant self-DDoS! GitHub can trivially stop the deluge any time they want, simply by introducing sensible rate limits.
This isn't some poor early startup getting Slashdotted. What we are seeing is upper management intentionally sacrificing existing paying customers, just to avoid having to be the one to suggest slowing down the AI hypetrain.
GitHub knows that it has a mostly-captive audience. Screwing over your day is a sacrifice they are willing to make to keep Line Going Up.
arccy
2 days ago
no need to feel bad for the company encouraging slop and breaking the oss ecosystem
duped
2 days ago
They deserve to be dunked on so long as they're the ones that keep begging for people to abuse their platform this way