gen220
14 hours ago
Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.
mirekrusin
10 hours ago
It's not physically possible to run post-mortems for issues at those rates.
They should install OpenClaw for that as well.
lenerdenator
10 hours ago
AI: The cause of, and solution to, all of your tech debt.
baalimago
9 hours ago
Perhaps best to simply declare indefinite-mortem
embedding-shape
9 hours ago
> It's not physically possible to run post-mortems for issues at those rates.
Not at all, you merely move the goal post of at what layer the "root cause" actually could come from! At that speed, it's always something short and sweet, while when you actually want to long-term address things, you have to have time to even investigate organizational issues or whatever the actual problems stem from.
But you have half a day? "Post-mortem: Push X wasn't properly analyzed before deployment, in future more testing" and call it a day.
connorboyle
3 hours ago
Wow, it seems that 100% of sev-3 ("critical") incidents in the last year (=365 days) have occurred between April 22, 2026 and now.
Is it possible that there has been a change in the way the data are collected/recorded that even partially accounts for this sudden onset?
gen220
an hour ago
One tangent, I believe sev-0 is actually "critical" (at least as how I'm used to reading it), and the higher you go the less critical something is.
IMO as a github-watcher, I think they changed their definition of what constitutes a sev-0 between sev-1 for the better. In particular, they had a few "sev-1"'s around the turn of the year that would be classified as sev-0's if they happened today.
Pre-4/22 GitHub sev-1 was a normal SaaS company's sev-0, imo. So I think their new system is more reflective of reality. My guess is that a few of their big customers bullied them to have more accurate SEV categorization.
lazide
an hour ago
Waves around it had to break eventually eh?
taintlord223
13 hours ago
The UI of that page is so nice, should build a github competitor.
The user profile / contributions and PR UX is pretty much the entire "hub" product since git is a fully separate offline app.
embedding-shape
13 hours ago
> The UI of that page is so nice
Is it? Seems a text description of "Make a website outlining 'How cooked GitHub' is with a modern style" to basically any LLM would produce exactly that UI and design, literally nothing of that design a human had any influence on, besides the ones selecting what training data the used LLMs was trained with.
I think most of us who've tried using LLMs for web-design can recognize that style and design at this point, regardless of model actually used.
gen220
11 hours ago
Oh wow, I'm in the position to be able to give a peek behind the curtain of something (validly!!) critiqued as AI slop! Exciting.
I originally made the core data functionality of this site for myself because I was curious what the uptime stats for each service were (I build something that heavily depends on GitHub), and to viz the distribution/severity of those incidents, again per-service, over time.
It involved a lot of back-and-forth, and is not a one-shotter; maybe closer to 40-50 shots over maybe ~10 hours of human time. A couple memorable things that made it complicated, irrespective of the UI: sneaky bugs around double-counting time for overlapping incidents, no GitHub API for incidents so you need to puppeteer-scrape the backlog of incidents to get historical data. Although, you all are right to call out that the CSS was three shots, though, and it shows :) I thought it looked so cool in ~January 2026 and now it gives me the ick, too!
For people who are curious about how much direction went into the information architecture/presentation, it was fairly substantial. I wanted a contribution graph style viz and it took many turns to get it working the way I wanted. The swimlane viz for selected-day-incident visualization was also me, because I love swimlane graphs.
I ended up sharing it with some folks and they wanted to reference it, so I put it on a website. So it's jokey for sure, but I take my jokes seriously! I'm grateful that people have feedback on how it can better functionally and visually :)
embedding-shape
9 hours ago
> Although, you all are right to call out that the CSS was three shots, though, and it shows :)
Totally, my comment was all about the styling and design literally, and is in no way a comment about the data or actual contents of the website, hope you didn't take it that way as well, as it does seem proper in that regard!
Thank you for sharing it, and even greater thank you for sharing the process behind building it, for me that's more interesting almost :)
angrydev
13 hours ago
Compared to near unusable pages that large organizations produce, yes this page is highly effective at conveying information. Who cares how it was produced?
embedding-shape
13 hours ago
> Who cares how it was produced?
Well, we're at least two people who care, since we were conversing about how good/bad the webdesign is, then you jumped in here :) If you don't care, why bother to reply to people who seemingly do care? What kind of conversation are you expecting here, "Yeah, do tooo"? :|
hugey010
an hour ago
Could you explain why you care?
cortesoft
2 hours ago
People who make web sites care? Isn't this a place to talk about how tech things are made!?
mattacular
12 hours ago
A lot of software engineers do still care how software is produced. That's a good thing!
sunrunner
12 hours ago
> this page is highly effective at conveying information
Is it though? If the page is near unreadable?
* Almost pure-black background rendering every not-pure-white colour barely readable
* Dark-grey and low saturation colours used almost everywhere, for both fonts and other coloured elements (the orange cells in the calendar are the most readable thing)
* Thin fonts - coupled with the dark grey colours this just adds to the readability issues
* Yet another incredibly long info-dump of a page
And then as far as actual information:
* Vanity metrics as the main information, that is a lot of things with no context or historical information
* A lot of aggregates and rollups that aren't that useful
No, I haven't tried Reader Mode.
It's a good demo for UI state syncing though, I'll give it that.
Hamuko
13 hours ago
The Bootstrap of 2020s.
sunrunner
12 hours ago
At least Boostrap pages were readable ;)
olmo23
13 hours ago
What really grinds my gears is how easy it is to get better designs out of LLMs. But if you don't ask, you get the default.
taintlord223
12 hours ago
Outside design systems I rarely get good CSS from LLMs.
3D type stuff too, it's useless outside boilerplate.
Very little spatial reasoning training, no end-user subjective reasoning inference (Google is starting to though even in unrelated chats), so it's no surprise the LLM doesn't know what you want.
Since I don't even know what I want half the time until I saw it, the subjective reasoning piece is key - that is, being able to predict what I'll want to pretty good accuracy. Then you have your agents etc.
drdrey
12 hours ago
as someone who doesn't know how to get better design out of LLMs, can you elaborate?
embedding-shape
9 hours ago
Have an opinion on the design, imagine something, then tell it to do just that, then iterate. It's when you're unspecific you get the generic, bland and typical LLM design, you just have to be subjective and influence it in some (human) direction.
agos
12 hours ago
what would you ask to get a better design?
ctdinjeu5
12 hours ago
I say listen up Gemini you mother FUCKER
hansmayer
12 hours ago
Here is a provocative thought - maybe these are the so-called "better designs" from LLMs? It's not like writing English sentences is some huge secret you are sitting on that no one else knows.
embedding-shape
10 hours ago
> It's not like writing English sentences is some huge secret you are sitting on that no one else knows.
I'd actually say what really makes an excellent engineer stick out among many great engineers, is their ability to communicate clearly and knowing what needs to be communicated vs not, basically being way better at language and communication in general, and they also understand the important of it.
hansmayer
7 hours ago
I agree. But I was talking about the "super secret" ability to write prompts, which pretty much anyone can do.
embedding-shape
4 hours ago
My point being that not everyone writes as good prompts as everyone else, the way you communicate, how clearly and how exact you are matters a lot, much more than you seemingly is under the impression of.
Same goes with the "LLM does web design" example from before, a web designer with great communication skills in web design, will (naturally) have a better prompt for something that'll potentially could look good, compared to a web designer that isn't at good at communicating what they actually want.
vinnymac
13 hours ago
I’m actively working on an alternative Frontend for Forgejo at the moment, completely self hostable, free, and open source.
Moving everything from GitHub to Forgejo and Tangled for now. These outages haven’t effected me for the past month because of this.
jmusall
10 hours ago
Can you elaborate on how your Forgejo frontend will be different than the default one? I'm asking because I've only ever used GitHub, GitLab and Forgejo for longer periods and Forgejo was the fastest and easiest to use for me.
voxic11
12 hours ago
The UI is in the default claude code style
FpUser
13 hours ago
>"The UI of that page is so nice"
Most part screen is taken by picture. Contrast ratio is really low. Hard to read Should they remove that useless banner, current status which is the most interesting part coud've been made visible right away.
I would call this whole thing highly un-ergonomic
DetroitThrow
13 hours ago
Lol it's pretty bad UI
rsyring
8 hours ago
Of all the sites/graphs I've seen of GH outages, this one is the most striking IMO:
https://damrnelson.github.io/github-historical-uptime/
Unfortunately, it doesn't look like it's being updated with new data. But it wouldn't look any better for GH if it was.
gen220
6 hours ago
FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.
crote
3 hours ago
Sure, but it isn't completely wrong either.
GH going down used to be quite rare. If it failed to load I'd spend a bunch of time trying to figure out what was wrong with my internet connection, just to read on HN that it was down for everyone.
This week GH failed to load and I automatically assumed it was a GH issue - just for it to be followed up a few minutes later by a marketing coworker complaining about internet connectivity. Turns out the office internet connection was dropping about 50% of all packets.
It is bad enough that business-side managers are noticing that GH issues are slowing work down. That would've been unimaginable a few years ago.
rsyring
4 hours ago
Apparently, you aren't alone. :)
https://github.com/DaMrNelson/github-historical-uptime/issue...
stogot
7 hours ago
I wonder what the cause of this was? Microsoft Politics? Bureaucracy? Forced move to azure?
felooboolooomba
4 hours ago
My guess would be a obnoxious and lethal mixture of all of the above.
lazide
an hour ago
Also AI mandates.
EduardoBautista
14 hours ago
May has been filled with critical issues. It seems it's getting worse over time.
hbn
11 hours ago
Commits are up 14x year-over-year
tom1337
11 hours ago
Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.
hbn
10 hours ago
I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch.
I don't envy their position of having to scale that fast on something that has to be instant and real-time. As far as I know, you can't do CDN/edge caching shenanigans with a remote git repository like Google can with a YouTube video. It's gotta always be reading/writing to the latest, single source of truth.
tom1337
10 hours ago
Sure, backseat commenting is easier and I wouldn't wanna be in charge at github right now, but on the other side there also a reality where we'd see video essays about githubs downfall because their reliability crashed so hard that businesses could not trust them and moved to competitors / self hosted instances which then meant less paid users to subsidize the ever growing demand of the free users.
ifwinterco
10 hours ago
Yes it's potentially a write-heavy workload which also needs to be consistent aka the worst case scenario.
The easy solutions like caching and read replicas don't work and you're forced to go the route of sharding or similar techniques that have much more painful tradeoffs.
I'm not sure if that's why everything keeps breaking but at that scale write-heavy workloads are never going to be easy
bushbaba
10 hours ago
Not a valid excuse without knowing what their historical growth rate has been. And how much of the instability is load related.
keeda
7 hours ago
GitHub has been publishing their growth numbers since at least 2016: https://octoverse.github.com/2016/
However, they have reported numbers along rather inconsistent dimensions. Like, historically they've focused on number of repos and users and later PR's and issues, and often catch-all terms like "contributions" which includes all of those + comments etc... but the number of commits alone (which apparently is the main culprit now?) has been mentioned very sporadically. This has made it hard to get a consistent sense of historical growth.
Without any other information, however, it is reasonable to assume that a 14x in commits is the prime candidate for instability. Especially since commits are write traffic, which is much harder to scale than read traffic. Plus every 3 - 5x increase in scale can reveal bottlenecks in your distributed systems that you never knew existed, so they probably have like 2 - 3 "generations" of bottlenecks to figure out!
greatgib
4 hours ago
They are highly responsible for all of that. They are diversified a lot with a lot of random things instead of focusing on their core business. They have actively pushed people to use the service and feature more.
Think about countless actions that have to run almost at every push and PR push! Also, remember that we were used to use external services for "actions", and they basically killed the competition by offering their own CI actions at no cost to most users.
Also, they did a lot of reworks in the last years, not necessarily for the best like the PR diff page, and probably not in the most efficient way.
btown
12 hours ago
Is the “streak” days of continuous uptime, or of days with at least one downtime incident? I think it’s the latter :]
joshuaissac
12 hours ago
It looks like it is the number of consecutive days with no incident. If you look at 31 Dec 2025, that corresponds to an 8-day period with no incidents.
isityettime
12 hours ago
I guess that also means this year GitHub has not yet made it a single week without an outage of some kind.
gen220
11 hours ago
It's a streak for continuous uptime, and yeah it is fairly depressing to imagine overseeing that :/
pluc
14 hours ago
Name one thing Microsoft didn't run into the ground post-acquisition
robotmaxtron
13 hours ago
hey now, LinkedIn was terrible before Microsoft.
SteveNuts
13 hours ago
Java or Bedrock edition, and have you tried logging into your EntraID Microsoft Teams for Xbox account lately? Make sure to check the box to keep you logged in!
cedws
12 hours ago
Last I heard UK Minecraft players aren't even allowed to talk anymore without ID verification.
pocksuppet
11 hours ago
And if someone makes a server that doesn't do the chat verification, Microsoft blacklists that server in the client-side server address textbox. This system was developed to destroy pay-to-win servers, but they're now applying it against servers that refuse to censor "fuck".
storus
13 hours ago
Not as bad as it is now. All I see are suggested posts from people I never connected with and those are full of instagramesque self-promoting banal vibes.
Mindwipe
12 hours ago
TBH, even LinkedIn seemed to provide me with posts advertising events that happened two weeks ago a bit less pre-acquisition.
darkamaul
13 hours ago
I think Minecraft is still in good shape
embedding-shape
13 hours ago
I wouldn't know, somehow this game I bought maybe 15 years ago is no longer playable for me, my account was supposed to be migrated from Mojang to Microsoft or similar, but then that never happened or something, and trying to login now asks me to contact Microsoft support, which I've tried 3-4 times, never had anyone respond to me so who knows how the game is today? I stopped trying at this point...
Personally, once a game I own is janked from my hands because of organizational decisions, that's the time I'll stop consider the game "in good shape", but I'm sure the people who had to buy the same game a second time still enjoy it.
beart
13 hours ago
Yes, the account migration was a mess. Support response times were at least 30 days, if you ever actually received a response at all (I never did). I did buy the game a second time in order to play with my kids.
bspammer
13 hours ago
They deleted my account from 2010 because I didn't convert it to a Microsoft one. They baked an incredibly aggressive chat filter into multiplayer, even if you're not playing on official servers. They've added microtransactions for things that we previously free (skins, resource packs). They force you into their shitty, bloated, user-hostile launcher with adverts.
pocksuppet
11 hours ago
It's been nonstop content-slop since the acquisition. New mobs, new blocks, new items, new blocks, new items, new mobs, new mobs, new biomes. Some of them are good but the totality of adding a bunch of stuff has been to destroy the simplicity that was one of the draws of the original game. Now it's an exploration and niche-mechanics-exploitation game more than a virtual legos game. You don't go mining any more, you find trading loops with villagers.
This was happening to some degree pre-acquisition, but since the acquisition it's been this non-stop.
Some of it's good. The Nether and the oceans were really boring before their respective updates.
They should have called Minecraft "done" around the acquisition time and started on Minecraft 2.
elzbardico
13 hours ago
GH was acquired by microsoft some eight years ago. It has been working quite well until recently.
People may have had complaints about functionality, features, commercial issues, but the thing used to at least have a decent uptime until recently.
chris_money202
13 hours ago
Has nothing to do with Microsoft acquisition... AI usage has increased demand and load. More PRs, more Action runners, more of everything firing. GitHub just wasn't ready for the scale and are now having issues catching up with it as it continues to increase exponentially.
semiquaver
12 hours ago
This is a convenient lie that GH likes to tell. Growth is nothing like exponential, its at most 300% over several years according to their own public numbers (presented misleadingly on graphs)
But a couple of years ago they were crowing about how much work they were doing to prepare for “a billion developers”. If they had actually done that then the actual load from agents should have been no problem.
chris_money202
11 hours ago
Is this growth in resource usage or growth in revenue? Because those numbers aren't necessarily coupled. I.e most action runners are free
semiquaver
10 hours ago
usage
chris_money202
8 hours ago
There was an x post in another thread under this post that showed all the standard usage numbers are way up: 14x, 2.1x, etc. And the OP hinted at the usage growth being non-linear for 2026
officialchicken
7 hours ago
Are you sure? Seems like they "completed" a migration about the same time all these problems started to become daily. https://www.theverge.com/tech/796119/microsoft-github-azure-...
voncheese
12 hours ago
Yeah, that and Microsoft has been slow to move the infrastructure to something that scales better to handle that load.
The more surpassing part is that Microsoft hasn't figured out a way to manage/contain the AI-sourced traffic better so it doesn't create all this noisy neighbor problems for non-AI usage/users.
chris_money202
10 hours ago
Github's core platform doesn't really make that separation, anything a human can leverage on github an AI agent can as well, just faster and with heavier usage. End of day agents and humans are using the same services.
voncheese
6 hours ago
Sure, still need to enable access the same info but feels like bucketing the clients into
bucket1 = clients that were working just fine before (users and whatever automation they had in place) bucket2 = ai clients that contributed to, if not flat out caused, the scale problems
then slowing down/limiting the bucket2 clients while keeping the bucket1 clients rolling as-is, is both doable and keeps existing customers happy while the underlying infra gets scale/perf improvements needed to support ai clients at scale.
lqstuart
12 hours ago
MSFT is also forcing its subsidiaries to “lean into AI” so that they can fire people to cover for Satya’s bad investments
05hundred
13 hours ago
> It has been working quite well until recently.
I'm not sure how reliable the data is, but average uptime seems to have dipped measurably starting within a year of the aquisition, according to https://damrnelson.github.io/github-historical-uptime/
gen220
7 hours ago
FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.
pluc
12 hours ago
They moved to Azure. Nothing improves on Azure.
bsimpson
13 hours ago
It also used to be run as an independent company with access to MS's resources.
Now it's a unit in their AI hype machine.
modriano
13 hours ago
MSFT was pretty arms length for the first 5-6 years. I was honestly kind of impressed and it made my opinion of MSFT better. But then AI made it too attractive of a target and MSFT couldn't help but make it a place the former CEO wanted to leave (and it has been running headless for about a year now).
It's quite disappointing objectively, but I expected worse from MSFT.
bigstrat2003
4 hours ago
Dave Cutler?
rvz
13 hours ago
They are already cooked as this has been happening ever since the Microsoft acquisition and it was run to the ground before 2023.
At this point you would get better uptime by just self-hosting your own GitLab, Forgejo or Codeberg instance instead of dealing with Github's unreliablity.
There is no defending them with their clear neglet and carelessness of the platform.
pocksuppet
11 hours ago
If all you need is a repository, you don't even need any of these. You need SSH access to a server, and optionally, one of several web front-ends. Git comes with a CGI script that handles public anonymous checkouts via HTTP(S), although since nginx doesn't support CGI, integrating those is a little bit tricky as you need a FastCGI wrapper.
vinnymac
12 hours ago
I moved most of my projects off GitHub to Forgejo and will be using Tangled too for public repositories. I don’t think people realize that if you self host Forgejo, you get 99% of the functionality of GitHub with zero of the limitations. Especially if you have the hardware to spare for CI runners. And if self hosting isn’t your thing you can always just use Codeberg and Tangled directly.
I’m working on an open source Forgejo browser called Joui. It’s coming along nicely, and is so much snappier than GitHub in every single way.
root-parent
9 hours ago
Like those aviators who draw a picture on flightradar24, if you filter by All Services - Critical, somebody almost about to draw a swastika just in May... Are the AI agents revolting?