As a joke I built an extension to the GitHub CLI called “omens” so you can run “gh omens” before you’re planning to use GitHub and it’ll tell you if it’s likely to work in idiomatic Aussie.

As an example:

$> gh omens

reckon github's about to go full bin chicken

—-

You can find it here: https://github.com/sandermvanvliet-stack/gh-omens

A browser extension to indicate githubstatus being yellow/red:

https://chromewebstore.google.com/detail/is-github-down/lcfo...

A VSCode extension:

https://marketplace.visualstudio.com/items?itemName=RuslanRy...

Firefox:

https://github.com/matagus/github-status-checker

Caveat: https://news.ycombinator.com/item?id=49450924

(disclaimer: none of these are recommended by me to use - merely sharing to build on for this discussion)

For me this started happening when I type "stat" too.

Which surprised me because of the many hundreds of services I rely on regularly that also have status pages.

I can tell it’s getting bad because I went to close some Safari tabs and got confused because the tab next to this was [a dupe of] https://news.ycombinator.com/item?id=49330597.

I notice I sometimes wake up before my alarm because I get an SMS about GitHub being down.

It could be a new Microsoft feature !

i noticed it today as well - typing "g" is enough to go to the statuspage. and surprisingly often when i go there by accident, they have an incident…

[flagged]

One of my favorite articles posted here for this year, it's a great read and really gives you an idea of just how dysfunctional azure is.

Azure's going to suffocate github. I'm curious to see what's next. Will self-hosting the code repository come back in vogue or will another social-coding platform take off?

I've heard that companies that run enterprise Github instances on-premises have almost equally poor availability compared to github.com. There are probably plenty of reasons why they have so many outages, though I agree one of them is probably Azure. The outages started around the same time as the number of pushed commits skyrocketed, and the number of Dependabot vulnerabilities started growing as well, which in its turn increases the amount of PRs and Actions activity. Also, LLMs regularly import dependencies that are already out of date, meaning that when doing agentic development, one commit often results in several Dependabot PRs being created instantly.

"fun"

I thought I was cynical enough about goings on at Microsoft / Azure, but apparently not.

> Probably just a coincidence that Github started to have issues after beginning their move to Azure at the end of last year.

Right.

I'd like to think that Azure has improved in a meaningful way in the past half a decade or so... but it has not. Maybe less inexplicable 400 errors in random API calls, I guess.

And yet Microsoft keeps posting record growth. Unbelievable.

Keep in mind the seniority of the author. This was not written by a staff+ engineer.

Microsoft and Oracle have the market cornered on selling stuff to corporations who are relaxed about whether it works.

Sure but first you gotta be Microsoft.

They have that-ish as an option: https://docs.github.com/en/enterprise-cloud@latest/admin/dat...

I'm told that GitHub has asserted to us that moving to this model means we would not be exposed to github.com outages. It's not at feature parity with github.com though.

According to their status pages (e.g. https://eu.githubstatus.com/, https://us.githubstatus.com/), their Enterprise Cloud uptime for Actions is significantly higher.

That's what I don't understand. They could mitigate their name so much if they just split free/paid/enterprise. It's already shown that enterprise is much more estable and is largely unaffected from service disruptions. Why don't they go one more layer? For sure it's worth the extra complexity.

That's what Azure DevOps is supposed to do, but for some reason GitHub has a redundant enterprise division.

surely if they did that everybody would complain how github "lost its touch with open source since they now prioritize paid services"

enterprise is mostly separate, is it not? uptimes are significantly more reasonable on the enterprise status pages

It would probably be better to run projects with extremely high commit/merge frequency on a separate "slop infrastructure", basically like MMOs move cheaters to their own servers ;)

I think the authors of SMTP had a healthy attitude towards server uptimes:

   Retries continue until the message is transmitted or the sender gives
   up; the give-up time generally needs to be at least 4-5 days. 

I don't think it's been "normalized". Github uptime is literally a joke in the tech community. They have first mover advantage and a behemoth behind them so they're not going away, but everyone knows how shit it's uptime is. It takes time for organizations to move away from services like this but I would bet anything that many are starting to try to move away, as well as new companies knowing they shouldn't use the service.

Vitess is a distributed mysql database. Github could very well be managing it entirely on their own. I have only seen people managing their own vitess, its entirely open source afaik.

I read that as an upstream service they own, but I agree the wording a bit weird.

Why is that a problem if it _is_ an upstream vendor problem? (assuming it is)

At this point, maybe it'd be more appropriate for people to post about github to HN when githubs actually working.

https://www.dayswithoutgithubincident.com/

> Update - primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues

Good for you. These kinds of migrations never feel productive at the time, but the right tools can make your life and work so much better.

Sure. The problem is just an easy fix that nobody there thought about yet.

realistically the problem is vibe coding.

Obviously we can because we are choosing to. Because servers are scary.

Apparently we can because a lot (most?) of us are still using GitHub even after all their outages recently.

Except nobody moves to a privately hosted "gitweb"...

2.9B commits per month; 100M action runs per day; I think they probably have some sharding.

I wonder what is this database, and why it is hard to fall-over automatically.

Did you not read it? Just because there's a database primary doesn't mean there is 1 primary database. There's likely man redundancies and they have issue with how they're allocating traffic to them which is in turn causing an issue with how much traffic redundancies are receiving.

Why shouldn't it? Most companies run on a single database server. If they can immediately fail over to a replica, that's doing it right.

Maybe you expect that part of GitHub to have a scale where a single database can't handle it, but evidently that isn't true.

We can criticise them for not splitting up free and paid customers but again, most companies don't do that.

It very well may be that the paying customers are actually the problem. Those can actually afford to throw enough money at AI providers to significantly increase the number of commits & other actions.

The status page is the last one to get the update. Reddit, HN, X, company chat are all always reporting it sooner.

I started the campaign to move us from Bitbucket to Github before the Microsoft acquisition. If I knew MS would be involved, I'd have campaigned for something else. I would never recommend any Microsoft product.

It took ages for us to get permission and licenses. We only recently finished moving the last repos over and shut down Bitbucket.

If AWS offered a product like GH, I could probably unilaterally start moving to that. Any other alternative would take 3-4 years of meetings, budgets, lawyers, and other such nonsense before we could declare we'd left Github.

I don't like any open source solution, Codeberg and Gitlab are more of the same and Cursor isn't ready yet.

This level of scale is extreme but it can be handled with proper time investment and careful engineering.

But that doesn't fit into the "spend gobs of money as fast as possible so you can be a hypothetical feudal lord in a sci-fi dystopia novel you read when you were twelve" line of thinking.

[dead]