Yeah, definitely: having lots of separately distributed and likely rarely and unreliably updated trust roots would not inspire a lot of confidence.

Yes, but the irony here is that Apple used to (or perhaps still has) forbidden using other HTTP(S) clients in iOS apps (or perhaps Appstore in general) than the built-in out of security concerns like this would address by using 1 central storage instead of multiple ones were one won't get timely updates.

> To be honest, git deferring to the OS is the right thing to do.

Tangentially related: I fucking hate so much that browsers now all have their own DNS resolvers and caches, it's just Another Fucking Thing to debug when you're working especially on locally hosted stuff. This is not a browser's job, it's a task that should fall to the OS.

You can have it "both ways" if you want, but it quickly becomes very muddy - defer to the OS, but override the deferral in some cases.

Certificate management is fifty kinds of all messed up - and in THEORY everything is designed perfectly and properly, in practice it's all bandaids on bandaids that nobody cares about as long as "things mostly work".

Wouldn’t be true either. Stuff that uses the OS certificate store (that you can view in the keychain) won’t trust the compromised certs.

Correct. I Googled to ask about location of certs on disk: /etc/ssl/cert.pem

Is it possible for users to remove the eMudhra certificates themselves? I have no experience with MacOS, so I don't know if certain actions are prohibited. To be clear: I am not suggesting that normies do this! Rather, I am asking if it is possible for highly skilled users.

I guess we are going to learn how important secure communication is based on the stories (or lack of stories) about how it is being exploited in the wild.

I could swear I saw that title here earlier today almost verbatim maybe even referencing the same link.

Exceeds character limit, but admins can modify if there’s a better shorter title.

I'm using the latest Homebrew git (2.56.0 confirmed with `which git` and `git --version`), and it remained vulnerable until I manually patched the SYSTEM per https://support.apple.com/en-us/149555

    > Homebrew ships its own CA bundle

Outside of an enterprise environment, I never thought about having multiple "golden sources" of CA certs on a single host. (I have no experience working outside of classic enterprise environments, so please do not read my surprise as immediate dismissal of this idea!) For those unaware, in enterprise environments, it is very common to use a common CA cert pack from Oracle/Mozilla/Google/Microsoft/Amazon/Apple, but add 2-5 CA certs from your own company that act as trust roots.

I did some Googling on this matter. It looks like the "Big Four" of CA cert packs are: Mozilla, Microsoft, Apple, and Google. (I guess there is one or two in China also, be leave that alone for the moment.) What happens if they are in disagreement? This is a complex problem! If anyone is involved in this process, I would like to learn more.

Homebrew’s git (unlike homebrew’s curl) seems to default to using the system’s CA bundle, not homebrew’s. Is that a bug?

> (Source: I am one of Homebrew’s maintainers.)

I would've preferred a doc or a code repo link as a source. Otherwise, this just reads like a brag. A decent one, to be fair :)

> There needs to be an easy way for end users to personally drop trust of CAs they don't need and have that decision respected and remembered.

There is. On macOS go to Keychain Access > System Roots and disable the kind of trust you want to disable. There's probably a scriptable method for that.

On Windows, open the management console to "Certificates". Then go through the list and move/remove/distrust any certificates you dislike. There are also powershell tools you can use.

On Linux, the situation is a bit annoying. You can remove all CA certificates manually and make the directory read-only to break any attempt from your distro to interfere with the list. You need to check your distro documentation to see if they use /etc/ca-certificates or /etc/ssl or something like /etc/pki for reference. Some distros offer a partial GUI as well.

Programs that deem themselves smarter than the OS (true in the case of macOS and various Linux distros), like browsers, the JVM or NodeJS applications, can have their own cert stores.

You will need to acknowledge half a dozen certificate prompts for every website you visit after this, though. Make sure to compare the fingerprints of the CA root certificate with the real ones you might find elsewhere, because it's easy to generate your own CA and copy all certificate information signed by your self-generated key.

There are three futures, each being dystopia than current:

- Only ISRG(Let's encrypt) being the sole CA and benevolent dictator of the internet. (then forced to be dissolved due to antitrust?)

- Each country controls their own CA and you need to purchase 200+ certs in order to make your website generally accessible. (then country XXX will charge you $200/year)

- Big techs owning OS/Browsers control their own CA and cross license each other (then Apple still being lazy handling incidents) (then they form a cartel and block others, like current Tier 1 ISP, email senders, credit card networks)

edit: put enter between numbered items.

Someone has to decide on a set of default CAs to trust... It's natural that it's your OS vendor, since the trust store is an OS thing.

Have you checked with the newsgroups and gopher web on why this became a thing?

Honestly that last paragraph I totally agree. I did edit to “never trust” these clowns’ root CAs in Keychain Access on my Mac, and this is the first I’m hearing that there was any possibility of that being flipped back. Will have to look into that.

If I can speculate though on why these bundles come from the browser or OS and aren’t fully in the user’s responsibility. You’d have to start from scratch for it to not have a bunch of stuff the user had ‘no need for’ to be present, and the user would need to see a cert window semi-frequently that gives the possibility to start trusting a CA, and the bottom 99.99% of users in sophistication has no idea whether they should say yes or no. I mean, without being signed by a root CA you already trust, even the name on the new CA could be completely made up. “Do you want to trust the ‘Apple Root Certificate Authority’?”

Because nobody is going to be arsed to figure this shit out, even the PGP fanatics don't bother.

So someone has to do it, and a pop-up on first boot asking you which "certificate providers" you want to trust would be as pointless as the "which browser do you want to use" ones were.

?

The idea of the internet is that you can reach it across the world, globally.

If you want access to the whole internet and not just part of it, it has to be "one size fits all users". (At least with how the internet works currently.)

Rather than prompting the mechanical parrot, you can also refer to Apple's own website: https://support.apple.com/en-us/149555

I don't know why they haven't rolled out an update if they've already wrote a whole article about patching the problem yourself.

Shallow or misguided criticisms of Apple—of which there are countless—get shredded on this website. This is not a shallow or misguided criticism of Apple. It is clearly and obviously unacceptable.

Better luck next time.

Apple makes incredible hardware and could easily be the best development machine on the market. But I ultimately prefer Linux because I own the maintenance story and it's simple + fast. System packages and xcode are horrible experiences, and there's no reason for this to be the case in 2026.

It has the davantage that you can finally find out what those apps are sending back home, including your private data.

> Apple gets a free pass on this website. … But instead the Apple fanboys will be out in force to tell us why actually its a good thing for Apple to leave parts of the OS insecure.

I have no idea how you've come to this conclusion, and clearly nothing of the sort is actually going on here.

I’m somewhat of an Apple fanboy, and I find this absolutely unacceptable.

We clearly just don't know what complex level security Apple has put into their OS. Clearly their OS isn't vulnerable or else they would patch it! (/s)

Browsers were the original TLS trust store vendors. Operating systems started shipping trust stores because their OS came with a browser, not because they were the natural root of trust. Browsers also have tools to update revocation near-live, while operating systems lag behind by hours, or days in this case.

For your tool, remember to nuke Windows' automated certificate authority updates (separate from normal Windows updates) and possibly the automated intermediary cache. Microsoft publishes a list of trusted CAs/intermediaries periodically and they get automatically applied to Windows so websites don't need to send the full cert chain during TLS negotiation, and it has a tendency to cache trusted intermediaries as well.

Mozilla's trust store is one of the first to get updated when a revocation happens, as well as Google's CRLsets. Downloading full CRLs repeatedly would be a waste of time and bandwidth, that's why there's a specific distribution method for CA updates. I would extend your tool with a daemon that checks for revocation automatically, so you don't accidentally eMudhra yourself while browsers and the OS get updates next time something like this happens.

I was also surprised to find Java’s key tool returning successfully for the delete request without actually doing anything: list, delete, and list again would still show the CA. Debian had this right during the 90s, all they had to do was follow the system convention.

This feels like all of the C code getting attacked now where everyone spent the last three decades building up cruft because fixing it would be a chore, and now the separate bills are coming due all at the same time.

The better way then comes with the drawback where one vendor has all the power.

Then someone would ask "there has to be a better way" again, when the reason for the new inconvenience gets forgotten, and it oscillates.

That fast-track method was used, but it only applies to "graphical" software which really means, software that uses Apple's macOS specific APIs and frameworks, not the lower level UNIX stuff like /etc/ssl

I suspect the fast-track background security updates can only update the databases and revocation lists that live in the proprietary APIs, not any arbitrary file on the system

Yes because that thing is completely useless and provides basically no value

Apple has one job here and it isn't to make UIs that look like melted plexiglass. I am fucking pissed.

I think the main problem is that the weak eMudhra intermediates were cross-signed by Sectigo. Certificate revokations are needed to handle this correctly.

Dropping the Sectigo root along with the eMudhra root is the easy and safe solution, and maybe this is what Apple should have done (it’s the workaround they recommend!). But it means that some small percentage of certificates that are fine won’t be accepted.

People would have to use it in some drastic catastrophic way for them to force it out in an update, which they might push it out in the next macOS update, which don't come nearly as frequently as Windows bloatware updates do.

If you're editing /etc/ssl/cert.pem yourself, you're setting yourself up for failure. Use the proper mechanism to add your own trust roots (on macos it's "security add-trusted-cert").

Whatever reason Apple had for not updating /etc/ssl/cert.pem yet, I am certain it's not out of consideration for users that might have put custom certs in there.

Certainly you're aware that every certificate sits in a separate file and an update can just remove the interested files?

Apple’s handling of this has been a disgrace. It took 4 days before my iPhone and MacBook Pro got the CRL, and now I hear they have done this. Dreadful.

Apple, for the company that prizes itself on believing privacy is a human right, why are you failing your customers so badly on PKI management?

This is such a Linux way to resolve the issue lol

Why would it be more secure to post 4 manual commandline instructions than a single file?

Safari (and other browsers) is safe. The issue is terminal tools such as curl and git

Why arent they just pushing this patch out on their security update channel?

Why are they not just pushing out the update asap? Unless they're relying on the compromised certs, then it feels it should be incredibly trivial to push

Interestingly on a much older OS (Yosemite) cURL, etc. links to Security.framework which uses the system's trust store instead a separate .pem file in /etc/ssl (I can't find any .pem inside /etc/ssl)

I wonder why they stopped doing that and now just seem to do "standard" builds and end up with this (at least) 2 separate trust store situation.

Some depending on the executable, sometimes it goes through /etc/ssl/cert.pem and sometimes through the Apple framework. This OS is a mess.

They did send out an update this week…that undid this mitigation for users that already removed the certs.

I imagine their response would be: "we need to parse a new file format, that may have been edited by users, reliably. That takes time."

But the point of the OP is that git, curl, pip, python, etc don't use certificate transparency. That's why it's so egregious that apple still trusts these certs by default in these non-browser edge cases.

My point is less NSA and more NSO. The UAE was denied a root CA[0]. DarkMatter applied to be its own root CA until it was denied[1] over concerns raised about potential spying/espionage[2].

There are people from these CAs who were denied in the past explicitly for concerns about spying who now work at eMudhra[3,4].

    0: https://bugzilla.mozilla.org/show_bug.cgi?id=1474556
    1: https://bugzilla.mozilla.org/show_bug.cgi?id=1427262
    2: https://groups.google.com/g/mozilla.dev.security.policy/c/nnLVNfqgz7g/m/TseYqDzaDAAJ
    3: https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c43
    4: https://emudhra.com/en/blog/scott-rea-joins-emudhra-as-senior-vice-president-global-strategic-advisor

XProtect removes malware, not certificates

Just like Apple has. The problem here is whoever made the decision not to use that capability:

https://support.apple.com/en-us/102657

I think they meant that Apple’s updates seem to be the opposite of incremental, and they’re not wrong. If anything, Apple has moved in the exact opposite direction in the past decade, moving towards using enormous cryptographically-signed volumes (“sealed” is the term). This makes Apple really happy because they can make sure the users aren’t allowed to modify any of the “system“ files themselves for any reason, but also means that they can’t incrementally update things as easily as they did before. They ship multi-gigabyte “patches” every time.

As far as I'm aware the typical command-line stuff and most things just using SSL libs or language standard libraries don't do revocation at all by default.

Similar reason (though with the opposite effect) as how a cert can be fine for browsers but fail for curl when they don't send an intermediate: the "vanilla"/library/command-line stuff is just checking for a path to a trusted cert and not doing the CRL or AIA stuff a browser does.

The root certs may have signed other intermediaries that are still trusted.

Yeah, it increases the attack surface of mitm attacks to your network from the local os level.

Doesn't affect app bundles though, only *nix commands. This'd be detected pretty quickly, at least by the significant portion of patched devices.

Why would these public network would allow guest hosts to communicate with each other?

Even a simple rouge AP is enough to become a MITM for this.

I don't know why you're downvoted, any public wifi access point in a shop could be compromised and users would never notice.

Client isolation is very common on wireless and wired LANs these days in enterprise and guest network environments. You also have tools like dynamic arp inspection to block poisoning attempts via the network switch.

If you get past all of that, you still need a victim who is actively using a cli tool on macOS using https.

It’s not impossible, but the conditions are much more specific than those needed to inject porn into a webpage on a poorly configured school network in 2006.

> all you need is physical access to the network

Yeah that's not so much "trivially easy"

> all you need is physical access to the network.

yeah, it's trivial to punch somebody in the face in the morning, all you need to do is to be by their bad in the morning when they are waking up.

I think "everything" is an overstatement. Browsers included the compromised intermediates in their CRLs within hours of disclosure.

So this isn't like Firesheep [0]. Yes, there would be vulnerable traffic, like the git client this thread is discusing, but it's not "everything"

[0] https://en.wikipedia.org/wiki/Firesheep

Appealing to complexity is always fun. Just throwing "a lot" in there for good measure and you won't have to think about practical security implications.

You might be detected but if, for example, you were smart enough not to MITM the same client twice after an error, I suspect a ton of people would ignore it and just assume the airport/coffee shop WiFi was flaky.

I’d also bet more things use libcurl behind the scenes than we appreciate, and that at least a few of them have things like silent updates which retry after a failure without notifying a user. I’ve seen a lot of agents which just try to periodically check in because they’re designed not to be noisy on lousy networks.

Exactly, I've been to public places where the Wi-Fi password is on the wall and the router's admin password is "admin".

Anyone running Intel macbook because Homebrew dropped support for them because Apple dropped support for them. You can still use Homebrew but it means rebuilding from source. I downloaded uv recently and it took about 8 hours to fully install because it had to build llvm, uv, some other stuff all from source.

Heaps of us. I'd been using it for years before Homebrew even existed and I never felt that any of the problems that Homebrew purported to solve were real.

I know, it always makes my jaw drop when I realize that it’s still a significant number of people, since I remember it being a thing I used circa 2005. I don’t think git was even a thing back then.

Intel Mac users. I recently installed MacPorts for the first time after Homebrew dropped support.

Their branding, mainly.

They changed the title, it said github client before I believe.

The permanent members of the Security Council, obviously.

The US, which already centralises the internet by running .com and developed most of the software using these certificates?

It's not like they cannot introduce a bad certificate right now. We would be just be taking the ability from all other countries from doing the same.

That's one issue from one distro 18 years ago. Have there been recent things? This thread is about a today's MacOS, and an issue known publicly for more than a week.

> the random maintainer had no idea what they were doing

Upstream was involved in it as well. Sure it was a communication problem, but it's not like the maintainer just decided to this unilaterally. Also this WAS undefined behaviour, so from a language perspective it was already broken, even if it worked accidentally in practice.

In 2008.

I don't think its wise to make current security decisions based on screw ups from almost 2 decades ago.

You are going so back that gmail was served over HTTP (not HTTPS) in that era.

If that's the worst you have against Linux then it looks like it has done very well.

Besides singling out a single bug from a single distro two decades ago, Linux offers maintainers who actually care to push security fixes within a few hours of globe-spanning vulnerabilities being disclosed? Not sure how that's confusing or controversial.

Let's pretend there's a night and day difference between pushing some MP3 onto all iPhones and distrusting a chain of certs that hold the value of '4'. At minimum Apple need to revoke everywhere these joke-level certs live and as fast as possible, and on all devices which also means out of support. Breaking is acceptable here as CA trust is fully compromised.

Yes, there would be crowds of angry people, furious that Apple... released a critical security patch? Framing removing compromised certs as "changing peoples systems deeply" seems intentionally in bad faith.

It's funny, when I do steal something I get called bad, and when I do nothing and turn the blind eye to others stealing, I get called bad too.

Morality doesn't solely depend on action or inaction, it also matter what you act or don't act to.

I've seen this referred to as the "All Roads Lead to Rome" fallacy.

Eh? Changes systems without consent? Is that the ONLY way to update a fatal flaw in system? Normal update channels don't work?

Besides, how does this relate to my Linux point, where no one installs anything without you knowing it?

I don’t think you understand what consent means.

Now would be the time :)

No, no, I meant it to show that the machine is running the current/latest macOS, and the results I am getting from curl on the two URLs.

Its no more wishful than expecting a root of trust to stay trustworthy or expecting a trusted name to consistently resolve to the same party.

And there's a lack of motivated attackers for a reason, the attack scales poorly. Maybe you can bamboozle a target or two, but if you go for hundreds, somebody's gonna start wondering what's up with the keys. If you're trying to take over the world, or sell to somebody who is, you're better off focusing on compromising a root of trust or tampering with name resolution, because attacks like that are sustainable at scale.

good point -- since the revocation is not recorded in the /etc/ contents.

Yes, it’s insane. Step 2 does next to nothing for security without step 3.

Exactly, so the title wrongly frames git as trusting certs that it shouldn't, when it's correctly trusting the system certs, and MacOS is wrongly trusting certs.

The problem is macOS has two system CA stores, keychain and /etc/ssl/cert.pem.

That is unequivocally bad, similar to Java which ships its own keystore.jks and many applications shipping its own copy on top of that. True nightmare fodder.

> Is using the system CA not the way to go?

Yes, and that makes not dealing with this issue worse on Apple's part. If MacOS' setup somehow doesn't allow untangling a (still deemed trustworthy?!?) root certificate issued by eMudhra from the broken intermediate certificates, then revoke that whole chain including the root & accept collateral damage.

Failing that just provides an incentive for apps to do their own chain-of-trust implementations. Which like you said is not the way to go.

It's not factually wrong, but it implies that this is something specific to Git, and possibly might even be interpreted that it is Git's fault.

In reality it's macOS shipping the outdated certificates, and it affects various other command-line tools, like Curl.

Because it isn't got specifically, it is anything that uses the installation of OpenSSL provided by Apple as part of macOS (at least is my understanding).

The problem is that the title implies that it's somehow the fault of the package. That's it, that's the problem; the title is disingenuous in its claim.

"Ten days later, Apple Image Viewer on macOS still trusts the compromised eMudhra certificates" doesn't sound as catchy, but is equally correct because the apps use system CA.

When a significant portion of devices are patched (including gui apps on mac), pulling off an MITM without detection is pretty much impossible. That would be usual culprits of telco, ISP, or public wifi. That's what I mean by the window has closed. Anyone attempting is taking significant risk of being detected; at least in my neck of the woods would be illegal to do as well.