rdtsc
4 days ago
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
overtomanu
4 days ago
It was "multiple human errors". But yeah no matter what, in any case or exceptional circumstances, this thing was not supposed to be screwed up.
On Mozilla’s tracker, eMudhra said the keys were created on 30 April 2025 in a key generation ceremony, a recorded procedure in which keys are produced inside hardware security modules (HSMs), tamper-resistant devices built to hold keys. The keys were meant to be moved to data centres in the United States and Europe for customers needing data stored in those regions. To move them, the company added a step to ‘wrap’ each key, or encrypt it for transfer, and ‘unwrap’ it at the other end. eMudhra said a flaw in the software script created a second, weaker set of keys during this step, and because the two sets had similar names, staff used the weaker ones by mistake. It said human errors occurred at multiple steps.
source:
What's the eMudhra security scare that has rattled the internet?
https://theprint.in/tech/whats-the-emudhra-security-scare-th...
wolvoleo
4 days ago
These "wrapping" keys were not "weaker". They were completely unfit for purpose. They have not explained why they used a 3-bit key for this purpose.
If they go to the trouble of wrapping them, use real keys. Or don't do it.
Also, it's a big "human error" to not notice this for a year and a half.
silisili
4 days ago
I have to imagine that the 3 bit numbers were slots/indexes on the HSM, and rather than actually retrieving the key(or maybe it did), the slot number got used as the key. I can't otherwise imagine how they'd generate such low number keys...
fifilura
4 days ago
I have not read up on all of this, but I would have guessed they were artifacts created and validated by a testcase. "testcase 1-8".
mort96
3 days ago
The key '4' was just the smallest key, there was a set of keys all below 300
gschizas
3 days ago
Dude, use nice, round numbers. The keys were all below 256!
pseudohadamard
2 days ago
They're still safe from attack by quantum "computers", so what are you worried about?
Foobar8568
3 days ago
AI ceremony.
patch_cable
4 days ago
Having worked around these types of devices and their interfaces, I doubt they intended to create weak keys.
It is more likely they had a script which placed a parameter in the wrong field, due to something like using positional arguments. Or they misread the documentation for some interface.
Still incompetence. And I would love to see my theory confirmed or refuted.
motionlessveloc
4 days ago
The previous discussion advanced the theory that they fucked up the string handling. Eg in Python, output = key[0] would result in one-byte keys like "4" if you expect "key" to be an array, but it's actually a string.
Amezarak
3 days ago
There were also several two and three digit keys.
motionlessveloc
3 days ago
AfAIK all the keys were exactly one byte (2^8) in length, only the decimal representation was multi-digit.
fzeindl
3 days ago
Could still have been the same issue.
Code assumed an array of strings, instead got an array of bytes and picked the first one.
oofbey
4 days ago
I hear you in terms of how these kinds of mistakes typically happen. But why did the script ever produce two sets of keys? The initial root cause was somebody picked the weak key not the strong key. But what was the reason for ever producing a weak key in the first place? They say the script produced two outputs - what on earth for??
gopher_space
4 days ago
Stepping back a little to look at the point and gravity of the exercise makes me wonder why they didn’t just fly it over.
ChuckSneed
4 days ago
> worked around these types of devices and their interfaces..
This is not a problem of devices and interfaces. You are supposed to generate the wrapping key on the HSM itself. And the HSM ensures that the keys are generated with a strong enough security parameter, not some silly 1 byte string. Then the HSM wraps the key inside of itself and exports it in the encrypted form. You take this ciphertext, and the wrapping key and send it over to the other party.
So if they generate the wrapping key with a funny value, that means they didn't use HSM to generate it but their system openssl. But then, HSMs typically don't allow you to export private keys without wrapping them first unless you first tweak around with security flags (not recommended at all), or simply generate the private key using your system openssl too (LMFAO).
Source: I work for one of the major HSM vendors.
lenerdenator
4 days ago
I mean, why have the ability to generate weak keys to begin with? Serious question, I've never done this sort of work. What sort of utility does that bring to the procedures and workflows one does in that line of work?
If you didn't know better, you'd think this had been done on purpose by some three-letter agency.
NetMageSCW
4 days ago
Testing could be a reason.
To be clearer, I’m not sure these keys were generated, but being able to use weak keys is something you may want to do for testing.
SAI_Peregrinus
3 days ago
What testing would require it? I've done plenty of testing setting up internal CAs & HSMs, and none of it required weak keys. Generating a key with the same parameters as a real key is trivially easy, and has the benefit of testing your key generation at the same time!
cogman10
4 days ago
Yeah. I can't think of any development process where you wouldn't simply use a production grade key in development. In fact, that's exactly what we do in development with snake oil keys.
Generating a key with "4" was probably done as a joke, but also is insane to have ever lived outside some devs computer.
rstuart4133
4 days ago
> These "wrapping" keys were not "weaker". They were completely unfit for purpose.
Oh come on. They clearly didn't intend to use 4 as a private key or to wrap it. They had just purchased millions of dollars of HSM's to store them. It was something more subtle than that.
Have a little imagination. The transfer procedure sounds bespoke - possibly just a shell script using openssl to encrypt things at one end, and decrypt them at the other. Let the man who has written 100 shell scripts without a bug in one cast the first stone.
But not so subtle that good software engineering procedures wouldn't have picked it up. In fact today just an LLM review of the shell script probably would have caught it. The Internet Death Sentence was warranted - because poor software engineering procedures are unforgivable in a CA. But those procedures and policies aren't easy or cheap to come by. If they were just "don't chose 4 as a primary key" my grandkids could do it.
telebub
4 days ago
Unwrapped extraction is the kind of thing security by checkbox covers and process review might catch so it is never clear in a case like this that some party with the capability to influence the process didn't want keys intentionally weak to emulate unwrap.. Then someone else may have introduced the swap of keys that would never have been seen with keys that would (be forever) in a public scrutiny DB.
rockdoe
4 days ago
I don't think it's implied the wrapping keys were the weak ones. More like, the script that ran the wrapping still had the fast "debug" generation of a pair of keys to test the wrapper on enabled, for example.
redanddead
4 days ago
vibe coded?
and imagine being responsible for halving your corp's stock
xbar
4 days ago
Well said. I choose not to believe this was an error.
cdelsolar
4 days ago
wait, a 3-bit key??
Tuna-Fish
4 days ago
One of the 384-bit keys was literally 4. All of them fit in a byte.
sippingabonedry
4 days ago
Their complaint about social media reeks of "and we would have gotten away with it, if it wasn't for those meddling kids". [1]
rdtsc
4 days ago
We deeply and sincerely apologize for getting caught and not being able to cover it up.
khriss
4 days ago
This is pretty normal in India unfortunately. Crony capitalism is the dominant business engine in the country, so pretty much any significant business, esp in regulated sectors like govt mandated services(which is the bulk of eMudhra's operations) is impossible to set up without significant bribes being paid.
As a consequence, these businesses are very well protected from normal market forces and are pretty much above the law. Had this incident not been an international one, they had a very good chance of getting away with it, and the people reporting this would have landed in jail.
thraway3837
4 days ago
This is 100% accurate. State run media would have squashed this article if it was not for an international incident response. They have federal laws that prevent such disclosures because it affects the national image and could cause social unrest.
moralestapia
4 days ago
Since you seem knowledgeable about this topic.
Would you say the system forces people to be like that or people are the reason why the system is like that?
Would you do business (i.e. trust) with someone that comes from that system?
extraextra
4 days ago
You're asking a chicken and egg question, I believe it becomes a self-reinforcing loop
In regards to doing business - someone that comes from a corrupt society with a culture of taking advantage of others should be approached more carefully than someone that grew up in the same town as you, of course
This is a lesson learned the hard way, where I moved to a different country and treated everyone with openness and generosity (of time, resources, willingness to help), just to realize people from certain cultures mistake this behavior for weakness and try to take advantage as much as possible
dennis_jeeves2
a day ago
> You're asking a chicken and egg question, I believe it becomes a self-reinforcing loop
Generally true, with a nuance. The root cause is often people, specifically its people with low IQ. When a society is corrupt you will notice that the mean IQ is lower.
da02
4 days ago
Which country is this? So I can avoid it.
extraextra
3 days ago
I think the country itself is irrelevant, I'm an expat there as well as the people who exhibit the behaviors discussed (different backgrounds though)
Anyway, the main point remains, not all cultures have the same values and if someone comes from a culture where taking advantage of others is celebrated, (for your own sanity) beware
qart
4 days ago
> Would you say the system forces people to be like that or people are the reason why the system is like that?
It's the courts. Courts are meant to bring in an error correction mechanism. When courts fail, the country goes into open-loop control. No errors are corrected, and eventually, people stop even detecting errors. https://x.com/captgouda24/status/2110089855380369511
> Would you do business (i.e. trust) with someone that comes from that system?
Vet rigorously and continuously! Start with small business, and grow it based on interpersonal trust. Since the system lacks accountability, you must ensure that yourself.
franktankbank
4 days ago
Wild to me that our healthcare data lives there.
bigfatkitten
3 days ago
I worked for a bank who had a team within its security org dedicated to dealing with fuckups caused by Indian outsourcers.
lenerdenator
4 days ago
Depends on the contract your healthcare providers/payers/etc. have with those who do their IT.
Source: work on healthcare IT projects and can't let anyone outside of the US look at prod data no matter what due to the contract we have with our clients.
squidmaster
4 days ago
[dead]
aboardRat4
4 days ago
>capitalism is the dominant business engine in the country,
The government _mandates_ that certain services be used. In which world does this rhyme with "capitalism"?
khriss
4 days ago
I said _crony_ capitalism is the dominant business engine in the country.
The govt mandated services are owned by private capital(hence the capitalism part) and govt picks winners by granting licenses to a select few(hence the crony prefix)
KingMob
4 days ago
Any world where sufficiently wealthy capitalists can bend the govt to their will, i.e., most of them.
hnlmorg
4 days ago
It does, but I wonder how much of that phrasing is cultural.
When working with different businesses across the globe, I’ve noticed Indian correspondence to incidents tends to be more a show of strength even if it’s transparently obvious that there’s no merit to the content.
Whereas western European correspondence is often more apologetic and empathetic. Even if it might sound insincere to those affected.
But it’s all just words at the end of the day. It never means much unless the business is willing to back real change to mitigate similar issues in the future and resolve undisclosed existing problems. However there’s no way for us to know if this is happening or not.
ryandrake
4 days ago
I don't know if it's really specific to one culture. Look at US companies after they go through a lengthy legal saga, it's obvious they screwed up, they're paying fines or having to adhere to court-imposed consequences. Their press releases:
"We admit no wrongdoing. We are only settling to move past this. Our business is strong/secure/competent."
It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
aboardRat4
4 days ago
>It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
It's the first rule you learn after joining basically any organisation larger than 5 people.
People on HN who are surprised are seemingly self-employed prodigies who dropped out of middle school.
hnlmorg
4 days ago
I’ve worked for a lot of enterprises across the globe and I can tell you that you’re flat out wrong. That’s simply not how all organisations operate.
Just take a look at any of the thousands of postmortems published over the last decade and you’ll see that the tone of incident response communication can vary massively.
expedition32
4 days ago
The truth would have come out eventually- after a bunch of security cluster fuck incidents.
gquere
4 days ago
Their OTP solution is bruteforceable as well: https://mastodon.catgirl.cloud/@49016/117475921584308861
rdtsc
4 days ago
This becomes an inverse Russian roulette: only one of six chambers is not loaded.
Waiting for a release like "... the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP 1FA place in our business. In view of this, after detailed deliberations our board has concluded that withdrawing from global OTP 1FA is the right decision for eMudhra and its shareholders"
throw0101c
4 days ago
> This becomes an inverse Russian roulette: only one of six chambers is not loaded.
At what point does it become Russian roulette with a Glock?
boricj
4 days ago
One chamber is enough to play Russian roulette... once. However, the probability to get to play a second time is rather slim.
VorpalWay
4 days ago
Link is broken, gives some randomly generated text when using Brave on Android. Their AI bot detection is not very good.
rdtsc
4 days ago
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
rauli_
4 days ago
That Web site thinks that my browser is an AI scraper for some reason.
anakaine
4 days ago
It's not clear what you have linked there.
rdtsc
4 days ago
Paste from mastodon:
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
thraway3837
4 days ago
"However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP's place in our business"
pigggg
4 days ago
Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.
sippingabonedry
4 days ago
Then indirectly the security and trust of the entire Internet rests in the hands of a few government babus in India... that's scary.
pigggg
4 days ago
Their Internet registry people also weaponized AS0 (zero) as a method to take networks off air via BGP RPKI for the ranges they're responsible for. There's been a number of situations where multinational companies who operate in India who are forced to use IPs from the India RIR have been fat fingered by some staffer who makes changes then goes home for the day: meanwhile a multinational bank is down for 24 hours within the country because said agency doesn't believe in 24/7 support.
Typically they do this for non pay but they've dropped the hammer multiple other times incorrectly (and good luck getting a RFO out of anyone there).
sandeepkd
4 days ago
For good or bad, thats by design. These is a significant portion of physical infrastructure associated with internet data flow. The only challenge governments face as of now is being able to segment the traffic to their wishes in its entirety.
Companies already control all the traffic flow on their infrastructure using end-point security solutions, the same can be extended at ISP level, and then Country level
miohtama
4 days ago
It’s called regulatory capture.
It’s get worse when these companies work with regulators to create more regulation that they can sell more of their services.
https://www.amazon.co.uk/Compliance-Industrial-Complex-Opera...
cylemons
4 days ago
Regulation like this makes sense, but then the government should also srutinze these companies
geodel
4 days ago
Well Indian regulator must have visited office canteen to see no unapproved food was served there.
Only when fully satisfied they'd have given license to operate this security business.
debarshri
4 days ago
"bias"
midtake
4 days ago
"out of proportion to SSL/TLS’s place in our business" is an insane comparison. They are saying they care so little about SSL/TLS to begin with that it's not fair.
It would be like me practicing surgery 1 hour a week and saying "well surgery only has a very small place in my 80-hour work week" in response to flagrant failure of medical care.
cogman10
4 days ago
"We are bad at this job, but we don't do it often so don't think we are bad at other jobs".
Yeah, the proportion if anything is too small. This is the sort of company that should not be doing anything related to security.
mrweasel
3 days ago
You're reading it the wrong way, I think. Their certificate business is so small, 1% of revenue, that it's not worth trying to rescue it after the amount of attention it has been getting. It simply wouldn't be profitable.
The more telling thing to me is that they explain that THEIR core business is unaffected, but they don't explain that they fucked up so badly that everyone else had to step in and actively update their trust chains to work around eMudhra to avoid putting every single internet user at risk.
Quite frankly I doubt they understand how massively they've screwed up, or they are trying to hide it to avoid absorbing any blame for the fallout.
thraway3837
4 days ago
They got their image hurt, and pulling the culturally appropriate response here which is to never apologize and to never accept fault. Instead quit/resign and say that other stuff is just fine. It's really not fine, and you can bet that the same corporate philosophy that made the CA error will have serious security errors in their other products.
cebert
4 days ago
Right? I don’t know why anybody would trust this company for anything.
fabioborellini
4 days ago
Someone higher up in the hierarchy telling you to trust this. You have no option to challenge it.
user
2 days ago
paustint
4 days ago
This is what I found regarding how it happened, but still not a full explanation - but there was some name confusion and obvious incompetence.
>According to eMudhra, proper strong keys were created in the HSM at the key ceremony on 30-04-2025. The script had been changed to test wrapping and unwrapping keys, because the intermediates were to be moved to data centres in the US and Europe, and a bug in that change created a second set of weak keys. The keys had almost identical names, and the weak ones were used for the certificate requests. The same request was signed by eMudhra's own root and sent to Sectigo, which cross-signed it. The strong keys never made it into any certificate (Bugzilla).
https://www.fairssl.se/en/articles/emudhra-ca-noegler-knaekk...
raffraffraff
4 days ago
Fantastic key ceremony. Great time had by all. Can't remember a damn thing, we were all so plastered.
CobrastanJorji
4 days ago
I wonder. Say I wanted to prevent something stupid like this from happening in the future. What would be a sensible check? The higher level authority doesn't see the private key itself, right? Just the certificate request. And I wouldn't want someone to manually inspect the keys to see that it looks random-ish. Perhaps there could be a small check script added to the generation of certificate requests that says "hey, your key a.) is mostly zeroes, b.) appears to be ASCII text, or c.) seems otherwise extremely non-random, please use -f to continue anyway".
MadnessASAP
4 days ago
I don't know if it will work on something as small as a private key, but you can compress the key. If the compressed key is significantly smaller then the original, it lacks entropy. Pure random data is mostly incompressible.
Note that 32 0x00 is a possible output from a pure random generator while still being quite compressible, you would probably still want to throw that one out though. After buying a lottery ticket.
throw93939e8
4 days ago
Do not allow devs to modify ceremonial hardware.
fc417fc802
4 days ago
I was about to reply with "they didn't have a choice" but on second thought perhaps CAs are important enough that they should never be allowed to export keys from an HSM and should instead be required to physically deliver and install the HSM wherever it needs to be in the world. Inconvenient and expensive but likely for the best.
SAI_Peregrinus
3 days ago
Exporting keys under wrap isn't the problem. Importing new keys not generated by an HSM would be, if that's what happened and the HSM didn't actually generate the ridiculous keys. Since the HSM shouldn't be capable of generating such weak keys, it's likely they generated them externally somehow (typing by hand? all decimal digits being in increasing order is unlikely even for rand(255)-style insecure RNG calls) and imported them to the HSM.
fc417fc802
3 days ago
Sure, but "must never leave the HSM" is much easier to audit for and much more difficult to screw up. This entire thing (allegedly) happened because of faulty logic; categorically disallowing as much logic as possible seems like the straightforward fix to me.
If transfer were to be allowed I'd think it should be entirely facilitated by the hardware itself and that capability audited as part of the approval process. (I assume the HSM used in this context is already subject to approval?)
saidnooneever
4 days ago
Russian Roulette is pretty accurate if not an understatement.
If they expect people to still trust them on their other services they have obviously no clue on how Trust works, meaning they shouldn't be in the game in the frist place.
As a trust provider you get one chance. Squander it and you're out. That is how it should be.
kjs3
4 days ago
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
In any legal dispute involving them, Exhibit A will be "look at how they ran their SSL/TLS business". That won't go well.
teknologist
4 days ago
I think it doesn't matter what they say because they'll just get away with it. It's as simple as that. They'll get away with it.
MrBuddyCasino
4 days ago
[flagged]
kshri24
4 days ago
There is no such honor code and stop spreading this BS propaganda. Izzat is not even an Indian word used in any Indian language. It is Arabic which got mixed in with Persian which in turn got mixed in with Urdu.
anakaine
4 days ago
Well reposted.
throw93939e8
4 days ago
West has something similar. Politicians never apologize, and avoiding Indian certificate authorities (and recuriters) is racisist!!!