A Faulty Python Script and Five Missing Controls: What eMudhra's Bugzilla RCA Actually Says
A faulty Python key-transfer script and five missing controls left eight eMudhra CA private keys recoverable for 18 months. Here is what the Bugzilla RCA says.
A faulty Python script, written for a one-time key-transfer job in April 2025, left eight certificate authority private keys so weak they could be reconstructed in seconds. eMudhra's incident report — filed on Mozilla Bugzilla as bug 2083449 — describes not a single point of failure but a layered process breakdown that went undetected for roughly 18 months before an external researcher surfaced it in October 2026. eMudhra has since decided to exit the publicly trusted TLS certificate business entirely.
What the incident report says happened
According to eMudhra's own account on the Bugzilla thread and as summarized in secondary coverage by OfficeChai, the trouble began on April 30, 2025, during a key-generation ceremony. eMudhra needed to create sixteen new subordinate CA key pairs — eight ECDSA and eight RSA-4096 — for planned US and European data centres. The hardware for those facilities had not yet arrived, and according to the company's report as described in secondary coverage, a root-program policy deadline meant the keys had to be generated immediately on existing hardware security modules (Utimaco HSMs, confirmed in the Bugzilla thread as FIPS 140-2 Level 3 compliant), with a planned later transfer to destination hardware from a different manufacturer.
To perform that cross-vendor transfer, eMudhra's engineering team wrote a custom Python script — the first and only time such a script had been used for this purpose. That script is where the failure originated.
The ECDSA truncation
For the eight ECDSA (P-384) keys, the script's read-back step — where it reconstitutes a key object from the value extracted from the HSM — accidentally truncated each private key to its first byte. What should have been a large random number was reduced to an integer between 0 and 255: a range small enough for a basic computer program to exhaust in well under a second.
Mozilla engineer Dennis Jackson, who opened bug 2083449 as the reporter, confirmed in the thread that the eight impacted keys all used integers less than 256. The Mozilla dev-security-policy mailing list records Corey Bonnell's observation that the incident report itself highlights no end-entity certificates were signed using these CAs — only CRLs — and that those CRLs failed signature validation, consistent with the signing flow using the real HSM-resident key while the certificates embedded the public key derived from the truncated value.
eMudhra's account on the thread describes the sequence: correctly generated keys were created on the HSM; a deviation from the standard ceremony introduced a wrap/unwrap test for the planned export; a flaw in the code produced a second, weaker key pair; the original and the recreated key objects had nearly identical internal labels; staff mistakenly used the weaker recreated keys when issuing the certificates; the same weak keys were included in the material sent for cross-signing.
The RSA problem: a different, quieter failure
The eight RSA-4096 keys were not cryptographically weak in the same mathematical sense — eMudhra states their signatures verify correctly and the key values were not corrupted. However, the same Python script handled the sensitive private-key components of the RSA keys in ordinary, unprotected host memory during the export process. eMudhra states it cannot demonstrate those values were never exposed, which constitutes a compliance failure under CA/Browser Forum requirements for private-key protection. With no way to rule out exposure, eMudhra revoked the RSA intermediates as a precaution rather than as evidence of theft.
The affected trust-chain artifacts
Eight intermediate CAs were directly implicated by the weak ECDSA keys. The names documented in the Bugzilla thread's attachments (including the HSM mapping spreadsheet uploaded by Naveen Kumar ML) are:
- EM DV TLS CA – G3B and G3C
- EM EV TLS CA – G3B and G3C
- EM OV TLS CA – G3B and G3C
- EM OV CS CA – G3B and G3C (code-signing)
According to the windowsforum.com summary citing the Bugzilla incident record, six of the eight are TLS intermediates trusted by the Chrome, Mozilla, Microsoft, and Apple root programs; the other two are code-signing intermediates trusted by Microsoft. A second group of eight RSA intermediates using the same G3B/G3C naming pattern was also revoked due to the host-memory handling problem, despite no evidence of key misuse.
Why nobody caught it for 18 months
eMudhra's report identifies two primary control gaps that allowed the problem to persist from April 2025 until an external researcher identified the weakness in October 2026:
No key-strength verification step. The key-generation ceremony had no procedure to confirm that a newly certified CA key was actually cryptographically strong, or that the key embedded in the issued certificate matched the key originally generated inside the HSM. Such a check would have caught the truncation immediately after the ceremony.
Ambiguous key labels. The original, correctly generated key objects and the faulty reconstituted ones had nearly identical internal labels. Staff selecting the key to use for certificate issuance had no reliable way to distinguish the strong original from the truncated copy.
There was an earlier opportunity to catch the problem. As documented in bug 1970728, CRLs for these same six P-384 TLS intermediates were failing signature verification by mid-2025 — because the CRLs had been signed with the original HSM keys while the certificates themselves had been issued under the truncated recreated keys. eMudhra's incident report for that earlier bug describes the root cause as "incorrect CA key configuration used during manual CRL generation" and a failure of post-signature verification. At the time, the team simply re-signed the CRLs with the key that matched the certificate and moved on. The earlier bug was reported by Sectigo to eMudhra's problem-reporting address on June 3, 2025 — and that email was quarantined by spam filters, adding a further communication gap. No one asked why two different key objects existed for the same issuer, missing a chance to uncover the truncation roughly 18 months before disclosure.
eMudhra raised a pointed observation on the Bugzilla thread, stating directly: "The affected keys were published in April 2025 and sat in public, cross-signed, disclosed in CCADB, and covered by two years of WebTrust audits, for approximately 18 months before an external researcher identified the weakness. We have accepted our error without qualification. But a serious incident review for the benefit of relying parties also has to ask why the weakness was not detected earlier by any of the other checks the ecosystem relies on."
Community members pushed back on that framing. In comment 54, Q Misell argued that the full source code of the flawed key ceremony script should be released for public accountability, since characterising its release as "sensitive" was not an adequate excuse given the scale of the incident. Rik Peeters, in comment 55, noted that in an HSM-backed environment, the combination of hardware security architecture and operational controls would typically make the introduction of noncompliant keys difficult without an explicit exception, a bypassed control, or a configuration failure — a question that the public record had not fully answered at the time of writing.
Browser and ecosystem response
On Saturday, October 17, 2026, at 20:06 UTC, Mozilla received the external report. By 22:59 UTC that same evening, Mozilla had completed the process of blocking the affected certificates in Firefox using OneCRL, its centralized hardcoded revocation list, as Dennis Jackson confirmed in his October 20 post to the Mozilla dev-security-policy list. Jackson noted that OneCRL allows this protection to be delivered without requiring users to update or restart Firefox.
Chrome's response is documented in the same mailing list thread, where the Chrome team stated it completed corroborating the findings on October 17 and pushed a CRLSet update blocking the Subject Public Key Info (SPKI) hashes of all six affected TLS-issuing intermediate CAs. Because the CRLSet update blocks by SPKI, Chrome users were immediately protected against both the eMudhra-issued intermediate certificates and any Sectigo-issued cross-certificates sharing those keys, without a browser update or restart. The two compromised code-signing CAs did not require a CRLSet block, as Chrome's certificate verification logic does not permit building a TLS path through intermediate CAs constrained to the code-signing Extended Key Usage.
Mozilla also released an update to the NSS trusted certificates list on October 20, 2026, removing the root certificate used to sign the affected intermediates. Chrome indicated it would release a Chrome Root Store update (version 43) removing the same upstream eMudhra root.
All intermediates were revoked. The revocations are documented in the Bugzilla thread's attachments and the CCADB record. No timeline or reason code for the revocations is separately verifiable from the captured excerpts reviewed here.
Remediation and consequences
eMudhra announced it would exit the public TLS certificate business entirely. Its remaining certificate customers, primarily a university network in North America operating through CertiNext and InCommon, were to be transitioned to SSL.com as the replacement issuer, with November 16, 2026 set as the hard deadline for all remaining eMudhra issuance to cease.
The Bugzilla thread records significant unresolved questions about the scope of that transition. In comment 81, Daniel Luedke asked whether KPMG's independent investigation — referenced in the thread as engaged to review the incident — covers shared infrastructure used by CertiNext and InCommon, whose certificates were historically issued through the eMudhra emSign hierarchy. Luedke also asked which functions under the SSL.com replacement arrangement remain under CertiNext or eMudhra control, and whether the identified control deficiencies had been assessed across eMudhra's continuing PKI operations. Those questions had not been answered on the public record at the time of writing.
eMudhra states that no known case of misuse of the weak keys to forge a fraudulent certificate has been identified. The company also states that the Utimaco HSMs themselves "worked exactly as designed" — the fault, in its characterisation, lies entirely in the custom script written around them, not the secure hardware.
What the public record cannot resolve
The Bugzilla record, as it stands, leaves several questions open:
- The source code of the Python ceremony script has not been published. Community members including Q Misell have called for its release; eMudhra has characterised it as sensitive and has not committed to doing so.
- The mechanism by which a FIPS 140-2 Level 3 HSM produced a key object that survived as a truncated value outside the device is not fully explained in the public thread. Rik Peeters noted that such hardware architectures are specifically designed to make non-compliant key creation difficult, and the question of whether an explicit exception or bypassed control was involved remained open.
- Whether the RSA intermediates' private components were actually exposed in host memory, or merely could have been, is not established.
- The scope and findings of the independent investigation referenced in the thread had not been published at the time of writing.
- Whether CertiNext or InCommon operations share any of the implicated HSMs, scripts, personnel, or verification controls has not been answered on the public record.
What the record does support is that this was not a narrow operational accident. The truncation bug is the proximate cause, but the 18-month gap before detection points to a systemic absence of post-ceremony verification, a CRL anomaly treated as a workflow problem rather than a symptom, a spam filter that swallowed an early warning, and audit coverage that did not extend to testing key strength in issued certificates. eMudhra's exit from the TLS business is the clearest signal that the trust consequences were judged irreparable.
Comments ()