In 2017, while attempting to get some DRM-enabled video player to work on my Mac, I stumbled upon a hard-coded private key. The corresponding public key was used in a valid and publicly trusted Cisco certificate. This further piqued my interest in the internet PKI, and made me wonder how many private keys I would be able to find. In the months that followed I found and reported many hundreds of certificates of which the private key was compromised. In this post, I want to focus on one particular compromised key.
I started out by building tools to scan the internet and public data sources (e.g. GitHub’s public data set) for PEM and DER-encoded keys and certificates. After scanning for a while, I inserted the collected raw certificates and keys into a local document database for further analysis. This allowed me to quickly identify publicly trusted certificates with known-compromised keys by filtering out all non-publicly trusted certificates, and joining the certificates with the private keys on the SHA-256 hash of the extracted public key (Subject Public Key Info). Grouping and counting the certificates on this hash showed that there were multiple certificates that shared the very same RSA public key. Searching the certificate transparency logs showed there were a few dozen in total.
Details of the compromised RSA key:
- SHA-256 hash of the SubjectPublicKeyInfo:
- Raw PEM-encoded private and public key.
- Certificates that use this public key: crt.sh, Censys.
In the end I had a list of a little over 60 certificates that all used the same compromised key (full list). These certificates were valid for DNS names of seemingly-unrelated customers, which ruled out the possibility that one customer simply reused their private key in different certificates. This made me curious to figure out the source of the compromise.
I had two leads to go on:
- The raw private key was found in two unrelated GitHub repositories that hosted the source code of a WordPress website.
- A significant number of the DNS names for which these certificates were valid pointed (or had pointed to in the past) to DigitalOcean’s IP space.
I also recalled one incident from a few years ago where Hetzner Online, a well-known German web hosting provider, had provisioned servers with identical SSH host keys. Which made me think that this case might as well be related to using some base image with hard-coded keys. What strengthened this thought was that I found that DigitalOcean, for a while, allowed customers to quickly install popular web apps using virtual machine images provided by Bitnami (archived page).
After scanning some of the virtual machine images for WordPress, I quickly found the private key in one of the WordPress images. These are the steps to reproduce:
- Download the file bitnami-wordpress-4.9.1-0-linux-debian-8-x86_64.zip.
- The zip file should match the SHA-256 checksum:
- Unzip the contents (a single VMX configuration file, and multiple VMDK files).
- Find the private key in the
bitnami-wordpress-4.9.1-0-linux-debian-8-x86_64-s001.vmdkfile at offset
0x163C6000(and the next 4096 bytes).
- Extract the key using a tool such as binwalk.
Provided you saved the extracted private key with the original file name,
server.key, you could verify that it matches the compromised private key by calculating the SHA-256 hash of the Subject Public Key Info using OpenSSL:
$ openssl rsa -in server.key -pubout | openssl rsa -pubin -outform der | openssl dgst -sha256 writing RSA key writing RSA key 2bf95e73d665fbdcf525f08420573df075ea81759e46b7554a2afa1b68d3bebe
This issue was reported to Bitnami on January 14, 2018. As per my suggestion, they released new versions of the virtual machine images for WordPress (and all Apache-powered catalog apps) to generate the default key at first-boot time. They also updated the existing documentation with instructions on how to furnish a new key before requesting the issuance of an SSL certificate. The fix is included in WordPress image version 4.9.2-1 and later.
After the original issue was fixed, and after I had reported several dozen certificates, I thought the case was closed. But then, over the course of 2018, I noticed that there were still new certificates being issued with the aforementioned compromised key, including, most recently, these two:
|crt.sh ID||Subject common name||Issuer organization name||Not after|
Surprisingly (to me, at least), this means that:
- Certificate authorities allow the same key to be reused by different customers. They do not appear to proactively check whether a key was used before, and block the key accordingly.
- And worse, certificate authorities allow keys to be reused even after they themselves previously revoked a certificate with that key (with the reason code “keyCompromise”).
At least Comodo (now Sectico) eventually added the compromised key to their internal blacklist to prevent future issuance of certificates with that key. It appears this is not something certificate authorities do proactively when a key compromise is reported to them. As far as I understand, per the Baseline Requirements, they are not required to do so.
In a brief email correspondence, Rob Stradling also pointed out that he previously made some suggestions for future improvements in this area.