Get Me Out of Here!

…Or, How To Fix Your Outbound SSL/TLS Connection Problems

So, the other day I was working on an problem for a customer, and I hit up against a very common glitch using SSL/TLS going outbound. I’ve stumbled onto this more than once now, and so I thought that it would be helpful to share the fix. If nothing else, this post will serve as a record of the procedure so that I will be able to quickly reproduce my own fix next time.

The Context

Enterprises need to protect their perimeter, and will (almost) always use SSL Termination at their “edge” routers.  This means that any SSL/TLS connections you try to make going outbound are going to be intercepted and eavesdropped, in order to inspect the contents of the traffic. The actual infrastructure will vary of course, but the relevant network gear will typically be a box from, say, Cisco/Ironport, or an F5 Big IP, or BlueCoat, etc.

When you try to create an outbound SSL connection to, say, https://www dot any site dot com/downloads, you can expect that the network box handling your request will prematurely terminate that SSL/TLS connection, and examine the contents of the request in real-time. If the request looks OK, then the router will repeat your original request to that target URL.  In effect, your SSL/TLS connection is actually being subjected to a classic man-in-the-middle (MITM) attack.  Except that in this case it’s not really an attack — it’s being done to defend the enterprise, and to comply with the applicable government regulations.

As the old saying goes, you can’t manage what you can’t see.  And so the CISO couldn’t effectively claim “due care” compliance with data protection and privacy regulations when any user can create an opaque SSL/TLS connection going outbound from the enterprise. Think of security use case scenarios such as a disgruntled employee, or an undetected strain of malware, exfiltrating private customer data over an encrypted channel.

So there are good reasons for doing this, and in most cases the users will never even see it.  However, if your SSL/TLS request was initiated from a developer VM in the lab, using an Ubuntu command line tool — rather than from an enterprise managed browser — then your request will likely fail.

Where the Rubber Meets the Road

When you are working from your favorite browser, and you try to do an HTTPS GET from your favorite download site, the browser will check it’s local trust store to see if it recognizes the x.509 certificate that the target site has presented.  If the certificate for the site has been signed by a CA that your browser trusts, then all is well, and you’ll smoothly connect.

If not, then you may see a security warning message, asking you if you want to accept the new certificate.  Different browsers will behave differently, but in general you’ll see that type of security exception for any new, unrecognized certificate.  If you look carefully at the certificate contents you might notice that the certificate you’re being asked to accept has a subjectName which is consistent with the site you’re trying to reach.  However, the certificate issuerName is going to be the local enterprise security operations team, rather than a widely recognized CA.  So, if you’re on site working for a client named Acme Widgets, you’ll see something that effectively says: “Security warning:  Do you want to accept the certificate for Amazon.com issued by Acme Widgets, Inc.?”  “Hmmmm…”, you think, “I didn’t know Amazon got their PKI certificates from Acme?!”

They didn’t.  What’s happening is that your outbound SSL/TLS request has been intercepted, and a certificate was issued (often, on-the-fly) stating that the enterprise vouches for the target site.  Once you accept that certificate, your SSL session to the edge router is established.  And a new SSL session is then created, as a next hop, to the target site.  The trust for that second hop is established using the actual site certificate for the target site.  The enterprise’s edge router is acting as a man-in-the-middle.

Usually, the browser’s trust store has already been updated as part of installing the security infrastructure, and so the local enterprise CA is already considered a trusted authority.  The typical user won’t even see the warning message, and everything just works.  The user gets to do their uploads or downloads, and the enterprise gets to inspect exactly what content is traversing their infrastructure.

Beyond the Browser

So what’s the essential security architecture problem here?  Too many trust stores!

The annoying glitch occurs when you are initiating that SSL/TLS request from a program that does not use the browser’s trust store.  The first symptom is likely to be a connection timeout, excessive connection retries, or a stack trace complaining about network connectivity.  Of course when you try to debug this, and you connect to the target site using the browser, it just works.  Grrrr!  After investigating further you’ll find that the only time you can connect to the required site is from a browser, but never from any other command line utilities.  Again, the reason it just works from the browser is that the MITM CA certificate is already there, in the browser trust store.  But if you use a command line utility like curl or wget it will fail, because the MITM CA certificate is not in the corresponding trust store.

OpenSSL to the Rescue

You can use openssl to diagnose and fix this issue.  Do the following from a shell prompt:

$  openssl s_client -showcerts -connect anysite.com/downloads:443

The result should be a fairly verbose (but fascinating) view of the SSL handshake negotiation in progress.  You’ll see that the client connect request is sent, and consistent with the protocol spec, the receiver presents the server’s certificate chain.  Adding -debug will show even more.  Looking carefully at that certificate chain will reveal that it is probably not the one you expect, and the last line of the output will appear as follows:

Verify return code: 20 (unable to get local issuer certificate)

Translated, this means that the CA certificate for the issuer (i.e. the enterprise MITM Certificate Authority) is not present in the default trust store.  Assuming the software you are running is leveraging the OpenSSL certificate store, then by updating that common store, you can fix any number of utilities or applications that rely upon it.  If an application or utility uses it’s own trust store, then you’ll have to update that specific trust store as well. (For example, both Ruby and Java use their own trust stores).

You can use your browser to get a copy of the MITM certificate(s), or just do a cut and paste from the verbose output obtained above.  The text that appears between the markers

-----BEGIN CERTIFICATE-----

and

-----END CERTIFICATE-----

is a PEM (Privacy Enhanced Mail) encoding of the certificate (Base64).

If you used a browser, and the certificate was not already known, you’ll want to accept the certificate and be sure to check that box that says “permanently store this exception.” This will place a copy of the certificate into your browsers trust store. Either way, you can then export it from the browser trust store, and do an scp to copy it to the target machine, and import it into the required trust store. Depending upon the browser you are using you may be able to save the certificate directly in the PEM format, or you might have to first save it as binary file (DER format with a file extension of .der, .cer, or .crt). Once you have a copy of the certificate in one format then you can always use OpenSSL to convert the certificate to any other format, for example:

$ openssl x509 -inform der -in mitm-ca-cert.der -outform pem -out mitm-ca-cert.pem

Where to Find the Trust Store

It’s a security best practice to the store CA certificates trusted only by the local site separately from the CA certificates that are distributed with the OS.  On any recent Ubuntu/Debian installation, the certificate store for the CA certs that come with the distro will be located in  /etc/ssl/certs. The CA certificates to trust that are specific to the local site belong in /usr/local/share/ca-certificates.

As an aside, it’s also worth noting that on Ubuntu, the Firefox browser will store globally trusted certificates in /usr/share/ca-certificates/mozilla, while the user-specific CA certificates (that is, user personal preferences) are stored in the users profile, under ~/.mozilla/firefox.  To export certificates from Firefox, you can use the menu choice Edit /Preferences, and then choose Advanced button, and then the Certificates tab. In the case of Mac OS X, the certificates are located in /Users/<username>/Library/Application Support/Firefox/Profiles. Of course, if you’re on a Mac, you can just use the Keychain Access application to view and manage certificates, including an export in PEM format.

As regular readers of this blog already well know, the Java trust store is usually located in $JAVA_HOME/jre/lib/security/cacerts. You can use the java keytool to import the MITM certificates you need using a command line.  Using OpenJDK7 that looks something like the following:

$JAVA_HOME/bin/keytool -importcert -alias mitm-ca-cert -file mitm-ca-cert.cer -keystore cacerts

Note that the Java keytool has options for different keystore types (typically “jks”, or “pkcs12”) and you can import either the binary (.cer) or base64 (.pem) encoded certificates, as needed. For this post I’d like to stay focused on the native platform keystore, so I won’t say any more about Java here.  Instead we’ll discuss Ruby and Java in more detail in a subsequent post.

Updating the Trust Store

The /usr/local/share/ca-certificates directory should contain one file for each additional certificate that you trust, in PEM format. Just drop the needed mitm-ca-certificate.pem file(s) into this directory, and make sure that they are owned by root, and have a file permission mask of 644. Then run the c_rehash command to create the required hash entries.

$ sudo mv ~/mitm-ca-cert.pem /usr/local/share/ca-certificates/mitm-ca-cert.pem
$ sudo chown root:root /usr/local/share/ca-certificates/mitm-ca-cert.pem
$ sudo chmod 644 /usr/local/share/ca-certificates/mitm-ca-cert.pem
$ cd /usr/local/share/ca-certificates
$ sudo c_rehash .

That last command deserves some explanation.  The c_rehash command comes from OpenSSL.  For each PEM file found in the directory, it creates a SHA-1 hash of the certificate subjectName and then creates a new filesystem entry with that hash as the filename, which is created as a soft link to the PEM file containing that subjectName (within the same directory).  Basically, this is done for performance reasons. There can be many certificates in this directory, and this is a good way to locate what we need quickly.  These hashed filenames follow the format HHHHHHHH.D where “H” is a hexadecimal digit and “D” is an incrementing integer (the “D” is used just in case there are hash collisions).

Finally, not all applications and utilities are aware of using both /etc/ssl/certs and /usr/local/share/ca-certificates. That is, some programs will only look in /etc/ssl/certs. To make sure these programs are able to find the newly added MITM CA certificates, we can create some additional soft links on the file system:

$ cd /etc/ssl/certs
$ sudo ln -s /usr/local/share/ca-certificates/mitm-ca-cert.pem
$ sudo c_rehash .

This ensures that there are soft link entries in the /etc/ssl/certs directory that point to each of the new certificates dropped into /usr/local/share/ca-certificates, for those that look only in /etc/ssl/certs.

You can easily show the relationships between these files using a simple ls -la command.

$ ls -la /etc/ssl/certs

...<SNIP>...

lrwxrwxrwx   1   root  root      15    Mar 13 16:20   ad21de5b.0 -> mitm-ca-cert.pem

...<SNIP>...

lrw-r--r--   1   root  root    1245    Mar 13 16:20   mitm-ca-cert.pem -> /usr/local/share/ca-certificates/mitm-ca-cert.pem 

...<SNIP>...

Create as many PEM files and soft links as you need in order to establish the full chain of trust for any egress points that you will be using. That is, there may be an enterprise root CA, followed by a number of geographic or divisional CAs, who in turn issue an end-entity certificates to their SSL Termination router(s). The SSL/TLS protocol handshake will attempt to establish the chain of trust from the certificate that is presented by the counter party, to a trusted certificate found in the appropriate trust store.  This attestation may require one, two, or possibly more hops. YMMV.  You can always test whether you have the necessary CA certificate(s) in the right places, by explicitly naming a CA PEM file (or directory) on the OpenSSL command line:

$# A test to check if we have the CA cert needed to connect outbound
$  openssl s_client -connect targetsite.com:443 -CAfile mitm-ca-cert.pem

$# Now try it with the whole directory, instead of a specific file
$ openssl s_client -connect targetsite.com:443 -CApath /usr/local/share/ca-certificates

$# response should look like the following:

...<SNIP>...
Verify return code: 0 (ok)

If tests like these succeed then you have successfully established a complete chain of trust, and all of your future outbound SSL/TLS connections should complete without any issues.

Conclusion

Of course, it’s not our purpose here to defeat the SSL/TLS Termination, or to work around it.  Rather, we’re working hard to comply with it, so that we can succeed in making that outbound SSL/TLS connection.  We just want our Linux command line operations to succeed, so that we can go back to getting our real work done!