Get Me Out of Here!

…Or, How To Fix Your Outbound SSL/TLS Connection Problems

So, the other day I was working on an problem for a customer, and I hit up against a very common glitch using SSL/TLS going outbound. I’ve stumbled onto this more than once now, and so I thought that it would be helpful to share the fix. If nothing else, this post will serve as a record of the procedure so that I will be able to quickly reproduce my own fix next time.

The Context

Enterprises need to protect their perimeter, and will (almost) always use SSL Termination at their “edge” routers.  This means that any SSL/TLS connections you try to make going outbound are going to be intercepted and eavesdropped, in order to inspect the contents of the traffic. The actual infrastructure will vary of course, but the relevant network gear will typically be a box from, say, Cisco/Ironport, or an F5 Big IP, or BlueCoat, etc.

When you try to create an outbound SSL connection to, say, https://www dot any site dot com/downloads, you can expect that the network box handling your request will prematurely terminate that SSL/TLS connection, and examine the contents of the request in real-time. If the request looks OK, then the router will repeat your original request to that target URL.  In effect, your SSL/TLS connection is actually being subjected to a classic man-in-the-middle (MITM) attack.  Except that in this case it’s not really an attack — it’s being done to defend the enterprise, and to comply with the applicable government regulations.

As the old saying goes, you can’t manage what you can’t see.  And so the CISO couldn’t effectively claim “due care” compliance with data protection and privacy regulations when any user can create an opaque SSL/TLS connection going outbound from the enterprise. Think of security use case scenarios such as a disgruntled employee, or an undetected strain of malware, exfiltrating private customer data over an encrypted channel.

So there are good reasons for doing this, and in most cases the users will never even see it.  However, if your SSL/TLS request was initiated from a developer VM in the lab, using an Ubuntu command line tool — rather than from an enterprise managed browser — then your request will likely fail.

Where the Rubber Meets the Road

When you are working from your favorite browser, and you try to do an HTTPS GET from your favorite download site, the browser will check it’s local trust store to see if it recognizes the x.509 certificate that the target site has presented.  If the certificate for the site has been signed by a CA that your browser trusts, then all is well, and you’ll smoothly connect.

If not, then you may see a security warning message, asking you if you want to accept the new certificate.  Different browsers will behave differently, but in general you’ll see that type of security exception for any new, unrecognized certificate.  If you look carefully at the certificate contents you might notice that the certificate you’re being asked to accept has a subjectName which is consistent with the site you’re trying to reach.  However, the certificate issuerName is going to be the local enterprise security operations team, rather than a widely recognized CA.  So, if you’re on site working for a client named Acme Widgets, you’ll see something that effectively says: “Security warning:  Do you want to accept the certificate for issued by Acme Widgets, Inc.?”  “Hmmmm…”, you think, “I didn’t know Amazon got their PKI certificates from Acme?!”

They didn’t.  What’s happening is that your outbound SSL/TLS request has been intercepted, and a certificate was issued (often, on-the-fly) stating that the enterprise vouches for the target site.  Once you accept that certificate, your SSL session to the edge router is established.  And a new SSL session is then created, as a next hop, to the target site.  The trust for that second hop is established using the actual site certificate for the target site.  The enterprise’s edge router is acting as a man-in-the-middle.

Usually, the browser’s trust store has already been updated as part of installing the security infrastructure, and so the local enterprise CA is already considered a trusted authority.  The typical user won’t even see the warning message, and everything just works.  The user gets to do their uploads or downloads, and the enterprise gets to inspect exactly what content is traversing their infrastructure.

Beyond the Browser

So what’s the essential security architecture problem here?  Too many trust stores!

The annoying glitch occurs when you are initiating that SSL/TLS request from a program that does not use the browser’s trust store.  The first symptom is likely to be a connection timeout, excessive connection retries, or a stack trace complaining about network connectivity.  Of course when you try to debug this, and you connect to the target site using the browser, it just works.  Grrrr!  After investigating further you’ll find that the only time you can connect to the required site is from a browser, but never from any other command line utilities.  Again, the reason it just works from the browser is that the MITM CA certificate is already there, in the browser trust store.  But if you use a command line utility like curl or wget it will fail, because the MITM CA certificate is not in the corresponding trust store.

OpenSSL to the Rescue

You can use openssl to diagnose and fix this issue.  Do the following from a shell prompt:

$  openssl s_client -showcerts -connect

The result should be a fairly verbose (but fascinating) view of the SSL handshake negotiation in progress.  You’ll see that the client connect request is sent, and consistent with the protocol spec, the receiver presents the server’s certificate chain.  Adding -debug will show even more.  Looking carefully at that certificate chain will reveal that it is probably not the one you expect, and the last line of the output will appear as follows:

Verify return code: 20 (unable to get local issuer certificate)

Translated, this means that the CA certificate for the issuer (i.e. the enterprise MITM Certificate Authority) is not present in the default trust store.  Assuming the software you are running is leveraging the OpenSSL certificate store, then by updating that common store, you can fix any number of utilities or applications that rely upon it.  If an application or utility uses it’s own trust store, then you’ll have to update that specific trust store as well. (For example, both Ruby and Java use their own trust stores).

You can use your browser to get a copy of the MITM certificate(s), or just do a cut and paste from the verbose output obtained above.  The text that appears between the markers




is a PEM (Privacy Enhanced Mail) encoding of the certificate (Base64).

If you used a browser, and the certificate was not already known, you’ll want to accept the certificate and be sure to check that box that says “permanently store this exception.” This will place a copy of the certificate into your browsers trust store. Either way, you can then export it from the browser trust store, and do an scp to copy it to the target machine, and import it into the required trust store. Depending upon the browser you are using you may be able to save the certificate directly in the PEM format, or you might have to first save it as binary file (DER format with a file extension of .der, .cer, or .crt). Once you have a copy of the certificate in one format then you can always use OpenSSL to convert the certificate to any other format, for example:

$ openssl x509 -inform der -in mitm-ca-cert.der -outform pem -out mitm-ca-cert.pem

Where to Find the Trust Store

It’s a security best practice to the store CA certificates trusted only by the local site separately from the CA certificates that are distributed with the OS.  On any recent Ubuntu/Debian installation, the certificate store for the CA certs that come with the distro will be located in  /etc/ssl/certs. The CA certificates to trust that are specific to the local site belong in /usr/local/share/ca-certificates.

As an aside, it’s also worth noting that on Ubuntu, the Firefox browser will store globally trusted certificates in /usr/share/ca-certificates/mozilla, while the user-specific CA certificates (that is, user personal preferences) are stored in the users profile, under ~/.mozilla/firefox.  To export certificates from Firefox, you can use the menu choice Edit /Preferences, and then choose Advanced button, and then the Certificates tab. In the case of Mac OS X, the certificates are located in /Users/<username>/Library/Application Support/Firefox/Profiles. Of course, if you’re on a Mac, you can just use the Keychain Access application to view and manage certificates, including an export in PEM format.

As regular readers of this blog already well know, the Java trust store is usually located in $JAVA_HOME/jre/lib/security/cacerts. You can use the java keytool to import the MITM certificates you need using a command line.  Using OpenJDK7 that looks something like the following:

$JAVA_HOME/bin/keytool -importcert -alias mitm-ca-cert -file mitm-ca-cert.cer -keystore cacerts

Note that the Java keytool has options for different keystore types (typically “jks”, or “pkcs12”) and you can import either the binary (.cer) or base64 (.pem) encoded certificates, as needed. For this post I’d like to stay focused on the native platform keystore, so I won’t say any more about Java here.  Instead we’ll discuss Ruby and Java in more detail in a subsequent post.

Updating the Trust Store

The /usr/local/share/ca-certificates directory should contain one file for each additional certificate that you trust, in PEM format. Just drop the needed mitm-ca-certificate.pem file(s) into this directory, and make sure that they are owned by root, and have a file permission mask of 644. Then run the c_rehash command to create the required hash entries.

$ sudo mv ~/mitm-ca-cert.pem /usr/local/share/ca-certificates/mitm-ca-cert.pem
$ sudo chown root:root /usr/local/share/ca-certificates/mitm-ca-cert.pem
$ sudo chmod 644 /usr/local/share/ca-certificates/mitm-ca-cert.pem
$ cd /usr/local/share/ca-certificates
$ sudo c_rehash .

That last command deserves some explanation.  The c_rehash command comes from OpenSSL.  For each PEM file found in the directory, it creates a SHA-1 hash of the certificate subjectName and then creates a new filesystem entry with that hash as the filename, which is created as a soft link to the PEM file containing that subjectName (within the same directory).  Basically, this is done for performance reasons. There can be many certificates in this directory, and this is a good way to locate what we need quickly.  These hashed filenames follow the format HHHHHHHH.D where “H” is a hexadecimal digit and “D” is an incrementing integer (the “D” is used just in case there are hash collisions).

Finally, not all applications and utilities are aware of using both /etc/ssl/certs and /usr/local/share/ca-certificates. That is, some programs will only look in /etc/ssl/certs. To make sure these programs are able to find the newly added MITM CA certificates, we can create some additional soft links on the file system:

$ cd /etc/ssl/certs
$ sudo ln -s /usr/local/share/ca-certificates/mitm-ca-cert.pem
$ sudo c_rehash .

This ensures that there are soft link entries in the /etc/ssl/certs directory that point to each of the new certificates dropped into /usr/local/share/ca-certificates, for those that look only in /etc/ssl/certs.

You can easily show the relationships between these files using a simple ls -la command.

$ ls -la /etc/ssl/certs


lrwxrwxrwx   1   root  root      15    Mar 13 16:20   ad21de5b.0 -> mitm-ca-cert.pem


lrw-r--r--   1   root  root    1245    Mar 13 16:20   mitm-ca-cert.pem -> /usr/local/share/ca-certificates/mitm-ca-cert.pem 


Create as many PEM files and soft links as you need in order to establish the full chain of trust for any egress points that you will be using. That is, there may be an enterprise root CA, followed by a number of geographic or divisional CAs, who in turn issue an end-entity certificates to their SSL Termination router(s). The SSL/TLS protocol handshake will attempt to establish the chain of trust from the certificate that is presented by the counter party, to a trusted certificate found in the appropriate trust store.  This attestation may require one, two, or possibly more hops. YMMV.  You can always test whether you have the necessary CA certificate(s) in the right places, by explicitly naming a CA PEM file (or directory) on the OpenSSL command line:

$# A test to check if we have the CA cert needed to connect outbound
$  openssl s_client -connect -CAfile mitm-ca-cert.pem

$# Now try it with the whole directory, instead of a specific file
$ openssl s_client -connect -CApath /usr/local/share/ca-certificates

$# response should look like the following:

Verify return code: 0 (ok)

If tests like these succeed then you have successfully established a complete chain of trust, and all of your future outbound SSL/TLS connections should complete without any issues.


Of course, it’s not our purpose here to defeat the SSL/TLS Termination, or to work around it.  Rather, we’re working hard to comply with it, so that we can succeed in making that outbound SSL/TLS connection.  We just want our Linux command line operations to succeed, so that we can go back to getting our real work done!

How To Maintain RHEL Firewall Policies Using iptables

In my last post I described how to use JConsole to monitor an application that was running in a JVM on a remote host.  The main challenge I had encountered in this task was dealing with the network connectivity issues that always exist between the developer’s laptop (the client) and the application service, running on a server in the lab.  Specifically, we accounted for incorrect hostname to IP address resolution, and configuring the appropriate access policies for the Linux NetFilter firewall on the target RHEL machine. Long story short, the solution steps I described included instructions to edit the iptables file directly. That is  — being a hacker at heart — I just found it way simpler to open “/etc/sysconfig/iptables” using vi, and just edit the access policies in raw form, rather than using the proper interactive administration commands.

Of course, we claim to be security professionals around here, and so we want all our security policies to be managed appropriately: using standard, repeatable processes, and with a suitable audit trail.  So, as of today, I pledge no more editing iptables configuration files by hand. In this post, I’ll attempt to redeem myself, by describing the correct way to modify these firewall policies, using the iptables command line.

The iptables Concept of Operations

NetFilter is the linux kernel firewall. It is maintained by the command line tool “iptables”.  As we’ve already seen, the name of the relevant, underlying configuration file is “/etc/sysconfig/iptables.

The basic concept of operations is just like other firewalls — the system uses rules to decide what to do with each network packet seen.  These rules are maintained in a couple of data tables (hence, the name “iptables”).  There are actually three tables of interest, and these are called the “Filter” table, the “NAT” table, and the “mangle” table.  Each of these tables serves to administratively organize a set of rules needed for different purposes:

  • Filter Table – The rules in this table are used for filtering network packets.
  • NAT Table – The rules in this table are used for performing Network Address Translation.
  • Mangle Table – The rules in this table are used for any other custom packet modifications that may be needed.

In order to enable the remote debugging we would like to do, we’ll be working with the Filter table.

The next thing you need to know is that iptables operates using an event-based model. There are a number of predefined lifecycle events that occur between the time that a packet is received on a host network interface, and when it is passed through to a user process.  Similarly, there are predefined lifecycle events that occur after a packet has been sent outbound from a user process, but before it actually exits the host through the network interface.

The full set of lifecycle events is specified as follows:


It is important to note that not all combinations of lifecycle events and tables is supported.  More on this in a moment.

The PREROUTING lifecycle event is defined to occur just after the packet has been received on the local network interface, but before any other handling occurs. This is where we have first opportunity to affect the handling of an inbound packet.  This is the place to do things like alter the destination IP address to implement Destination NAT, or DNAT.  The packet is received and may be destined for a particular port and address on the local network.  You can write a rule that alters where the packet is delivered, i.e. sending it to a different destination IP address and/or port on the network.

The next lifecycle event is called INPUT. This event is defined to occur after PREROUTING, but before the packet is passed through to a user process.  This lifecycle event where we can choose to enforce rules like dropping packets that have been received from a known-bad address.

Conversely, the OUTPUT event occurs just after a network packet is sent outbound from a user process. An example application of using the OUTPUT lifecycle event could be to do outbound filtering, or accounting on network usage for an application doing an upload. This event provides us with an opportunity to affect the packet handling immediately after the user process has done the send, but before the packet has been transferred to the (outbound) network interface.

The POSTROUTING event occurs just before the network packet goes out the door, i.e. just before it actually leaves on the outbound network interface. This is our last chance to apply any policy to the packet. This is the right place to implement rules for Source NAT, or SNAT. For a system that serves the role of an internet proxy or gateway, we can use the POSTROUTING event as an opportunity to do apply a rule to set the source IP address of the outbound packet. One common use case is to use both PREROUTING and POSTROUTING events to prevent external hosts from seeing any internal IP address. Implementing both source and destination NAT enables a gateway host to expose only its own public IP, and keep the addresses of the internal hosts hidden from the outside world.

Finally, the FORWARD lifecycle event applies to packets that are received on one network interface, and will be sent right back out on another interface. Again, this is a common function for a proxy or gateway host. The FORWARD lifecycle event is relevent to those packets that are handled entirely within the network stack, and are not delivered to a user processes.

As a convenience, we refer to the set, or “chain” of policies that are associated with a specific lifecycle event by using the name of that event.  So, when talking about this we might say something like “we need to modify the INPUT rule chain of the Filter table.”

Again, it is important to note that not all of the combinations of tables and lifecycle events are supported. Only the combinations that are required (i.e. that are meaningful) are actually supported.

So, for the case of the NAT table, there are 3 rule chains that are meaningful:


For the Filter table, the following 3 rule chains are supported:


For the mangle table, all of the lifecycle events are supported.  This is the most general case, and thus the mangle table can be used in situations where the NAT table and the filter table are for some reason not sufficient.  I’ve never needed to deal with the mangle table in my production work, so I won’t cover that in detail here. In general, it is used for specialized processing requirements, such as being able to adjust status bits in an IP header (i.e. changing IP protocol header option flags).

Tell Me Where it Hurts

Recall that after the initial failure of JConsole to connect to our remote JVM, we speculated that we had a firewall issue. My educated guess was that the port I was trying to reach was being blocked by the iptables policies. So, the first step is to review the existing iptables policies to see what we have in place.

The command we need to show the existing firewall policies is as follows:

# iptables -L INPUT -n -v --line-numbers

Sample output would look something like:

Chain INPUT (policy ACCEPT)
num target  prot  opt  source    destination 
1   ACCEPT  all    --  anywhere  anywhere state RELATED,ESTABLISHED 
2   ACCEPT  icmp   --  anywhere  anywhere 
3   ACCEPT  all    --  anywhere  anywhere 
4   ACCEPT  tcp    --  anywhere  anywhere state NEW tcp dpt:ssh 
5   ACCEPT  tcp    --  anywhere  anywhere state NEW tcp dpt:rmiregistry 
6   ACCEPT  udp    --  anywhere  anywhere state NEW udp dpt:rmiregistry 
7   ACCEPT  tcp    --  anywhere  anywhere state NEW tcp dpts:irdmi:8079 
8   ACCEPT  tcp    --  anywhere  anywhere state NEW tcp dpt:webcache 
9   ACCEPT  udp    --  anywhere  anywhere state NEW udp dpt:webcache 
10  REJECT  all    --  anywhere  anywhere reject-with icmp-host-prohibited 
[root@localhost ~]#

The -L says to list the policies, and the -v means, as usual, to be verbose, and finally we ask to include ordinal line numbers in the output. The argument INPUT identifies the rule chain associated with the INPUT lifecycle event. By default iptables operates on the Filter table.

From this output one can verify whether the specific IP address, port number, and protocol that we need for JConsole will be ACCEPTed or REJECTED.  At this point, the port we need is not listed and so our connection attempt will not be ACCEPTed.

In order to add the new rule, we would do the following command:

# iptables -I INPUT 10 -m state --state NEW -m tcp -p tcp --dport 18745 -j ACCEPT

In this commands we are saying that we want to add an additional rule to the INPUT chain of the filter table (again, the filter table is the default if no table is specified). That is, when a NEW packet arrives over the protocol “tcp” for destination port (“dport”) 18735, we want the policy to be to to ACCEPT that packet. The -j actually means to “jump” to the indicated target. We choose to jump to the built-in rule ACCEPT. It’s important to note that the new rule should be made the tenth one, i.e. added after the existing rule found in position 9. That is why we used the –line-numbers option in the list command. The rules are processed in order so it is important to insert new rules in the proper place. For example, if we placed the new ACCEPT rule after a more general REJECT rule, then the ACCEPT rule will never be reached.

The -m flag is invoking the state module, which is followed by the option that indicates that we are interested in connection requests that are in the “NEW” state, (as opposed to, say, connections in the “ESTABLISHED” state.

The output of a -L list operation would now appear as follows:

Chain INPUT (policy ACCEPT)
num  target  prot  opt  source    destination 
1    ACCEPT  all   --   anywhere  anywhere state RELATED,ESTABLISHED 
2    ACCEPT  icmp  --   anywhere  anywhere 
3    ACCEPT  all   --   anywhere  anywhere 
4    ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpt:ssh 
5    ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpt:rmiregistry 
6    ACCEPT  udp   --   anywhere  anywhere state NEW udp dpt:rmiregistry 
7    ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpts:irdmi:8079 
8    ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpt:webcache 
9    ACCEPT  udp   --   anywhere  anywhere state NEW udp dpt:webcache 
10   ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpt:18745 
11   REJECT  all   --   anywhere  anywhere reject-with icmp-host-prohibited

Of course, using the above command, one could add as many additional rules as needed. Just specify the source and destination addresses, protocol, and port(s) as needed.

In order to delete a rule, we can again specify the line number of the rule to the delete command. Here we delete the 10th rule from the INPUT chain of the filter table.

# iptables -D INPUT 10

After doing an add or a delete, it’s a good idea to list the rules again, in order to make sure you have what you think you need, and the rules have been inserted or deleted as expected, and in the right order.

So, now that we’re all comfortable administering iptables firewall policies via a proper command line (or even better, via a version controlled script), there’s no longer any excuse to edit the iptables files directly using vi.  While that may be quick and easy, it is not necessarily a reliable, repeatable process.   And in this business, accuracy counts.  And, if we have to do something more than once, it always makes sense to automate it.