How To Maintain RHEL Firewall Policies Using iptables

In my last post I described how to use JConsole to monitor an application that was running in a JVM on a remote host.  The main challenge I had encountered in this task was dealing with the network connectivity issues that always exist between the developer’s laptop (the client) and the application service, running on a server in the lab.  Specifically, we accounted for incorrect hostname to IP address resolution, and configuring the appropriate access policies for the Linux NetFilter firewall on the target RHEL machine. Long story short, the solution steps I described included instructions to edit the iptables file directly. That is  — being a hacker at heart — I just found it way simpler to open “/etc/sysconfig/iptables” using vi, and just edit the access policies in raw form, rather than using the proper interactive administration commands.

Of course, we claim to be security professionals around here, and so we want all our security policies to be managed appropriately: using standard, repeatable processes, and with a suitable audit trail.  So, as of today, I pledge no more editing iptables configuration files by hand. In this post, I’ll attempt to redeem myself, by describing the correct way to modify these firewall policies, using the iptables command line.

The iptables Concept of Operations

NetFilter is the linux kernel firewall. It is maintained by the command line tool “iptables”.  As we’ve already seen, the name of the relevant, underlying configuration file is “/etc/sysconfig/iptables.

The basic concept of operations is just like other firewalls — the system uses rules to decide what to do with each network packet seen.  These rules are maintained in a couple of data tables (hence, the name “iptables”).  There are actually three tables of interest, and these are called the “Filter” table, the “NAT” table, and the “mangle” table.  Each of these tables serves to administratively organize a set of rules needed for different purposes:

  • Filter Table – The rules in this table are used for filtering network packets.
  • NAT Table – The rules in this table are used for performing Network Address Translation.
  • Mangle Table – The rules in this table are used for any other custom packet modifications that may be needed.

In order to enable the remote debugging we would like to do, we’ll be working with the Filter table.

The next thing you need to know is that iptables operates using an event-based model. There are a number of predefined lifecycle events that occur between the time that a packet is received on a host network interface, and when it is passed through to a user process.  Similarly, there are predefined lifecycle events that occur after a packet has been sent outbound from a user process, but before it actually exits the host through the network interface.

The full set of lifecycle events is specified as follows:

  • PREROUTING
  • INPUT
  • FORWARD
  • OUTPUT
  • POSTROUTING

It is important to note that not all combinations of lifecycle events and tables is supported.  More on this in a moment.

The PREROUTING lifecycle event is defined to occur just after the packet has been received on the local network interface, but before any other handling occurs. This is where we have first opportunity to affect the handling of an inbound packet.  This is the place to do things like alter the destination IP address to implement Destination NAT, or DNAT.  The packet is received and may be destined for a particular port and address on the local network.  You can write a rule that alters where the packet is delivered, i.e. sending it to a different destination IP address and/or port on the network.

The next lifecycle event is called INPUT. This event is defined to occur after PREROUTING, but before the packet is passed through to a user process.  This lifecycle event where we can choose to enforce rules like dropping packets that have been received from a known-bad address.

Conversely, the OUTPUT event occurs just after a network packet is sent outbound from a user process. An example application of using the OUTPUT lifecycle event could be to do outbound filtering, or accounting on network usage for an application doing an upload. This event provides us with an opportunity to affect the packet handling immediately after the user process has done the send, but before the packet has been transferred to the (outbound) network interface.

The POSTROUTING event occurs just before the network packet goes out the door, i.e. just before it actually leaves on the outbound network interface. This is our last chance to apply any policy to the packet. This is the right place to implement rules for Source NAT, or SNAT. For a system that serves the role of an internet proxy or gateway, we can use the POSTROUTING event as an opportunity to do apply a rule to set the source IP address of the outbound packet. One common use case is to use both PREROUTING and POSTROUTING events to prevent external hosts from seeing any internal IP address. Implementing both source and destination NAT enables a gateway host to expose only its own public IP, and keep the addresses of the internal hosts hidden from the outside world.

Finally, the FORWARD lifecycle event applies to packets that are received on one network interface, and will be sent right back out on another interface. Again, this is a common function for a proxy or gateway host. The FORWARD lifecycle event is relevent to those packets that are handled entirely within the network stack, and are not delivered to a user processes.

As a convenience, we refer to the set, or “chain” of policies that are associated with a specific lifecycle event by using the name of that event.  So, when talking about this we might say something like “we need to modify the INPUT rule chain of the Filter table.”

Again, it is important to note that not all of the combinations of tables and lifecycle events are supported. Only the combinations that are required (i.e. that are meaningful) are actually supported.

So, for the case of the NAT table, there are 3 rule chains that are meaningful:

  • PREROUTING
  • POSTROUTING
  • OUTPUT

For the Filter table, the following 3 rule chains are supported:

  • INPUT
  • OUTPUT
  • FORWARD

For the mangle table, all of the lifecycle events are supported.  This is the most general case, and thus the mangle table can be used in situations where the NAT table and the filter table are for some reason not sufficient.  I’ve never needed to deal with the mangle table in my production work, so I won’t cover that in detail here. In general, it is used for specialized processing requirements, such as being able to adjust status bits in an IP header (i.e. changing IP protocol header option flags).

Tell Me Where it Hurts

Recall that after the initial failure of JConsole to connect to our remote JVM, we speculated that we had a firewall issue. My educated guess was that the port I was trying to reach was being blocked by the iptables policies. So, the first step is to review the existing iptables policies to see what we have in place.

The command we need to show the existing firewall policies is as follows:

# iptables -L INPUT -n -v --line-numbers

Sample output would look something like:

Chain INPUT (policy ACCEPT)
num target  prot  opt  source    destination 
1   ACCEPT  all    --  anywhere  anywhere state RELATED,ESTABLISHED 
2   ACCEPT  icmp   --  anywhere  anywhere 
3   ACCEPT  all    --  anywhere  anywhere 
4   ACCEPT  tcp    --  anywhere  anywhere state NEW tcp dpt:ssh 
5   ACCEPT  tcp    --  anywhere  anywhere state NEW tcp dpt:rmiregistry 
6   ACCEPT  udp    --  anywhere  anywhere state NEW udp dpt:rmiregistry 
7   ACCEPT  tcp    --  anywhere  anywhere state NEW tcp dpts:irdmi:8079 
8   ACCEPT  tcp    --  anywhere  anywhere state NEW tcp dpt:webcache 
9   ACCEPT  udp    --  anywhere  anywhere state NEW udp dpt:webcache 
10  REJECT  all    --  anywhere  anywhere reject-with icmp-host-prohibited 
[root@localhost ~]#

The -L says to list the policies, and the -v means, as usual, to be verbose, and finally we ask to include ordinal line numbers in the output. The argument INPUT identifies the rule chain associated with the INPUT lifecycle event. By default iptables operates on the Filter table.

From this output one can verify whether the specific IP address, port number, and protocol that we need for JConsole will be ACCEPTed or REJECTED.  At this point, the port we need is not listed and so our connection attempt will not be ACCEPTed.

In order to add the new rule, we would do the following command:

# iptables -I INPUT 10 -m state --state NEW -m tcp -p tcp --dport 18745 -j ACCEPT

In this commands we are saying that we want to add an additional rule to the INPUT chain of the filter table (again, the filter table is the default if no table is specified). That is, when a NEW packet arrives over the protocol “tcp” for destination port (“dport”) 18735, we want the policy to be to to ACCEPT that packet. The -j actually means to “jump” to the indicated target. We choose to jump to the built-in rule ACCEPT. It’s important to note that the new rule should be made the tenth one, i.e. added after the existing rule found in position 9. That is why we used the –line-numbers option in the list command. The rules are processed in order so it is important to insert new rules in the proper place. For example, if we placed the new ACCEPT rule after a more general REJECT rule, then the ACCEPT rule will never be reached.

The -m flag is invoking the state module, which is followed by the option that indicates that we are interested in connection requests that are in the “NEW” state, (as opposed to, say, connections in the “ESTABLISHED” state.

The output of a -L list operation would now appear as follows:

Chain INPUT (policy ACCEPT)
num  target  prot  opt  source    destination 
1    ACCEPT  all   --   anywhere  anywhere state RELATED,ESTABLISHED 
2    ACCEPT  icmp  --   anywhere  anywhere 
3    ACCEPT  all   --   anywhere  anywhere 
4    ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpt:ssh 
5    ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpt:rmiregistry 
6    ACCEPT  udp   --   anywhere  anywhere state NEW udp dpt:rmiregistry 
7    ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpts:irdmi:8079 
8    ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpt:webcache 
9    ACCEPT  udp   --   anywhere  anywhere state NEW udp dpt:webcache 
10   ACCEPT  tcp   --   anywhere  anywhere state NEW tcp dpt:18745 
11   REJECT  all   --   anywhere  anywhere reject-with icmp-host-prohibited

Of course, using the above command, one could add as many additional rules as needed. Just specify the source and destination addresses, protocol, and port(s) as needed.

In order to delete a rule, we can again specify the line number of the rule to the delete command. Here we delete the 10th rule from the INPUT chain of the filter table.

# iptables -D INPUT 10

After doing an add or a delete, it’s a good idea to list the rules again, in order to make sure you have what you think you need, and the rules have been inserted or deleted as expected, and in the right order.

So, now that we’re all comfortable administering iptables firewall policies via a proper command line (or even better, via a version controlled script), there’s no longer any excuse to edit the iptables files directly using vi.  While that may be quick and easy, it is not necessarily a reliable, repeatable process.   And in this business, accuracy counts.  And, if we have to do something more than once, it always makes sense to automate it.

How to Monitor a Remote JVM running on RHEL

Even when I’m not working on security, I’m still working on security.

I’ve recently been working on a large customer application deployment that has required some performance analysis and tuning. The usual first step in this process is to use a tool like JConsole, a very useful management and monitoring utility that is included in the JDK. In short, JConsole is an application developer’s tool that complies with the Java Management Extensions (JMX) specification. It has a nice GUI, and it allows you to monitor either a locally or remotely executing Java process, by attaching to the running JVM process via RMI. Of course, before you can attach to the running JVM you need to be appropriately authenticated and authorized. In this post I’ll provide a brief overview of the steps that were required to connect from JConsole running locally on my MacBook Pro, to the remote JVM process, which I was running on a RHEL v6 virtual machine in the lab.

Required JMX Configurations

Before you can run JConsole, there are a number of changes that need to be made to the JMX configuration on the target machine. In this specific case, I’m using OpenJDK on RHEL, and the relevant files are located in /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/lib/management.

The first step is to do a ‘cd’ into that directory, and edit the following files appropriately:

# cd /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/lib/management
# vi jmxremote.password
# vi jmxremote.access
# vi management.properties

There is a template file in the OpenJDK distribution called “jmxremote.password.template” that you can copy to “jmxremote.password” in order to get started. Note also that the permissions on that password file must be set correctly, or you’ll see a complaint when you start the JVM. While you are setting this up, be sure to do a ‘chmod’ to make this file read/write by the owner, only:

# chmod 600 jmxremote.password

In general, these configuration files contain good comments, and all that is really required is to uncomment the lines corresponding to the settings you plan to use. Just to get started, the easiest approach is to disable SSL, and use password authentication for a read-only user. You can edit jmxremote.password to contain your favorite username/password combination, and subsequently edit jmxremote.access to give that username appropriate access. In my case, this was just read-only access. Some sample lines from these two files follow:

#
#jmxremote.password:
#
architectedSecUser h4rd2Guesss!
#
#
#jmxremote.access:
#
architectedSecUser readonly

If you are on an untrusted network and you are planning to monitor a program that handles sensitive data, you’ll want to enable SSL for your RMI connection. Doing that means that you will need to go through the standard drill of configuring the JDK keystore and certificate truststore. I won’t go over those individual steps here since it would be beyond the scope of this post.  Hmmm…come to think of it, perhaps I should revisit the general topic of JDK keystore/truststore configuration in a subsequent post. The world can never have too many certificate tutorials 🙂

Listening on the Right Interface

The first real glitch I hit in this task was that the target RHEL machine had no resolvable hostname. This is actually pretty common with developer machines running in a virtualized setting. Machines are cloned and the IP address is changed, but frequently there is no unique hostname assigned, and DNS is never updated. Doing a ‘hostname’ command on the machine will yield something like “localhost.localdomain.” The problem with this situation is that when we run the target application JVM with JMX access enabled, it will be listening only on local looppback address 127.0.0.1, and won’t be accepting connections on the LAN interface. When we issue our RMI request from JConsole targeting the remote IP address on the local LAN (say, something like 10.0.0.22), we’ll see an error like “connection refused.”

To diagnose this situation, get a shell prompt on the target machine and issue the command:

# hostname -i

If it returns “127.0.0.1”, or “127.0.1.1”, or “localhost” you don’t have a proper hostname configured. You will either have to interactively update the hostname and/or edit the /etc/hosts file. Alternatively, you can ignore the hostname issue and just specify the IP address explicitly as a Java system property when you start the JVM. Here are the values I used when starting my JVM:

java \
-D<any other system properties needed>
-Dcom.sun.management.jmxremote \ 
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.port=18745
-Djava.rmi.server.hostname=10.0.0.22
-cp myApplication.jar myMainClass

You would now run JConsole on your local machine, and connect to the remote host, at the chosen IP address and port. If you are not using username/password authentication you can connect as follows:

# jconsole 10.0.0.22:18745

If you are using username/password authentication, you will have to enter your credentials in the New Connection… dialog box in the GUI. In most cases, that’s all there is to it.  But, of course, the connection still did not work for me.  Grrrr…those darned security guys! 😉

Configuring the Linux iptables Firewall Rules

Even after the target application JMV was up and running on the remote machine, and listenting on the correct address, I still could not get JConsole to connect to it from my local laptop. Since I was able to get an SSH session on the remote Linux box I immediately concluded that there had to be an issue with the specific port(s) JConsole was trying to reach…Hmmm…I just chose port number 18745 randomly (OK, it was really pseudo-randomly)… Maybe I’m hitting up against some firewall rule(s) for that port? Perhaps port 22 (SSH) is allowed, but port 18745 is not allowed? In fact, who knows what other dynamic ports JMX may be trying to open? So, in an attempt to determine what ports were being used, I next ran JConsole with some logging turned on.

To turn on the logging for JConsole, you can create a file named “logging.properties” in the directory from which you will be running JConsole, and set the configuration argument on the JConsole command line. Do the following to create the logging.properties file:

# cd /myproject
# touch logging.properties 
# vi logging.properties

Cut/Paste insert this into logging.properties:

handlers= java.util.logging.ConsoleHandler
.level=INFO
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
javax.management.level=FINEST
javax.management.remote.level=FINEST

Then, go ahead and start jconsole with:

jconsole -J-Djava.util.logging.config.file=logging.properties

Now, when you try to connect, you will see log output on your terminal window which reveals the dynamic port number that JConsole is trying to use for RMI lookup. An example is shown below:

Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] connecting...
Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] finding stub...
Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] connecting stub...
Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] getting connection...
Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] failed to connect: java.rmi.ConnectException: Connection refused to host: 10.0.0.22; nested exception is:
 java.net.ConnectException: Connection refused
Jan 28, 2014 1:57:56 PM RMIConnector close
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] closing.

In this case example, JConsole was attempting to create a connection to port 45219.  Now that we know that little tid-bit of crucial information, we can go ahead and update the Linux firewall policy in /etc/sysconfig/iptables to allow that specific port number.  Do the following:

# su root
# vi /etc/sysconfig/iptables
# add 2 lines similar to these to iptables policy.
-A INPUT -m state --state NEW -m tcp -p tcp --dport 18745 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 45219 -j ACCEPT

# service iptables restart

As shown, I needed to restart the firewall process after making the necessary policy changes.  After that, I was able to just reconnect from JConsole, and this time the connection to the remote machine succeeded, and I could proceed to monitor my application’s resource utilization.

Like I said, even when I’m not doing security, I’m still doing security.

Happy Remote Monitoring!