In my last post I described how to use JConsole to monitor an application that was running in a JVM on a remote host. The main challenge I had encountered in this task was dealing with the network connectivity issues that always exist between the developer’s laptop (the client) and the application service, running on a server in the lab. Specifically, we accounted for incorrect hostname to IP address resolution, and configuring the appropriate access policies for the Linux NetFilter firewall on the target RHEL machine. Long story short, the solution steps I described included instructions to edit the iptables file directly. That is — being a hacker at heart — I just found it way simpler to open “/etc/sysconfig/iptables” using vi, and just edit the access policies in raw form, rather than using the proper interactive administration commands.
Of course, we claim to be security professionals around here, and so we want all our security policies to be managed appropriately: using standard, repeatable processes, and with a suitable audit trail. So, as of today, I pledge no more editing iptables configuration files by hand. In this post, I’ll attempt to redeem myself, by describing the correct way to modify these firewall policies, using the iptables command line.
The iptables Concept of Operations
NetFilter is the linux kernel firewall. It is maintained by the command line tool “iptables”. As we’ve already seen, the name of the relevant, underlying configuration file is “/etc/sysconfig/iptables.”
The basic concept of operations is just like other firewalls — the system uses rules to decide what to do with each network packet seen. These rules are maintained in a couple of data tables (hence, the name “iptables”). There are actually three tables of interest, and these are called the “Filter” table, the “NAT” table, and the “mangle” table. Each of these tables serves to administratively organize a set of rules needed for different purposes:
- Filter Table – The rules in this table are used for filtering network packets.
- NAT Table – The rules in this table are used for performing Network Address Translation.
- Mangle Table – The rules in this table are used for any other custom packet modifications that may be needed.
In order to enable the remote debugging we would like to do, we’ll be working with the Filter table.
The next thing you need to know is that iptables operates using an event-based model. There are a number of predefined lifecycle events that occur between the time that a packet is received on a host network interface, and when it is passed through to a user process. Similarly, there are predefined lifecycle events that occur after a packet has been sent outbound from a user process, but before it actually exits the host through the network interface.
The full set of lifecycle events is specified as follows:
It is important to note that not all combinations of lifecycle events and tables is supported. More on this in a moment.
The PREROUTING lifecycle event is defined to occur just after the packet has been received on the local network interface, but before any other handling occurs. This is where we have first opportunity to affect the handling of an inbound packet. This is the place to do things like alter the destination IP address to implement Destination NAT, or DNAT. The packet is received and may be destined for a particular port and address on the local network. You can write a rule that alters where the packet is delivered, i.e. sending it to a different destination IP address and/or port on the network.
The next lifecycle event is called INPUT. This event is defined to occur after PREROUTING, but before the packet is passed through to a user process. This lifecycle event where we can choose to enforce rules like dropping packets that have been received from a known-bad address.
Conversely, the OUTPUT event occurs just after a network packet is sent outbound from a user process. An example application of using the OUTPUT lifecycle event could be to do outbound filtering, or accounting on network usage for an application doing an upload. This event provides us with an opportunity to affect the packet handling immediately after the user process has done the send, but before the packet has been transferred to the (outbound) network interface.
The POSTROUTING event occurs just before the network packet goes out the door, i.e. just before it actually leaves on the outbound network interface. This is our last chance to apply any policy to the packet. This is the right place to implement rules for Source NAT, or SNAT. For a system that serves the role of an internet proxy or gateway, we can use the POSTROUTING event as an opportunity to do apply a rule to set the source IP address of the outbound packet. One common use case is to use both PREROUTING and POSTROUTING events to prevent external hosts from seeing any internal IP address. Implementing both source and destination NAT enables a gateway host to expose only its own public IP, and keep the addresses of the internal hosts hidden from the outside world.
Finally, the FORWARD lifecycle event applies to packets that are received on one network interface, and will be sent right back out on another interface. Again, this is a common function for a proxy or gateway host. The FORWARD lifecycle event is relevent to those packets that are handled entirely within the network stack, and are not delivered to a user processes.
As a convenience, we refer to the set, or “chain” of policies that are associated with a specific lifecycle event by using the name of that event. So, when talking about this we might say something like “we need to modify the INPUT rule chain of the Filter table.”
Again, it is important to note that not all of the combinations of tables and lifecycle events are supported. Only the combinations that are required (i.e. that are meaningful) are actually supported.
So, for the case of the NAT table, there are 3 rule chains that are meaningful:
For the Filter table, the following 3 rule chains are supported:
For the mangle table, all of the lifecycle events are supported. This is the most general case, and thus the mangle table can be used in situations where the NAT table and the filter table are for some reason not sufficient. I’ve never needed to deal with the mangle table in my production work, so I won’t cover that in detail here. In general, it is used for specialized processing requirements, such as being able to adjust status bits in an IP header (i.e. changing IP protocol header option flags).
Tell Me Where it Hurts
Recall that after the initial failure of JConsole to connect to our remote JVM, we speculated that we had a firewall issue. My educated guess was that the port I was trying to reach was being blocked by the iptables policies. So, the first step is to review the existing iptables policies to see what we have in place.
The command we need to show the existing firewall policies is as follows:
# iptables -L INPUT -n -v --line-numbers
Sample output would look something like:
Chain INPUT (policy ACCEPT) num target prot opt source destination 1 ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED 2 ACCEPT icmp -- anywhere anywhere 3 ACCEPT all -- anywhere anywhere 4 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh 5 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:rmiregistry 6 ACCEPT udp -- anywhere anywhere state NEW udp dpt:rmiregistry 7 ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:irdmi:8079 8 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:webcache 9 ACCEPT udp -- anywhere anywhere state NEW udp dpt:webcache 10 REJECT all -- anywhere anywhere reject-with icmp-host-prohibited [root@localhost ~]#
The -L says to list the policies, and the -v means, as usual, to be verbose, and finally we ask to include ordinal line numbers in the output. The argument INPUT identifies the rule chain associated with the INPUT lifecycle event. By default iptables operates on the Filter table.
From this output one can verify whether the specific IP address, port number, and protocol that we need for JConsole will be ACCEPTed or REJECTED. At this point, the port we need is not listed and so our connection attempt will not be ACCEPTed.
In order to add the new rule, we would do the following command:
# iptables -I INPUT 10 -m state --state NEW -m tcp -p tcp --dport 18745 -j ACCEPT
In this commands we are saying that we want to add an additional rule to the INPUT chain of the filter table (again, the filter table is the default if no table is specified). That is, when a NEW packet arrives over the protocol “tcp” for destination port (“dport”) 18735, we want the policy to be to to ACCEPT that packet. The -j actually means to “jump” to the indicated target. We choose to jump to the built-in rule ACCEPT. It’s important to note that the new rule should be made the tenth one, i.e. added after the existing rule found in position 9. That is why we used the –line-numbers option in the list command. The rules are processed in order so it is important to insert new rules in the proper place. For example, if we placed the new ACCEPT rule after a more general REJECT rule, then the ACCEPT rule will never be reached.
The -m flag is invoking the state module, which is followed by the option that indicates that we are interested in connection requests that are in the “NEW” state, (as opposed to, say, connections in the “ESTABLISHED” state.
The output of a -L list operation would now appear as follows:
Chain INPUT (policy ACCEPT) num target prot opt source destination 1 ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED 2 ACCEPT icmp -- anywhere anywhere 3 ACCEPT all -- anywhere anywhere 4 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:ssh 5 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:rmiregistry 6 ACCEPT udp -- anywhere anywhere state NEW udp dpt:rmiregistry 7 ACCEPT tcp -- anywhere anywhere state NEW tcp dpts:irdmi:8079 8 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:webcache 9 ACCEPT udp -- anywhere anywhere state NEW udp dpt:webcache 10 ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:18745 11 REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Of course, using the above command, one could add as many additional rules as needed. Just specify the source and destination addresses, protocol, and port(s) as needed.
In order to delete a rule, we can again specify the line number of the rule to the delete command. Here we delete the 10th rule from the INPUT chain of the filter table.
# iptables -D INPUT 10
After doing an add or a delete, it’s a good idea to list the rules again, in order to make sure you have what you think you need, and the rules have been inserted or deleted as expected, and in the right order.
So, now that we’re all comfortable administering iptables firewall policies via a proper command line (or even better, via a version controlled script), there’s no longer any excuse to edit the iptables files directly using vi. While that may be quick and easy, it is not necessarily a reliable, repeatable process. And in this business, accuracy counts. And, if we have to do something more than once, it always makes sense to automate it.