How to Monitor a Remote JVM running on RHEL

Even when I’m not working on security, I’m still working on security.

I’ve recently been working on a large customer application deployment that has required some performance analysis and tuning. The usual first step in this process is to use a tool like JConsole, a very useful management and monitoring utility that is included in the JDK. In short, JConsole is an application developer’s tool that complies with the Java Management Extensions (JMX) specification. It has a nice GUI, and it allows you to monitor either a locally or remotely executing Java process, by attaching to the running JVM process via RMI. Of course, before you can attach to the running JVM you need to be appropriately authenticated and authorized. In this post I’ll provide a brief overview of the steps that were required to connect from JConsole running locally on my MacBook Pro, to the remote JVM process, which I was running on a RHEL v6 virtual machine in the lab.

Required JMX Configurations

Before you can run JConsole, there are a number of changes that need to be made to the JMX configuration on the target machine. In this specific case, I’m using OpenJDK on RHEL, and the relevant files are located in /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/lib/management.

The first step is to do a ‘cd’ into that directory, and edit the following files appropriately:

# cd /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/lib/management
# vi jmxremote.password
# vi jmxremote.access
# vi management.properties

There is a template file in the OpenJDK distribution called “jmxremote.password.template” that you can copy to “jmxremote.password” in order to get started. Note also that the permissions on that password file must be set correctly, or you’ll see a complaint when you start the JVM. While you are setting this up, be sure to do a ‘chmod’ to make this file read/write by the owner, only:

# chmod 600 jmxremote.password

In general, these configuration files contain good comments, and all that is really required is to uncomment the lines corresponding to the settings you plan to use. Just to get started, the easiest approach is to disable SSL, and use password authentication for a read-only user. You can edit jmxremote.password to contain your favorite username/password combination, and subsequently edit jmxremote.access to give that username appropriate access. In my case, this was just read-only access. Some sample lines from these two files follow:

#
#jmxremote.password:
#
architectedSecUser h4rd2Guesss!
#
#
#jmxremote.access:
#
architectedSecUser readonly

If you are on an untrusted network and you are planning to monitor a program that handles sensitive data, you’ll want to enable SSL for your RMI connection. Doing that means that you will need to go through the standard drill of configuring the JDK keystore and certificate truststore. I won’t go over those individual steps here since it would be beyond the scope of this post.  Hmmm…come to think of it, perhaps I should revisit the general topic of JDK keystore/truststore configuration in a subsequent post. The world can never have too many certificate tutorials 🙂

Listening on the Right Interface

The first real glitch I hit in this task was that the target RHEL machine had no resolvable hostname. This is actually pretty common with developer machines running in a virtualized setting. Machines are cloned and the IP address is changed, but frequently there is no unique hostname assigned, and DNS is never updated. Doing a ‘hostname’ command on the machine will yield something like “localhost.localdomain.” The problem with this situation is that when we run the target application JVM with JMX access enabled, it will be listening only on local looppback address 127.0.0.1, and won’t be accepting connections on the LAN interface. When we issue our RMI request from JConsole targeting the remote IP address on the local LAN (say, something like 10.0.0.22), we’ll see an error like “connection refused.”

To diagnose this situation, get a shell prompt on the target machine and issue the command:

# hostname -i

If it returns “127.0.0.1”, or “127.0.1.1”, or “localhost” you don’t have a proper hostname configured. You will either have to interactively update the hostname and/or edit the /etc/hosts file. Alternatively, you can ignore the hostname issue and just specify the IP address explicitly as a Java system property when you start the JVM. Here are the values I used when starting my JVM:

java \
-D<any other system properties needed>
-Dcom.sun.management.jmxremote \ 
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.port=18745
-Djava.rmi.server.hostname=10.0.0.22
-cp myApplication.jar myMainClass

You would now run JConsole on your local machine, and connect to the remote host, at the chosen IP address and port. If you are not using username/password authentication you can connect as follows:

# jconsole 10.0.0.22:18745

If you are using username/password authentication, you will have to enter your credentials in the New Connection… dialog box in the GUI. In most cases, that’s all there is to it.  But, of course, the connection still did not work for me.  Grrrr…those darned security guys! 😉

Configuring the Linux iptables Firewall Rules

Even after the target application JMV was up and running on the remote machine, and listenting on the correct address, I still could not get JConsole to connect to it from my local laptop. Since I was able to get an SSH session on the remote Linux box I immediately concluded that there had to be an issue with the specific port(s) JConsole was trying to reach…Hmmm…I just chose port number 18745 randomly (OK, it was really pseudo-randomly)… Maybe I’m hitting up against some firewall rule(s) for that port? Perhaps port 22 (SSH) is allowed, but port 18745 is not allowed? In fact, who knows what other dynamic ports JMX may be trying to open? So, in an attempt to determine what ports were being used, I next ran JConsole with some logging turned on.

To turn on the logging for JConsole, you can create a file named “logging.properties” in the directory from which you will be running JConsole, and set the configuration argument on the JConsole command line. Do the following to create the logging.properties file:

# cd /myproject
# touch logging.properties 
# vi logging.properties

Cut/Paste insert this into logging.properties:

handlers= java.util.logging.ConsoleHandler
.level=INFO
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
javax.management.level=FINEST
javax.management.remote.level=FINEST

Then, go ahead and start jconsole with:

jconsole -J-Djava.util.logging.config.file=logging.properties

Now, when you try to connect, you will see log output on your terminal window which reveals the dynamic port number that JConsole is trying to use for RMI lookup. An example is shown below:

Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] connecting...
Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] finding stub...
Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] connecting stub...
Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] getting connection...
Jan 28, 2014 1:57:56 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] failed to connect: java.rmi.ConnectException: Connection refused to host: 10.0.0.22; nested exception is:
 java.net.ConnectException: Connection refused
Jan 28, 2014 1:57:56 PM RMIConnector close
FINER: [javax.management.remote.rmi.RMIConnector: rmiServer=RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[10.0.0.22:45219](remote),objID:[5380841b:143da2d37a6:-7fff, 403789961180333858]]]]] closing.

In this case example, JConsole was attempting to create a connection to port 45219.  Now that we know that little tid-bit of crucial information, we can go ahead and update the Linux firewall policy in /etc/sysconfig/iptables to allow that specific port number.  Do the following:

# su root
# vi /etc/sysconfig/iptables
# add 2 lines similar to these to iptables policy.
-A INPUT -m state --state NEW -m tcp -p tcp --dport 18745 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 45219 -j ACCEPT

# service iptables restart

As shown, I needed to restart the firewall process after making the necessary policy changes.  After that, I was able to just reconnect from JConsole, and this time the connection to the remote machine succeeded, and I could proceed to monitor my application’s resource utilization.

Like I said, even when I’m not doing security, I’m still doing security.

Happy Remote Monitoring!