Customizing the Cloud Foundry Java Buildpack

In this post I’ll describe the process of customizing the Cloud Foundry Java build pack for the fortressDemo2 sample application.

In general, the requirement to do build pack customization arises any time you need to make changes to an application’s runtime stack. In terms of some of the specific security issues that we care about around here, you will need to customize the build pack any time you have a requirement to configure the set of CA certificates that are held in the JSSE Truststore of the JDK.  Many enterprises choose to operate their own internal CA, and most have policies about which third-party certificate authorities will be trusted, and so this is a very common requirement. Similarly, you’ll need to customize the Java build pack if you want to implement JEE container-based security via a custom Tomcat Realm. Fortunately, the Cloud Foundry build pack engineers thought about these issues, and so the procedure is fairly straight-forward.  We’ll show the specific steps that you’ll need to do, and we’ll touch on some of the security considerations of maintaining your build packs.

The basic procedure that we’ll need to perform goes as follows:

  • Fork the existing Java build pack from the GitHub repository.
  • Update the  /resources/open_jdk_jre/lib/security directory with your custom cacerts JSSE trust store file.
  • Add the jar file that implements your custom JEE realm provider to the /resources/tomcat/lib directory.
  • Do a git commit, and then push these changes back up to your repository
    • (and/or re-bundle an offline version of the build pack archive)
  • Push the application up to Cloud Foundry, and specify your customized build pack as an option on the push command.

In the following paragraphs, we’ll go through each of these steps in a bit more detail.

Fork the Repo

This part is easy.  Just fork the existing Java build pack repository at GitHub and then, using your favorite Git client, clone your copy of the repository onto your local machine. Keeping your customizations in a public repository enables you to share your good work with others who need those changes, and makes it easy to merge any upstream changes in the future. Also, depending upon how much work you need to do, consider creating a Git branch for your changes. You’ll probably want to isolate the changes you make to the build pack, just as you would do with any other development effort.

Log into GitHub, visit https://github.com/cloudfoundry/java-buildpack and press the “Fork” button. After that, use your favorite Git client to clone your copy of the repository.  In my case, that looked like this:

 $ git clone https://github.com/johnpfield/java-buildpack.git

Then, (depending upon your working style), create a Git branch, and we can start making some changes.

Update the JDK Trust Store

The Cloud Foundry build pack engineers designed a simple way for us to modify the JDK security configuration details. This enables you to adjust policy details such as the trust store, the java.policy file, or the java.security file.

There is a /resources subdirectory just below the Java build pack project root.  Below that, there are subdirectories for the Oracle JDK, the OpenJDK, and Tomcat. We’re going to use the OpenJDK for our deployment, so we need to copy our trust store file into the /resources/open_jdk_jre/lib/security subdirectory. This file is traditionally called cacerts, or more recently, jssecacerts. Assuming you are moving this over from a locally tested JDK installation, this would look something like:

$ cp $JAVA_HOME/jre/lib/security/cacerts  ~/projects/java-buildpack/resources/open_jdk_jre/lib/security/mycacerts

Of course, before doing this you should probably use the JDK keytool command along with SHA checksums to confirm that this trust store file actually contains only the certificates you’ll want to trust. Once that’s been done, just copy the trust store over to the designated place. Similarly, you can also customize the contents of java.policy or java.security as needed, and copy those over.

Adding the custom Realm Provider to Tomcat

Adding our custom JEE realm provider means putting the appropriate implementation jar onto the Tomcat container’s class path. Our preferred provider is Fortress Sentry. Assuming this is being migrated from a standalone Tomcat installation using a recent release of Fortress Sentry, this would look something like:

$ cp $CATALINA_HOME/lib/fortressProxyTomcat7-1.0-RC39.jar ~/projects/java-buildpack/resources/tomcat/lib/fortressProxyTomcat7-1.0-RC39.jar

As described in the Tomcat docs, actually enlisting the custom realm can be done at the level of the individual applications, or for a virtual host, or for all of the applications hosted on that container. In my recent PoC I was doing this for the specific application, which means there was no other configuration needed as part of the java-buildpack. The application-specific scope of the custom realm means we only needed to add that configuration to the META-INF/context.xml file, within the application war file.

If this custom realm needed to be activated for the whole container, or a virtual host, then we would need to edit the configuration of the Tomcat server.xml, and move that updated server.xml file over to  /resources/tomcat/conf/server.xml.

Easy, Expert, or Offline

Cloud Foundry build packs support three operational modes, called “Easy,” “Expert,” and “Offline.” The Easy mode is the default.  In this mode, the staging process will pull down the current release of the build pack from the repository maintained by Pivotal. This will ensure that the application is run with the “latest-and-greatest” runtime stack, and you’ll always have the latest security patches. This mode “just works,” and is what is recommended for everyone starting out.

Expert mode is similar, except that you maintain your own copy of the repository, which can be hosted inside the enterprise. This will be initialized by creating a local replica of the official Pivotal repository. Of course, this has all the benefits and responsibilities of local control, i.e. you maintain it. The main motivation for Expert mode is that since the repository is inside the enterprise, the staging process does not need to download stuff from the public internet every time an application is pushed.

The “Offline” mode is pretty much what you would think. Rather than referencing an external repository during staging and deployment, you can work offline, i.e. without making any connections to a live repository. In this mode, you create a ZIP file that contains your complete build pack, and upload that to your Cloud Foundry deployment. When you subsequently push your application(s), you’ll specify that build pack archive by name. Of course, this approach ensures consistency and repeatability. None of the runtime stack bits will ever vary, until and unless you decide to upload a new ZIP file. But you also run the risk of falling behind in terms of having the very latest JDK or Tomcat security fixes. Another potential downside of these ZIP files is bloated storage requirements. If every application team maintains their own ZIP files — all containing the same Tomcat release — there is likely to be a lot of process redundancy, and wasted disk.

At the end of the day, each of the methods has it’s pros and cons, and you’ll need to decide what makes sense for your situation. For the purposes of this post, Easy and Expert are equivalent, as they are both online options, and it’s just a matter of the particular URL that is referenced. Offline mode requires the additional step of creating and uploading the archive.

Custom Build Pack, Online Option

Assuming you want to work in the “online” style, you should commit and push your build pack changes to your fork of the repository. i.e.

$ cd ~/projects/java-buildpack 
$ # Modify as needed. Then...
$ git add .
$ git commit -m "Add custom JSSE trust store and JEE realm provider."
$ git push

Then you can do the cf push of the application to the Cloud Controller:

$ cd ~/projects/fortressdemo2
$ mvn clean package
$ cf push fortressdemo2 -t 60 -p target/fortressdemo2.war \ 
-b https://github.com/johnpfield/java-buildpack.git

Your application will be staged and run using the online version of the custom build pack.

Custom Build Pack, Offline Option

To use an offline version of the custom build pack, you will first bundle the ZIP file locally, and then upload this blob to the Cloud Foundry deployment. Finally, you can do the cf push operation, specifying the named build pack as your runtime stack.

To do this you’ll need to have Ruby installed. I used Ruby version 2.1.2, via RVM.

$ cd ~/projects/java-buildpack
$ bundle install
$ bundle exec rake package OFFLINE=true

After the build pack is ready, you can upload it to Cloud Foundry:

$ cd build
$ cf create-buildpack fortressdemo2-java-buildpack \
        ./java-buildpack-offline-bb567da.zip 1

And finally, you can specify that Cloud Foundry should apply that build pack when you push the application:

$ cd ~/projects/fortressdemo2
$ cf push fortressdemo2 -t 90 -p target/fortressdemo2.war \
    -b fortressdemo2-java-buildpack

That’s it! You can confirm that the application is running using your custom JEE realm and JSSE trust store by examining your configuration files, and logs:

$ cf files fortressdemo2 app/.java-buildpack/tomcat/lib

The response should include the Fortress jar, and look something like this:

Getting files for app fortressdemo2 in org ps-emc / space dev as admin...
OK

annotations-api.jar                15.6K
catalina-ant.jar                   52.2K
catalina-ha.jar                    129.8K
catalina-tribes.jar                250.8K
catalina.jar                       1.5M
...
<SNIP>
...
fortressProxyTomcat7-1.0-RC38.jar  10.6K
...

And you can also confirm that your custom certificate trust store and policy files are actually being used:

$ cf files fortressdemo2 app/.java-buildpack/open_jdk_jre/lib/security

The response will look something like this:

Getting files for app fortressdemo2 in org ps-emc / space dev as admin...
OK
US_export_policy.jar            620B
mycacerts                       1.2K
java.policy                     2.5K
java.security                   17.4K
local_policy.jar                1.0K

Finally, it is important to note that the intent for any Java build pack is that it be designed to support a class of applications, and not just a single application. So having a build pack specialized for Fortress Sentry deployments is in fact a very plausible use case scenario. The above URL referencing my GitHub repository is real, so if you want to quickly deploy the fortressDemo2 application in your own Cloud Foundry instance, feel free to use that repository, and issue pull requests for any changes.