DougHughes.net

This blog post is drawn from a presentation I gave recently to the Maryland CFUG. It is in some ways slightly simplistic so I will apologize to those for whom this information is “old-hat”. However there are many ColdFusion users who are not aware that since ColdFusion MX, the JVM is indeed The Heart Of The Matter for both ColdFusion and JRun. In this post I am attempting to explain where all the pieces fit, how they work together and how to tune and troubleshoot.

What Is The JVM?

Sun Microsystems created Java as a high-level programming language which should run on as many operating systems and hardware variations as possible. The goal of “write once run anywhere” meant that a predictable encapsulated operating system, no matter what the underlying hardware and software, was required. The Java Virtual Machine (JVM or JRE) is a multi-threaded environment abstracting the actual computer operating system from the application code.

Main JVM Suppliers

These are the main suppliers of Java Virtual Machines, there are others.

• Sun Hotspot JVM
• IBM JVM
• BEA JRockit
• MAC OSRuntime for OSX
• Oracle JVM aka JServer – OJVM
• Microsoft JVM

ColdFusion And The JVM

Since ColdFusion MX 6.0-6.1, CF has been a Java Application which needs what is known as a Servlet Container to run on. The most popular Servlet Container for ColdFusion is JRun. In the Enterprise edition of ColdFusion, the actual Servlet Container to be used can be selected during the install. In non-Enterprise installs JRun is automatically used. The main point being that there is always a Servlet Container.

Java J2EE – Java EE Servlet Containers

This is a list of suppliers of Java servlet containers which are certified as being J2EE compliant. Tomcat is not shown because I don’t believe it is fully certified as such by Sun. However, Apache Geronimo will be the first open-source certified J2EE/JavaEE servlet container.

• Geronimo from Apache
• JBoss from Red Hat
• JRun from Adobe
• Oracle OC4J
• Sun Java System Application Server (Sun One).
• WebLogic Server from BEA
• WebSphere Server from IBM

Where everything sits-fits

This is a simple diagram showing where the Operating System, JVM, Servlet Container and Java Application Server sit in relativity.

ColdFusion and the JVM

When Allaire ported ColdFusion to run on Java (previously the core was C++) they realized there were already many copies of ColdFusion in use and needed to engineer some backward compatibility into the newly released version, CFMX 6.0 code-named “Neo”. If we refer to the diagram above we can see that ColdFusion (the yellow rectangle in case it is hard to read the text) runs inside the Java Servlet Container (often JRun).

By ColdFusion MX 6.1 (code named Red Sky) CF was a fully compliant J2EE application server which involves items such as a particular directory-file structures. However Allaire-Macromedia wanted to make sure that existing users of CF could upgrade easily and relatively painlessly. They also wanted to market ColdFusion to the Java community who were used to a particular directory structure. As added benefit for Adobe-Macromedia, this presented a marketing opportunity to have two versions of CFMX; a Standard-Server version and an Enterprise version.

The Standard-Server version would follow the existing CF install where there could only be one instance of ColdFusion on a physical server. The Enterprise version would follow the J2EE standard where multiple application server instances are common and normal.

One very important point to note here is that whether ColdFusion is installed as a Standard-Server version or as an Enterprise version, the JVM is still at the heart of all.

This need to support existing CF users led to the creation of a convoluted directory structure for the Standard-Server version, I will attempt to describe that below. This is a typical directory structure for a Windows install of CFMX 7 Standard-Server version. Note that I have highlighted the “runtime” directory. That directory actually contains JRun and this is visually confusing as in actual fact, ColdFusion is running inside JRun, not the other way around as it appears in this directory structure. In this install there can only be one ColdFusion server instance on a single physical server.

Once again this is a typical Windows install, this time for the Enterprise version and shows ColdFusion (which is inside the “servers” directory) as actually logically where it is inside the JRun4 directory structure. In this install, multiple versions of ColdFusion are possible on a single physical server.

Again, one very important point to note here is that whether ColdFusion is installed as a Standard-Server version or as an Enterprise version, the JVM is still at the heart of all.

Garbage Collection

The JVM has a fairly efficient memory management paradigm which is seen better that the typical Windows methodology. Sun JVM’s use a generational garbage collector with two young generations and an old generation. New objects are always allocated in the young generation part of the heap. Every time the young generation is full a minor garbage collection takes place. Minor garbage collections don’t take very long. Objects which survived some minor collections, because they are still referenced from the application, are promoted to the older generation.

Usually the older generation is full at some point too, and needs to be cleaned. This leads to a major garbage collection or Full Garbage (Full GC) collection that removes garbage from both the old and the younger generations. Full GC’s are “stop-the-world” events and need to occur only when absolutely necessary. I emphasize here the part in italics in the paragraph above “because they are still referenced from the application“.

In an efficiently written CF-Java web application objects created in memory should be cycled and collected (that memory allocation released) whilst the object is in the “new” part of the overall memory space. Web applications by nature have short object lifecycles because of the request-response nature of web applications. As a very general rule the more objects that end up in the “old” memory space the more often Full GC’s are likely to occur.

Garbage Collectors

There are three main Garbage Collector algorithms available to ColdFusion – JRun applications; here they are.

• XX:+UseParallelGC Use parallel garbage collection, this is typically the default collector for ColdFusion installations. This collector is also referred to as the throughput collector. It uses a parallel version of the young generation collector. The old (tenured) generation is still cleaned with the default collector. Note This option can not be used in conjunction with -XX:+UseConcMarkSweepGC.
• XX:+UseConcMarkSweepGC Use concurrent garbage collection. This collector is also referred to as the concurrent low pause collector. It collects garbage in the old (tenured) generation concurrently to executing the application. Note that this option can not be used in conjunction with -XX:+UseParallelGC. Instead you may combine it with -XX:+UseParNewGC
• XX:+UseParNewGC Instructs the VM to use a parallel collector for the young generation. This option can be used in conjunction with -XX:+UseConcMarkSweepGC. As a note I have found that using the ConcMarkSweepGC along with ParNewGC can cause longer Full GC’s, I tend to use ParNewGC on its own. To change the garbage collector one of the above arguments is added to the jvm.config file.

Full Garbage Collections (Full GC’s) – Try To Minimize

One of the first methods I always want to employ when troubleshooting is to ascertain how many Full GC’s are taking place. To do so I add these arguments to the jvm.config file, which is located in one of these directories depending on the installation:

• Enterprise – {drive-volume}jrun4bin
• Standard/Server – {drive-volume}CFusionMX7runtimebin

I add these five arguments to the jvm.config file, it is important to remove these arguments when verbose garbage collection output is no longer needed.

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:yourname.log*

*This will send Garbage collection output to the {std}-out.log and will create a log of all GC activity with whatever name you gave yourname in either. This output log will go to one of the following locations.

• Enterprise – {drive-volume}jrun4bin
• Standard/Server – {drive-volume}CFusionMX7runtimebin

Full GC’s are “stop-the-world” events and need to occur only when absolutely necessary.

Verbose Garbage Collection Log Example

Here is an example of the Verbose Garbage Collection log output:

par new generation   total 97088K, used 0K [0x10010000, 0x171d0000, 0x171d0000)
eden space 77696K,   0% used [0x10010000, 0x10010000, 0x14bf0000)
from space 19392K,   0% used [0x14bf0000, 0x14bf0000, 0x15ee0000)
to   space 19392K,   0% used [0x15ee0000, 0x15ee0000, 0x171d0000)
<span style="color: red;">tenured generation   total 932096K, used 11109K</span> [0x171d0000, 0x50010000, 0x50010000)
the space 932096K,   1% used [0x171d0000, 0x17ca95b8, 0x17ca9600, 0x50010000)
compacting perm gen  total 26624K, used 26606K [0x50010000, 0x51a10000, 0x58010000)
the space 26624K,  99% used [0x50010000, 0x51a0b8a0, 0x51a0ba00, 0x51a10000)
} , 0.1523967 secs]
<span style="color: red;">3691.704: [Full GC</span>  {Heap before GC invocations=8:
Heap

This Full GC was almost certainly unnecessary because at the time it ran only 1109k out of 932096k of the tenured (old) generation had been used. I also note the timestamp (3691.704) of every Full GC in the log. If I see a regular pattern it almost always means something is calling a Full GC perhaps using System.gc()

Methods To Safely Minimize Full GC’s

One pattern I have seen over and over with ColdFusion is that Full GC’s are very often occurring at regular 60 second intervals. Although I have never been able to pin this down fully, I suspect it is due to a Java process called RMI(Remote Method Invocation). So then what do we do to control these unnecessary “stop-the-world” Full G’s? There are two alternatives:

1. Disable Full GC’s using the -XX:+DisableExplicitGC jvm.config argument. I am not totally happy doing this and in many cases it will not work
2. Run Full GC’s at longer regular intervals. Applying these arguments instead of -XX:+DisableExplicitGC will cause Full GC’s every 10 minutes (600,000 ms).

Dsun.rmi.dgc.client.gcInterval=600000
Dsun.rmi.dgc.server.gcInterval=600000

Using Metrics Logging In Troubleshooting The JVM

I have found that enabling metrics looking is invaluable in troubleshooting ColdFusion and JRun application issues. Metrics logging will show the status of the CF-JRun instance at pre-selected time intervals in terms of number of threads in use and available memory. In order to enable metrics logging we need to edit the jrun.xml file, here:

• Enterprise install {drive-volume}JRun4servers{servername}SERVER-INF
• Server-Standard install {drive-volume}CFusionMX7runtimeservers{servername}SERVER-INF

Here are the settings I usually use by making these changes in the jrun.xml file. Firstly uncomment the service class…

Then set “metricsEnabled” to true, we will leave other settings to default…

Lastly I like to split the logs out to make reading them easier by this change…

Conclusion

I have found that the only way to really determine the correct garbage collector to use for your applications and what the correct arguments are to pass to the JVM, via the jvm.config file, to effectively manage garbage collections is to first determine how many Full GC’s are occurring and control that if there are too many. Then load-test the application whilst tuning the arguments passed to the JVM, via the jvm.config file.