There have been several blog posts recently about class-loading issues apparently linked the use of the Sun-Java 1.6 (6.0) JVM. Mark Mandel has a detailed article on this here.
Before seeing this article we had been working on optimizing a ColdSpring-ModelGlue-Reactor application. We blogged our progress in this article. We thought a reality check was a good idea and ran some load tests comparing this same application performance in Java 1.5 and 1.6. Once again we observed better performance in 1.6; here are the results. These results are for a 50 Virtual User (vUser) Test for 1 hour with 8 second think time (delay between clicks) comparing Java 1 5 to 1.6.
Firstly Java 1.5
Total Number of Clicks: 13,345 (0 Errors) Average Click Time of all URLs: 5,298 ms
Secondly Java 1.6
Total Number of Clicks: 15,563 (0 Errors) Average Click Time of all URLs: 3,348 ms
As we observed before, Java 1.6 performs better in load tests than Java 1.5 and this includes instantiation time. Here are two graphs which show the same behavior…
Java 1.5
Java 1.6
In these two 3D graphs, the more users that are the left axis of the graph, the better. In the 1.5 version, most users are in the middle right part of the graph, whilst in the 1.6 version there are more users in the left part.
One thing we are saying here is to load-test all applications before deployment, ideally across different environments and particularly before down grading to an earlier version of anything. In addition, consider the base architecture of all applications, at the end of all that we do, user experience is paramount, users have no interest or concern as to what coding paradigms we use.
DougHughes.net 
Comments on: "On CF8? Be Cautious About Rolling Back To JVM 1.5" (5)
We’ve also seen some instability with the server monitor (when profiling is on) when load testing and running a 1.5 jre. This is not confirmed but a hunch – as we simply moved forward without the profiling on.
LikeLike
@Dave did you roll back to 1.5 or did you start out with 1.5? Thanks for the blog post by the way.
LikeLike
I’m still curious about the difference in instantiation time. I have seen through personal experience CFC-heavy applications taking minutes, and I mean like five minutes, to initialize on 1.6, where the same applicaiton initializes on 1.5 take 30-45 seconds. Since this is such a massive difference I’m interested to know why you think 1.6 is faster than 1.5 on instantiation time. The only possible answer I could see would be that Sun fixed the classloading bug in 1.6 in the most recent update, and you’re running that update?
LikeLike
@Brian thanks for your comment. The application I am using to test against is based on a Production application and is built using ColdSpring-ModelGlue-Reactor so we could certainly class it as cfc based-centric. I have to say in my load-testing that I am very much more concerned with the operation of the application over time not just the first hits after a restart of some kind. The reason for that is because typically that is the main concern of clients, in my experience.
I had a question, in the application(s) you observed taking minutes to instantiate was there any analysis carried out on ongoing memory usage whilst the application was under load?
LikeLike
I would agree with you about the long-term load-handling capability being as if not more important than initialization time. However, the difference can be SO marked (I’ve heard from others of startup times approaching 10 minutes on 1.6, which is just insane) that it actaully has huge implications on the way you can run your site. Most people don’t have clusters or failover servers, so having an app take 10 minutes to start is 10 minutes of requests queuing up and 10 minutes of an inactive site. Which means that even if the longer-term load handling is better on 1.6, the unbearable startup time overwhelms it because it’s just too long, regardless of overall load handling.
No, these apps weren’t analyzed in any way, just me running them locally and being stunned at the difference in initialization time. Since initialization is single-threaded anyway (one would assume), load wouldn’t seem to have any real bearing on the init time since no other requests are going to be processed until the first one is done and the initialization is finished, correct?
I’d be interested to hear, in your test, what WAS the actual difference in startup time between the two JVMs?
LikeLike