DougHughes.net

Using hprof with Coldfusion to troubleshoot slow code issues

I’m sure everyone has run into an issue that is difficult to isolate on a given system. Sometimes all the code looks fine, and none of the .cfm templates are really standing up and shouting “I’m your problem!” What to do when you are stumped and not sure where else to look for your performance woes?
Enter HPROF.

HPROF is a java profiler that you can invoke by modifying your jvm.config file. On ColdFusion multi server, that config file lives by default at {coldFusion root}/bin/jvm.config. When you open it up, you will see a few configuration variables being set, such as java.home (this tells coldfusion which JVM to use when starting up), java.library.path (sets the location for shared libraries to be loaded), and more, but the one we are concerned with is java.args. By default it will look something like this:

java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/

Once you modify your jvm.config file with the HPROF arguments (simply add the -agentlib:hprof=cpu=samples, or the sites argument, after the -server argument), you are instructing the JVM to gather the profiler information that you specify (in the hprof args), then once the JVM shuts down, dump that collected data to a file. This file created, by default, is called java.hprof.txt, and will be located in your {coldfusion root}/bin folder. So, to recap:

1. Modify your jvm.config file to add the -agentlib arguments as defined in bold below
2. Restart coldfusion with the modified config file
3. Run a good sample of your application (my preference is to run some jmeter testing)
4. Stop coldfusion (which triggers the generation of the java.hprof.txt file)

HPROF is a tool that is meant to be used locally for development, or on a test or staging server. Running most of these arguments *WILL* impact performance of your application, so please do not run this on a production server unless all other avenues of investigation have been exhausted. HPROF has many different switches, but there are 2 that I have found to be the most useful so far.
-agentlib:hprof=cpu=samples
Adding this switch is asking the JVM to gether time data for each method call. The report that comes back will give you the java class that is consuming CPU the most, in a list that goes from highest consumer to lowest. In the following output sample, this client has some database contention going on that is holding up other coldfusion threads, causing scalability issues.

As you can see, the report goes top down, from the most CPU time spent in a method, down to the least (not listed here for brevity). It gives you the count (number of times this method was invoked), the trace address (this refers to a stack trace ID, which is listed prior to this pretty list in the output file, and can be helpful in finding how these methods are called), and the method itself (which will quickly show you what exactly is being invoked). To find the associated stack trace, simply copy the trace number from this table, then search for it within the report. With this sample output, all of the classes are java, but I have seen CFC’s in a stack trace show as large consumers of resources as well. This is a great investigative tool when you are not sure what is taking up so much CPU, or why your application is not running as quickly as it should.
-agentlib:hprof=heap=sites,depth=4
This jvm.args switch instructs hprof to provide an analysis of the contents of your heap memory at the time the application exits. The sites argument tells the JVM to group the output based on a stack trace of a specific depth. The depth argument requests a specific depth of stack trace (each depth level will make your file take that much longer, and be that much larger, so only ask for stack traces of a minimal depth to figure out where the code is originating from). This is a pretty poor sample, but if you have an object you are instantiating and its not going away, it will stand out very quickly when you view this output:

This output is telling you that 29% of the objects in heap are of a type byte[] (an array of bytes), that there are less bytes marked as live than there are allocated (some of the records have been marked by the garbage collector as ready to be collected), and that these byte[] objects were created with methods that are detailed in stack trace 461666. That stack trace looks like this:

TRACE 461666:
com.adobe.fontengine.font.FontByteArray.<init>(FontByteArray.java:56)
com.adobe.fontengine.font.FontByteArray.<init>(FontByteArray.java:63)
com.adobe.fontengine.font.opentype.FontFactory$TableToLoad.load(FontFactory.java:218)
com.adobe.fontengine.font.opentype.FontFactory$TableToLoad.load(FontFactory.java:259)

Sometimes these traces can be helpful in isolating the source of an issue, or at least giving you some direction from which you can base your investigation on. When you have a stumper, sometimes a direction is all you need to get on the path to a resolution.

Has Flash Player 10 killed Papervision 3D?

I’m just throwing that question out there. Please don’t expect to find the answer to this or really any quandary from reading one of my blog posts.
Up until this past weekend I hadn’t really played around with the Papervision open source 3d engine in over a year. I remember Papervision being the new hotness back them. Seems like every other blog post was some sort of cool new 3D texturing globe component or amazing 3D navigation using it. Then Flash 10 came out with it’s own somewhat native 3D functionality and the Papervision buzz wore off.

So what became of this popular 3D engine?

Well, from what I can gather, Papervision is continuing to be used on various 3D projects across a wide variety of Flash application online. Check out the New York Times 3D rendering of the 15th hole at the US Open Golf Tournament here. Or check out the radical 3D navigation in this UFC promotional site here.

So why is it still being used, isn’t it native in Flash 10?

1. Flash 10’s 3D capabilities fall short of the expectations for your client’s particular project. The new 3D capabilities of the Flash 10 are more of a distortion of a bitmap with perspective and not truly a 3D engine. In fact the developers of Papervision are activily looking at how to better take advantage of the new player to speed up and increase the performance of their 3D engine.
2. The penetration numbers for Flash 10 aren’t there yet. While the Flash browser plug in is pretty ubiquitous among internet-enabled desktops, the latest version with 3D support is only being reported on about 87% of devices compared with Flash 9 at 99%.

Is the project still evolving?

Papervision’s blog seems to be pretty active and the source code continues to be actively updated. However the actual project download (swc or zip) is dated at March of 0, there hasn’t been an issue reported in a while and the project wiki hasn’t been updated for about half of a year. So it would appear that some of the hype has at least slowed.

My $0.02?

I’m a fan of using some subtle 3D techniques to enhance an application. Flash player 10 provides me with enough resources to create some interesting transitions and navigation elements without the overhead of adding an additional truly 3D engine. And as for penetration, that would differ from client to client. So maybe Flash player 10’s 3D hacks are going to cover me for 90% of my needs. That being said, I’d still turn to Papervision for any project that actually had any involved 3D elements. I’m thinking of applications that would take advantage of spacial dimensions such as a cargo loading application or maybe a 3D product design application like a build your own wine bottle website.

What about you guys? Where do you draw the line? Do you think Flash 10 killed the Papervision 3D engine or maybe it’s momentum?

Apache's Jmeter Part III – Using a csv to replicate real traffic

Time for post number 3 regarding Jmeter, and today I will go over simulating users performing searches for keywords feeding from a CSV file. Lets start with a basic test plan, with one thread group, a single web request to google, a random timer, and a results tree listener so we can see what is going on. It should look something like this to begin with (refer to my earlier posts to get to this point, either Jmeter part I or Jmeter part II)

Now you need to create or obtain a CSV file containing your search terms you wish to run through (or users to login, items to view, whatever you would like to change during each request). I made a super simple one here, feel free to download or create your own, and name it searches.csv.

Download a sample searches.csv file.

Now, save that csv file in the same directory that you saved your test plan file. Add a new element as a child of your thread group, called ‘CSV Data Set Config’.

Set the filename to ‘searches.csv’ (or whatever you named your data file), set the Variable Names to ‘TERM’, delimiter to a comma, and stop thread on EOF to false, and sharing mode to ‘All Threads’. The sharing mode just sets the scope of where this variable name (defined as term above) will be available, inside this thread group, inside a single thread, or to the entire test on any thread.

Now we have to tell our individual search to use this CSV data, which is actually really simple to do. Go to your HTTPClient sampler, and in the HTTP parameters list, change the q value, which is currently a string ‘coldfusion’, to ${TERM}.

Now when you save your plan and run it, take a look at your results tree, and click on ‘Request’. You will see that each search sent to google uses the next item from your CSV data file to perform the test.

This is a great technique to use for simulating multiple user logins. Combine this with the cookie manager, and you can fully simulate users logging in, performing a series of actions, and logging out at the end.

Upcoming CFUnited Presentation – Performance Tuning: Before the JVM

I just wanted to remind folks of my upcoming presentation at CFUnited, entitled Performance Tuning Coldfusion: Before the JVM. I will go over tooling and techniques you can use to identify difficult to find source code issues, as well as the methodoligies I use when diagnosing issues on a customers server that is having performance difficulties.  I will show you how to obtain and analyze a heap dump using VisualVM, how to review method usage and trace method calls using HPJmeter, how to isolate and resolve MS SQL performance issues using the Database Engine Tuning Advisor, and generally how to isolate problematic code issues and apply logical thinking to arrive at conclusions which will allow you to resolve performance issues.
I had the oppertunity to give this presentation last night at the Mid Michigan CFUG in Lansing, and I believe it was very well received. Much note taking was going on, and I think everyone walked away with a few new idea’s they could apply to their servers right away. Hope to see you at CFUnited 2009!

IntelliJ IDEA Is Now My IDE of Choice. Heres Why.

I’ve been using IntelliJ IDEA almost exclusively for about a month now, so I feel pretty comfortable saying that it walks all over Eclipse in many ways. I thought I would blog a bit about this IDE and offer some reasons that you might download the trial and give it a whirl.

To start with, all of the expected features are here. That includes code insight and completion, version control integration, Maven integration, multi-language support, integrating debugging, snippets, etc. That’s all a given.

Beyond the fact that it has all the key features of any IDE, it has a number of very nice features that make it a highly productive tool. I’ll start with language support.

First off, its Groovy support is light-years beyond what is available in Eclipse. I’ve tried to use Eclipse to do Groovy work, and it is a constant battle. Each new version of the Groovy plugin breaks something else for every thing it fixes. I’m not knocking the guys who write the plugin, since they do it on their own time for free, and they deserve thanks and acknowledgement for their effort. The reality is simply that I had a lot of problems with it, either random hangs, compiling issues (especially joint Java-Groovy compiling), code insight issues (or lack thereof), etc. IDEA’s Groovy support is far and away the standard that all other editors should be striving for.

Next, its Flex and ActionScript support is, in many ways, superior to FlexBuilder 3. FlexBuilder 4 may outdo them in some areas (support for the Flex 4 SDK, the network monitor, etc.), so I may be using both tools when it comes to Flex and AIR development. But even so, IDEA’s Flex features are very good and definitely in the same league as FlexBuilder’s.

Finally, it has a pretty nice CFML plugin too! While it won’t be on the same level as the upcoming ColdFusion IDE from Adobe, its inclusion in the wide range of features in IDEA is a welcome addition.

Of course, I’m only touching on the languages I use the most. The list of supported languages is extremely long. And I should mention that its code formatting options are very detailed, which is great, because I’m somewhat particular about how I like my code to look.

The IDE is also very “intelligent” in the hints and suggestions it offers you when it comes to refactoring or fixing errors. For example, a quick keystroke will bring up a context-sensitive code generation menu like the one on the right. In another few keystrokes, you can implement all the methods in any interfaces, override methods, and more. And since the menu is dependant on what you’re doing, it’s kept short and sweet.

It also has context sensitive “tips” menu, that you can see on the right. Depending on what you’re doing, it will offer a number of relevant commands to test things out or simplify the code that you’re editing. In a similar vein, there is a “quick fix” menu that looks the same, but has a red light bulb, and shows up when something is wrong. It will suggest importing a missing class, creating a class, renaming the class or package, etc. It’s very handy since you can trigger the menu and then pick a fix in just a second.

As you may have guessed from what I’ve said so far, I’ve been working on some Java and Groovy projects, and this is another area where IDEA offers some very cool extras. While Java offers some of the most powerful tools you’ll find anywhere, the cost is always in configuring things, managing the JAR libraries, and just trying to keep track of what the heck is going on.

I’ve become fond of a number of features that IDEA includes to make working with Java and JEE applications much more pleasant. Normally, if you’re building a JEE application, you have a web.xml file that configures numerous core elements. It’s a nasty XML file, and even XML editing and syntax features (which are here, of course), don’t help that much. IDEA has a nice graphical view of the elements of a JEE web application which you can see on the right. Trust me, it’s a lot easier to look through than the XML!

I’ll say this now: anyone who does Java or Groovy web development and doesn’t use Spring is insane. Anyone who has used ColdSpring knows the power of dependency injection. Spring does that, plus just about anything else you can imagine. But whether you use annotations to configure the dependencies, or whether you use an XML configuration file, it can be hard to keep track of what depends on what. This is especially true on large projects. Well, wonder no more. IDEA has some very cool Spring tools, but to me the best one is a dynamically-generated dependency graph. And it works whether you use annotations, XML, or both.

Along the same lines, IDEA’s Hibernate and JPA support are excellent. It offers tools to generate Hibernate configuration files, does code-insight on annotations, and just generally makes it easy to use. But one very nice feature is similar to the Spring one I just mentioned: a graph of the object associations, generated from either annotations or the XML configuration.

To add one more Hibernate-related feature, it comes with an integrated HQL (Hibernate Query Language) editor and datasource browser, to make writing HQL statements very easy. Before, I’d have to write a unit test just to try out a query, whether I needed a test or not. Thankfully, those days are behind me! You can see a screenshot of the HQL editor below:

Now, some of these features are actually available to one degree or another in Eclipse, if you install the right plugins and don’t run into any problems. But that may be one of the major differences between IDEA and Eclipse: the overall cohesion of the tool. Eclipse is open-source and is geared at letting people write things to plug into it. Unfortunately, that means a lot of things can be flaky or poorly implemented. Managing Eclipse plugins and version upgrades can take up waaay more time than it should. Because IDEA is made by a single company, it looks and acts much more, well, solid and integrated. Let me be quick to point out that you can (and many do) write free, open-source plugins for IDEA. There is an entire plugin development community. But in general, things that are very useful are actually rolled into the product itself (with proper attribution of course) and maintained by the company going forward, to ensure a high level of quality.

Speaking of quality, let me touch briefly on the testing capabilities within the IDE. Put simply: they rock. Things like JUnit, TestNG, FlexUnit, and others are included by default. And as you can see from the image on the right, you not only get test results, but integrated statistics and code coverage data as well:

The final area I wanted to mention is in the code analysis area. The features built into IDEA in this area are really quite amazing. There are full sets of customizable analysis tools that find errors and problem areas in a large number of categories. Hopefully the screenshot below gives you an idea of the depth of the information that the IDE can give you.

On top of the analysis advice, it offers some nice extras like this dependency matrix, which gives you concrete information of how coupled your classes are and where any cyclic dependencies may lurk:

OK, I’m finished gushing over IDEA. I’m sure you get it by now: I think it’s pretty good. The only flag anyone I’ve talked to about it ever raises is that it isn’t free. Yes, it’s not free. It costs $250 for a single-developer license. Some people balk at that, but I really can’t believe it when they do. $250?! Considering how much this tool does, they could probably charge $2500 and people would still buy it. And as I alluded above, you do sometimes get what you pay for. Eclipse is free, that’s true, but as a result it can also be flaky, incoherent, and hard to manage, especially when you start piling on the plugins.

All that said, you can download a free trial of IDEA to see what the fuss is about, if you like. If your programming duties involve any Java or Groovy work, this will sell itself. The fact that it *also* has good Flex and CFML support, along with all the tools and features I’ve been raving about, made this a no-brainer for me. So, anyone else out there using IDEA? Or, if anyone decides to give it a try based on this blog entry, please let me know what you think!

Ant4CF Released Run Build Files Remotely and Interact with ColdFusion Services From Ant Scripts

About a year ago I announced and released CFAnt. CFAnt was my first crack at creating tasks for Ant to made deploying ColdFusion applications with Apache Ant very easy. I never got the impression it was widely used outside of Alagad, but we used it internally to deploy essentially all of our applications. Those of you who did use it are aware that it had some shortcomings. For example, it required that you place CFAnt directly in your application’s webroot. Also, it exposed, perhaps, hundreds of poorly documented tasks for managing ColdFusion configuration. So, while CFAnt worked well for Alagad, I wanted to find a better and easier way to deploy ColdFusion applications using Ant. This spawned the genesis of Ant4CF, which is officially the second release of CFAnt, but with a new name. Ant4CF fixes a number of confusing issues with CFAnt and makes it really, really simply to remotely execute build files and to interact with services exposed through Ant4CF. These services include a collection of components which proxy the Admin API features. Ant4CF is made up of three Ant tasks packaged into a JAR file and a collection of CFML files which you can place anywhere on your server. The most common location for these files is under the CFIDE directory. With those two things in place you can easily create a build script that kicks off another build script remotely on your server. For example, this is a basic build.xml file:

<project name="Local Build Script" default="Build">
     <target name="Build">
         <input message="Please enter CFAdmin password:" addproperty="cfAdminPassword" />
         <taskdef name="configure" classname="com.alagad.ant4cf.Configure" classpath="antlib/ant4cf-java-2.0-SNAPSHOT-jar-with-dependencies.jar" />
         <taskdef name="remoteant" classname="com.alagad.ant4cf.RemoteAnt" classpath="antlib/ant4cf-java-2.0-SNAPSHOT-jar-with-dependencies.jar" />
         <configure ant4cfUrl="http://yourapp.com/CFIDE/ant4cf" adminPassword="${cfAdminPassword}" />
         <remoteant antfile="remoteBuild.xml" properties="cfAdminPassword,dsnPassword,svnUsername,svnPassword" timeout="120" />
     </target>
</project>

This build file essentially gathers the ColdFusion admin password, defines two of the Ant4CF tasks, configures Ant4CF and then remotely runs a build script named remoteBuild.xml.

To expand on that a bit Ant4CF provides three tasks, two of which are shown above: configure and remoteant.

The configure task tells Ant4CF where to find the Ant4CF files on your server and the password used to access the ColdFusion administrator.

The remoteant task is the more interesting of these two as it specifies a specific ant file to upload to the remote server and run. You can specify a set of properties to pass into that file as well. The remoteant task can also keep track of other third party Ant tasks you’re using and will upload those JARs to the server.

This means that you can essentially write an Ant script that does whatever you want and use the remoteant task to run it on your server without change.

Ant4CF also provides a service task that looks like this:

<service component="adminapi.801.datasourceProxy" method="setMSSQL" property="result">
    <argument name="name" value="yourapp" />
    <argument name="host" value="10.248.79.47" />
    <argument name="database" value="yourdatabase" />
    <argument name="username" value="dbusername" />
    <argument name="password" value="${dsnPassword}" />
    <argument name="selectmethod" value="direct" />
</service>

The service task is remnient of the cfinvoke tag in ColdFusion. In the case of the service tag you specify a component and a method and pass arguments into it via the argument tag. Any results are returned into the property specified.

By default, Ant4CF comes with a proxy for the ColdFusion Administrator API. This means that you have all of the capabilities of the Admin API built into Ant4CF. The service example above creates or edits a new MSSQL datasource for you.

In my applications a typical build script will export the application from Subversion, change the ColdSpring configuration, export Model-Glue from SNV and ColdSpring from CVS. I then create my datasource, enable and clear the trusted cache and reload the application.

Want to know how I do this? I type “ant” in the command line. The rest is handled for my automatically!

If you’re interested in Ant4CF you can download it here: http://alagad.com/go/products-and-projects/ant4cf/ant4cf

Ant4CF is free, open source, and, believe it or not, documented!

Why setting the right concurrent thread count is important

I’m sure that most of you have seen the ‘Request Tuning’ page on the Coldfusion Administrator. If not, take a quick peek here.

This seemingly innocuous page can make or break your web server performance under load. Lets think about a typical page request.

1. Request for a .cfm page comes to the web server
2. Web server passes it to the CFM engine
3. CFM engine has to either read the template from the disk and compile it, or compare the compiled in-memory template to the disk copy
4. CFM engine makes one or more requests to your database
5. CFM engine gets the database results, and renders HTML output
6. CFM engine passes the rendered HTML back to the browser
7. Browser makes additional requests for js, css, and image assets, which get pulled from the disk and returned to the browser.

Most of these actions get taken and completed so quickly we are not aware of them. When your site is under load, there may be X number of users that the CF engine is processing requests for all at once, where X is most often represented by the top number from that graphic, ‘Maximum number of simultaneous Template Requests’. Too many concurrent requests can put an extra burden on down-stream subsystems, and cause threads to be ‘waiting’ for replies from, perhaps, an overloaded disk array, a SQL server that has poor indexing, or even local network bandwidth. How might you tune this number such that you don’t have these type of downstream issues?

First of all, lets enable some basic metrics so we can see how many threads are running, and how many threads are waiting for something to happen. There are many posts out there detailing how to enable metrics, so rather than rehash them I will just link to a few.

• http://kb2.adobe.com/cps/191/tn_19120.html
• http://www.coldfusionmuse.com/index.cfm/2008/5/20/optimization-mike-brunt
• http://www.cfwhisperer.com/post.cfm/10-steps-to-a-stable-and-performant-web-application-step-2

In the last post there, Mike outlines the different metrics that can be enabled to show you what your threads are doing. The idea here is that, we want our threads to quickly handle a request (show as busy briefly, then go back to idle quickly). If our threads are showing as ‘Busy’ for extended periods, and you start to see your ‘delayed’ thread count climb, you have requests that are waiting for your threads to free up so they can be processed. you may think ‘That’s OK, I can increase my concurrent thread count and life will be grand!’, but this can actually cause you increased difficulties by giving your server, or a downstream component, more than it can handle at once. You can often determine what should be done by observing your CF or SQL server.

On windows, take a peek at task manager, and go to the performance tab.Click on ‘View’, then choose ‘Show Kernel Times’. On Linux, you can get this information using ‘top’ or ‘vmstat’. The red kernel line is showing you how hard you lower level ‘system’ is working. Think of the CPU as ‘Application’ work happening (the green), and the Kernel times as ‘Operating System’ work, such as disk I/O, network I/O, and communication to other attached devices or subsystems.A high kernel time, paired with a less than full CPU time, can indicate that your threads are getting stuck waiting for some form of I/O or system call, rather than working on your application code. Lowering your concurrent threads, to decrease kernel usage downstream, can often increase overall application performance by keeping your application working, rather than waiting for other operations to finish.

I think the important takeaway here is, simply looking at the Coldfusion server alone will not give you the full picture as to what is impeding full and fast performance of your application. JVM metrics can help you determine if there is a downstream issue by showing you threads that are waiting or staying busy, and investigating the system statistics on your various subsystems can help you isolate the location of a bottleneck when it occurs, and at times the best thing you can do is to decrease concurrent threads to remove the load from your downstream subsystems. Your Coldfusion system will only ever run as fast as its slowest component, and finding the slow subsystem is often more than 1/2 the battle!