DougHughes.net

I’m sure that most of you have seen the ‘Request Tuning’ page on the Coldfusion Administrator. If not, take a quick peek here.

This seemingly innocuous page can make or break your web server performance under load. Lets think about a typical page request.

1. Request for a .cfm page comes to the web server
2. Web server passes it to the CFM engine
3. CFM engine has to either read the template from the disk and compile it, or compare the compiled in-memory template to the disk copy
4. CFM engine makes one or more requests to your database
5. CFM engine gets the database results, and renders HTML output
6. CFM engine passes the rendered HTML back to the browser
7. Browser makes additional requests for js, css, and image assets, which get pulled from the disk and returned to the browser.

Most of these actions get taken and completed so quickly we are not aware of them. When your site is under load, there may be X number of users that the CF engine is processing requests for all at once, where X is most often represented by the top number from that graphic, ‘Maximum number of simultaneous Template Requests’. Too many concurrent requests can put an extra burden on down-stream subsystems, and cause threads to be ‘waiting’ for replies from, perhaps, an overloaded disk array, a SQL server that has poor indexing, or even local network bandwidth. How might you tune this number such that you don’t have these type of downstream issues?

First of all, lets enable some basic metrics so we can see how many threads are running, and how many threads are waiting for something to happen. There are many posts out there detailing how to enable metrics, so rather than rehash them I will just link to a few.

• http://kb2.adobe.com/cps/191/tn_19120.html
• http://www.coldfusionmuse.com/index.cfm/2008/5/20/optimization-mike-brunt
• http://www.cfwhisperer.com/post.cfm/10-steps-to-a-stable-and-performant-web-application-step-2

In the last post there, Mike outlines the different metrics that can be enabled to show you what your threads are doing. The idea here is that, we want our threads to quickly handle a request (show as busy briefly, then go back to idle quickly). If our threads are showing as ‘Busy’ for extended periods, and you start to see your ‘delayed’ thread count climb, you have requests that are waiting for your threads to free up so they can be processed. you may think ‘That’s OK, I can increase my concurrent thread count and life will be grand!’, but this can actually cause you increased difficulties by giving your server, or a downstream component, more than it can handle at once. You can often determine what should be done by observing your CF or SQL server.

On windows, take a peek at task manager, and go to the performance tab.Click on ‘View’, then choose ‘Show Kernel Times’. On Linux, you can get this information using ‘top’ or ‘vmstat’. The red kernel line is showing you how hard you lower level ‘system’ is working. Think of the CPU as ‘Application’ work happening (the green), and the Kernel times as ‘Operating System’ work, such as disk I/O, network I/O, and communication to other attached devices or subsystems.A high kernel time, paired with a less than full CPU time, can indicate that your threads are getting stuck waiting for some form of I/O or system call, rather than working on your application code. Lowering your concurrent threads, to decrease kernel usage downstream, can often increase overall application performance by keeping your application working, rather than waiting for other operations to finish.

I think the important takeaway here is, simply looking at the Coldfusion server alone will not give you the full picture as to what is impeding full and fast performance of your application. JVM metrics can help you determine if there is a downstream issue by showing you threads that are waiting or staying busy, and investigating the system statistics on your various subsystems can help you isolate the location of a bottleneck when it occurs, and at times the best thing you can do is to decrease concurrent threads to remove the load from your downstream subsystems. Your Coldfusion system will only ever run as fast as its slowest component, and finding the slow subsystem is often more than 1/2 the battle!