The amazing adventures of Doug Hughes

Archive for January, 2008

My new favorite Eclipse feature

Thanks to Brian Ghidinelli, I now have a new favorite Eclipse feature.

I know a lot of you probably know about this, but this could be one of the biggest time savers that doesn’t invovle actaully typing code. If you use ctrl+Shift+R (Sorry Mac users, I am not sure what you use in place of Ctrl, command maybe?) a window pops up that allows you to search for files in any open project. You just start typing and you will see the list of files that match what you typed, you can then select the file from the list to open it. This is a huge time saver if you are stepping into an application that you may not be familiar with, or you are working on a large application with a lot of directories and/or sub-directories.

As a runner-up , James Allen, told me of Ctrl+E, which will open a pop-up of all the open files and allows you to filter the list by entering the file name.

So, what is your favorite Eclipse feature or keyboard shortcut?

Looking For a Crystal Reports Contractor

Im trying to track down a programmer with some experience with Crystal Reports to work on contract with one of my clients. If this is you please send me a message at Or, if you know someone who might be interested please have them contact me.


Model-Glue event-hander?

Has anybody ever heard of a Model Glue “event-hander”? (Notice the misspelling of the word “handler) Apparently Model Glue 1.1 has. For a bit of background, I was recently working with an old Model Glue 1.1 application and we decided it was time to upgrade it to Model Glue Unity. Upon doing so, several of the events quit working and started giving the standard “There is no known event handler for …” error message.

For the life of me, I could not figure out why only a few events in this application “vanished”. I spent several hours tracing things, dumping errors, view states, and everything in between trying to figure out what was wrong. Finally in desperation, I started over – at least as much as I could. I create a new “test” event handler which worked just fine. Then, I started moving functionality out of one of the broken event handlers into the test one and everything still worked just fine. I even changed the name of the test event handler to match the broken event handler – and it worked.

At that point, I happened to notice the lines were different lengths between the original event handler tag and the new one I had just created. The new one looked like this:

<event-handler name="AssignCase">

and, the old one looked like this:

<event-hander name="AssignCase">

Anybody see the difference?

After finding this, I still cannot figure out how the application has been running all of this time. I went back and checked out Subversion logs and this typo had been in place for more than a year and the whole time the application worked just fine under Model Glue 1.1. I think Joe has some explaining to do here …..

ColdFusion Won't Start – What Do We Do Next?

This is a short but I hope sweet blog post with some information which has saved me a lot of grief many times. One of the many things I really love about CF is the logging capabilities. Even with all of the great tools we have at our disposal; Server Monitoring, Fusion Reactor, SeeFusion etc, I still use the logs day in and day out.

However, let’s say for instance that one day we can’t start CF and we get a message like this…

"Windows could not start the Adobe {instance-name} on local computer. For more information, review the System Event Log.  If this is a non-Microsoft service, contact the service vendor, and refer to service specific error code 2."

If we look to the ColdFusion logs for help, we won’t find anything because CF never started. However, usually JRun or whichever J2EE container we are using usually has logged something for us in what is known in the Java world as the std-out.log. Here is where this is located:

In an Enterprise install:


In a Standard-Server install


Where {instance-name} is the name of the instance.

My example here shows MX7 and a Windows install, just extrapolate this if that is not your specific O/S and CF version.

Don't Throw ColdFusion Out! We Can Help.

I am just going to summarize quickly what we did to tune a client CF Instance over the past week. What we did came from load-testing in our LA Lab that was carried out against the client application which had been running very slowly. This was on a Windows 2003, CF8 Enterprise install. The application was created using Model Glue, Reactor and ColdSpring. As a note point, we effected a good improvement in the application without making any code changes as yet, the improvements we got are simply from JVM-server side tweaks.

The first thing we did is add verbose garbage collection logging, this showed me that we were getting almost out of memory before a major garbage collection or Full GC. This was with the default 1.6 JVM which comes with CF8. At this point we switched to the 1.5 JVM which is recommended in some circles for “cfc-heavy” applications. This actually gave no improvement.

Analyzing the verbose garbage collection statistics we found that the permanent part of the generation, “permgen” (which is where JRun-CF sit) was running out of memory regularly. So we added a start size to permgen of 128MB and added an argument to run explicit Full GC’s every 5 minutes as with Java 1.5 and 1.6 Full GC’s do not seem to run Till we are literally almost out of memory.

I also added a start size of 512MB to the overall heap size. The permgen, by the way, is in addition to the overall heap, not part of it. I also made two changes in CF Admin, I turned off the saving of class files and turned on Trusted Cache. Trusted cache is not the beast it used to be and is well worthy of use in Production environments. The net effect was this, when I started a 90 minute 50 concurrent user load test on the client application before the changes we got these results…

  • Total Number of Clicks: 17,136 (10 Errors)
  • Average Click Time of all URLs: 7,638 ms

After the various changes shown above the same loadtests produced these results…

  • Total Number of Clicks: 26,153 (0 Errors)
  • Average Click Time of all URLs: 2,061 ms

I believe that probably 90% of CF web sites out there need to have the server/JVM tuned (actually whilst I was with Allaire-MM every server- JVM<CFMX on> to needed to be tuned). This issue is that few people look in the logs at all and if there is slowness they typically throw hardware at it and sometimes throw ColdFusion out. If you are having performance problems or if you just want to know what is going on with the JVM and Garbage Collection, ping us, we really can help.

Really Fixing Applications – Are We Inside Out?

Most web application infrastructure is far from optimally configured, I have seen that so many times in so many places in some of the World’s largest/most famous organizations. Some of that happens because, with the possible exception of network infrastructures and operating systems, companies will usually install items with the factory configured defaults and never really try to figure out what is optimal for their application needs. They never look in the core of application servers, for instance, in the logs, the configuration files.

Typically the reason is poor documentation and a general lack of good training-tutorials which creates a natural fear of touching anything. In the CF 8 World we are more fortunate than many because we have the CF Administration GUI to help in configuring and monitoring our applications. However, I still find most clients hardly touch anything beyond a few familiar settings.

In ColdFusion 8 Enterprise edition there are 167 different settings which are available via the Administrator GUI. There has never been any really effective training for this and documentation is nowhere near as complete as it is for ColdFusion development. So, typically few settings are changed from the defaults. Also, in a single CF Instance there are 151 xml files and in some cases settings cannot be changed from the GUI but need to be changed manually, for instance, adding metrics logging.

It is also unfortunate that many do not even look at CF or JRun logs and in lots of cases I had clients who did not know they even existed. The net results of all these things is that in trying to fix a slow or unresponsive application most people I had been sent to help typically look first in the code.

There is no doubt that there are often problems in the code but if the server core, the “engine” is not tuned properly, we are actually starting in the wrong place by looking in the code. Here at Alagad we are focused on increasing knowledge of the server side. We have already blogged several times around that subject and will continue to do so. In addition will shortly be launching two new service offerings which at the same time as helping to fix problems will also help all of us to learn more about “the engine room”. These offerings are:

  • Alagad Log Analytics – we will scour your logs and give you a report pinpointing potential and/or actual problems. We will also transfer our years of knowledge about log management and analysis to you.
  • Alagad Garbage Collection Analytics – in this offering we will work with you to configure verbose garbage collection logs and then analyze the results making and explaining the reasons, again another point of knowledge transfer. Efficient Garbage Collection is the key to highly efficient J2EE application servers.

In the longer term we are also looking at developing training sessions for all aspects of server management and optimization, including a good meaty session on ColdFusion Administration. Keep an eye on our blog, we really are dedicated to passing on our knowledge.

Is This Domain Name Fraud?

On Monday I was looking for a new domain name for a friend of mine and I searched and found all main extensions were available; .com, .net, .org. So I told my friend and early on Tuesday we went to purchase the .com and .net versions. To our surprise the .com version had been registered the day before and in Whois it was shown as client parked and registered by the largest most expensive registrar on Monday. Everywhere else I tried it was shown as taken and in one case it had a flag saying “Make an Offer”.

So for curiosities sake I went to the largest most expensive registrar’s site and searched for .com version and hey-presto, it was available. My friend wanted the .com version so I went ahead and bought it. The cost was over twice that of the nearest competitor. Today, Crystaltech, whom I have a lot of respect for, emailed their clients pointing out that they suspected exactly what I suspected might be happening.

When someone searches for availability on a Domain Name it is being purchased/held directly after that search by a registrar and either being offered for sale or being made available by the largest most expensive registrar. If this is what is happening I don’t know if it is legally fraud but is is certainly very unethical. Hat’s off to Crystaltech for trying to protect and warn their clients.

Alagad Is Hiring a Senior Technical Sales Representative

Alagad provides a range of website and web application development services to companies of all sizes. At Alagad we never sit still. We’re constantly creating web sites, applications, tools and more. And when we’re not doing that, we&rsquo;re writing presentations and contributing to open source projects.

Last year our company grew by 300%. We are on track for similar or better growth this year. Additionally, one of our primary competitors was listed on the INC500 fastest growing companies for 2007. We’re a smaller company right now, but we believe we can do anything they can do better. There’s no reason we can’t have similar growth. This would be strongly driven by you. This years sales goals are currently more than $1,000,000.

Senior Technical Sales Representative

We are in need of a highly talented and well connected sales person to sell internet applications, web development services and technical training to customers of all sizes, including the government. The ideal candidate will have a strong familiarity with Adobe development technologies including ColdFusion, Flex, Air and Livecycle. They will also have sold complex internet applications built using these tools in previous jobs.

The position is a telecommuting job. We don’t care where you’re located in the US. We don’t care where you work during the day. All we care is that we can reach you and that you consistently produce excellent results. You should be comfortable working remotely though. We frequently use Google Apps, Adobe Connect, and Instant Messaging for communication.

Our company makes a habit of hiring some of the best people in the Adobe toolset niche. We can do this because we have no central offices and very low overhead. We can afford to pay above average for the most talented programmers. We believe our reputation will be an asset when selling our services.

When you join us you will be the first sales person here. Historically, we’ve grown through word of mouth. Seeing as we’re very well connected with the community of developers who use the same technologies as we do, we tend to get a fair number of referrals. This should help you get started, but we want you to take us beyond the word of mouth marketing and into new opportunities that would not have necessarily fallen in our laps by themselves.

We will favor candidates who have worked in technical sales at other competing or similar companies with related expertise.


We have a very open an connected culture here. The company regularly reviews our goals and the company’s financial status with its employees. You will be constantly informed of where we are. At the moment the company is targeting 20% profit for this year. If this works out as planned then in late 2008 or early 2009 we will begin a profit sharing plan across all company employees which will be beyond your salary and commission.

The position will be paid with a base plus commission. You should be aware that our employees are paid hourly here. What that means is that if you work more than 40 hours per week we’ll pay your for your extra time. There’s no reason you should not be able to accomplish 40 hours of work in a week. Some of our employees have even elected to work 60 hour weeks.

At our company you will earn PTO according to the number of hours you work. For each hour you work an additional 7.5% that goes into your PTO account. This means that in an average 160 hour month you’ll earn 12 hours of PTO. And, because this is calculated based on the number of hours you work, the more you work, the more you get in PTO.

You can take your PTO whenever you want or you can collect it in cash. PTO time never expires. We quite literally put this time in the bank for you. You can collect PTO for your entire career with Alagad. When you leave you’ll get any left over time in your final check.

Also, you should be aware that we do not offer any standard holidays off. These are already factored into your PTO. We’re allowing you to take whichever days you want off and to work whichever days you wish to.

We also have a generous collection of benefits. We offer heath, vision and dental insurance, flex spending accounts, 401k (we don’t match yet – but will soon), 529 college savings plan, training programs, life insurance, adoption assistance, and more.

To Apply

To respond to this opportunity, please go to:

Please be aware that Administaff is not a staffing agency.&nbsp;

Know Any Good SEO and Web Marketing Contractors?

From time to time I have a need for Search Engine Optimization (SEO) work.&nbsp; It seems that a lot of small businesses out there have websites which get absolutely no traffic.&nbsp; From time to time I’ll get a call or an email asking for help with this.&nbsp;

This isn’t a core competency for Alagad and it would be nice to have someone we can call in to help us.

Do you know anyone who might fit the bill?&nbsp; If so, please point me to them or have them contact me at (888)Alagad4 or though the Alagad site contact us form.

The Heart Of The Matter – The JVM

This blog post is drawn from a presentation I gave recently to the Maryland CFUG. It is in some ways slightly simplistic so I will apologize to those for whom this information is “old-hat”. However there are many ColdFusion users who are not aware that since ColdFusion MX, the JVM is indeed The Heart Of The Matter for both ColdFusion and JRun. In this post I am attempting to explain where all the pieces fit, how they work together and how to tune and troubleshoot.

What Is The JVM?

Sun Microsystems created Java as a high-level programming language which should run on as many operating systems and hardware variations as possible. The goal of “write once run anywhere” meant that a predictable encapsulated operating system, no matter what the underlying hardware and software, was required. The Java Virtual Machine (JVM or JRE) is a multi-threaded environment abstracting the actual computer operating system from the application code.

Main JVM Suppliers

These are the main suppliers of Java Virtual Machines, there are others.

  • Sun Hotspot JVM
  • BEA JRockit
  • MAC OSRuntime for OSX
  • Oracle JVM aka JServer – OJVM
  • Microsoft JVM

ColdFusion And The JVM

Since ColdFusion MX 6.0-6.1, CF has been a Java Application which needs what is known as a Servlet Container to run on. The most popular Servlet Container for ColdFusion is JRun. In the Enterprise edition of ColdFusion, the actual Servlet Container to be used can be selected during the install. In non-Enterprise installs JRun is automatically used. The main point being that there is always a Servlet Container.

Java J2EE – Java EE Servlet Containers

This is a list of suppliers of Java servlet containers which are certified as being J2EE compliant. Tomcat is not shown because I don’t believe it is fully certified as such by Sun. However, Apache Geronimo will be the first open-source certified J2EE/JavaEE servlet container.

  • Geronimo from Apache
  • JBoss from Red Hat
  • JRun from Adobe
  • Oracle OC4J
  • Sun Java System Application Server (Sun One).
  • WebLogic Server from BEA
  • WebSphere Server from IBM

Where everything sits-fits

This is a simple diagram showing where the Operating System, JVM, Servlet Container and Java Application Server sit in relativity.

ColdFusion and the JVM

When Allaire ported ColdFusion to run on Java (previously the core was C++) they realized there were already many copies of ColdFusion in use and needed to engineer some backward compatibility into the newly released version, CFMX 6.0 code-named “Neo”. If we refer to the diagram above we can see that ColdFusion (the yellow rectangle in case it is hard to read the text) runs inside the Java Servlet Container (often JRun).

By ColdFusion MX 6.1 (code named Red Sky) CF was a fully compliant J2EE application server which involves items such as a particular directory-file structures. However Allaire-Macromedia wanted to make sure that existing users of CF could upgrade easily and relatively painlessly. They also wanted to market ColdFusion to the Java community who were used to a particular directory structure. As added benefit for Adobe-Macromedia, this presented a marketing opportunity to have two versions of CFMX; a Standard-Server version and an Enterprise version.

The Standard-Server version would follow the existing CF install where there could only be one instance of ColdFusion on a physical server. The Enterprise version would follow the J2EE standard where multiple application server instances are common and normal.

One very important point to note here is that whether ColdFusion is installed as a Standard-Server version or as an Enterprise version, the JVM is still at the heart of all.

This need to support existing CF users led to the creation of a convoluted directory structure for the Standard-Server version, I will attempt to describe that below. This is a typical directory structure for a Windows install of CFMX 7 Standard-Server version. Note that I have highlighted the “runtime” directory. That directory actually contains JRun and this is visually confusing as in actual fact, ColdFusion is running inside JRun, not the other way around as it appears in this directory structure. In this install there can only be one ColdFusion server instance on a single physical server.

Once again this is a typical Windows install, this time for the Enterprise version and shows ColdFusion (which is inside the “servers” directory) as actually logically where it is inside the JRun4 directory structure. In this install, multiple versions of ColdFusion are possible on a single physical server.

Again, one very important point to note here is that whether ColdFusion is installed as a Standard-Server version or as an Enterprise version, the JVM is still at the heart of all.

Garbage Collection

The JVM has a fairly efficient memory management paradigm which is seen better that the typical Windows methodology. Sun JVM’s use a generational garbage collector with two young generations and an old generation. New objects are always allocated in the young generation part of the heap. Every time the young generation is full a minor garbage collection takes place. Minor garbage collections don’t take very long. Objects which survived some minor collections, because they are still referenced from the application, are promoted to the older generation.

Usually the older generation is full at some point too, and needs to be cleaned. This leads to a major garbage collection or Full Garbage (Full GC) collection that removes garbage from both the old and the younger generations. Full GC’s are “stop-the-world” events and need to occur only when absolutely necessary. I emphasize here the part in italics in the paragraph above “because they are still referenced from the application“.

In an efficiently written CF-Java web application objects created in memory should be cycled and collected (that memory allocation released) whilst the object is in the “new” part of the overall memory space. Web applications by nature have short object lifecycles because of the request-response nature of web applications. As a very general rule the more objects that end up in the “old” memory space the more often Full GC’s are likely to occur.

Garbage Collectors

There are three main Garbage Collector algorithms available to ColdFusion – JRun applications; here they are.

  • XX:+UseParallelGC Use parallel garbage collection, this is typically the default collector for ColdFusion installations. This collector is also referred to as the throughput collector. It uses a parallel version of the young generation collector. The old (tenured) generation is still cleaned with the default collector. Note This option can not be used in conjunction with -XX:+UseConcMarkSweepGC.
  • XX:+UseConcMarkSweepGC Use concurrent garbage collection. This collector is also referred to as the concurrent low pause collector. It collects garbage in the old (tenured) generation concurrently to executing the application. Note that this option can not be used in conjunction with -XX:+UseParallelGC. Instead you may combine it with -XX:+UseParNewGC
  • XX:+UseParNewGC Instructs the VM to use a parallel collector for the young generation. This option can be used in conjunction with -XX:+UseConcMarkSweepGC. As a note I have found that using the ConcMarkSweepGC along with ParNewGC can cause longer Full GC’s, I tend to use ParNewGC on its own. To change the garbage collector one of the above arguments is added to the jvm.config file.

Full Garbage Collections (Full GC’s) – Try To Minimize

One of the first methods I always want to employ when troubleshooting is to ascertain how many Full GC’s are taking place. To do so I add these arguments to the jvm.config file, which is located in one of these directories depending on the installation:

  • Enterprise – {drive-volume}jrun4bin
  • Standard/Server – {drive-volume}CFusionMX7runtimebin

I add these five arguments to the jvm.config file, it is important to remove these arguments when verbose garbage collection output is no longer needed.

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:yourname.log*

*This will send Garbage collection output to the {std}-out.log and will create a log of all GC activity with whatever name you gave yourname in either. This output log will go to one of the following locations.

  • Enterprise – {drive-volume}jrun4bin
  • Standard/Server – {drive-volume}CFusionMX7runtimebin

Full GC’s are “stop-the-world” events and need to occur only when absolutely necessary.

Verbose Garbage Collection Log Example

Here is an example of the Verbose Garbage Collection log output:

par new generation   total 97088K, used 0K [0x10010000, 0x171d0000, 0x171d0000)
eden space 77696K,   0% used [0x10010000, 0x10010000, 0x14bf0000)
from space 19392K,   0% used [0x14bf0000, 0x14bf0000, 0x15ee0000)
to   space 19392K,   0% used [0x15ee0000, 0x15ee0000, 0x171d0000)
&lt;span style=&quot;color: red;&quot;&gt;tenured generation   total 932096K, used 11109K&lt;/span&gt; [0x171d0000, 0x50010000, 0x50010000)
the space 932096K,   1% used [0x171d0000, 0x17ca95b8, 0x17ca9600, 0x50010000)
compacting perm gen  total 26624K, used 26606K [0x50010000, 0x51a10000, 0x58010000)
the space 26624K,  99% used [0x50010000, 0x51a0b8a0, 0x51a0ba00, 0x51a10000)
} , 0.1523967 secs]
&lt;span style=&quot;color: red;&quot;&gt;3691.704: [Full GC&lt;/span&gt;  {Heap before GC invocations=8:

This Full GC was almost certainly unnecessary because at the time it ran only 1109k out of 932096k of the tenured (old) generation had been used. I also note the timestamp (3691.704) of every Full GC in the log. If I see a regular pattern it almost always means something is calling a Full GC perhaps using System.gc()

Methods To Safely Minimize Full GC’s

One pattern I have seen over and over with ColdFusion is that Full GC’s are very often occurring at regular 60 second intervals. Although I have never been able to pin this down fully, I suspect it is due to a Java process called RMI(Remote Method Invocation). So then what do we do to control these unnecessary “stop-the-world” Full G’s? There are two alternatives:

  1. Disable Full GC’s using the -XX:+DisableExplicitGC jvm.config argument. I am not totally happy doing this and in many cases it will not work
  2. Run Full GC’s at longer regular intervals. Applying these arguments instead of -XX:+DisableExplicitGC will cause Full GC’s every 10 minutes (600,000 ms).

Using Metrics Logging In Troubleshooting The JVM

I have found that enabling metrics looking is invaluable in troubleshooting ColdFusion and JRun application issues. Metrics logging will show the status of the CF-JRun instance at pre-selected time intervals in terms of number of threads in use and available memory. In order to enable metrics logging we need to edit the jrun.xml file, here:

  • Enterprise install {drive-volume}JRun4servers{servername}SERVER-INF
  • Server-Standard install {drive-volume}CFusionMX7runtimeservers{servername}SERVER-INF

Here are the settings I usually use by making these changes in the jrun.xml file. Firstly uncomment the service class…

Then set “metricsEnabled” to true, we will leave other settings to default…

Lastly I like to split the logs out to make reading them easier by this change…


I have found that the only way to really determine the correct garbage collector to use for your applications and what the correct arguments are to pass to the JVM, via the jvm.config file, to effectively manage garbage collections is to first determine how many Full GC’s are occurring and control that if there are too many. Then load-test the application whilst tuning the arguments passed to the JVM, via the jvm.config file.

Tag Cloud