The amazing adventures of Doug Hughes

Archive for July, 2005

I Need Contractors!

I’m glad to say, business seem to be booming. In fact, I may have rather over booked myself. Thus, I am beginning to look for two to three contractors to help me out on two rather large projects for three months. Without getting into detail, I am ideally looking for two extremely talented ColdFusion programmers who can dedicate 40 or more hours a week. I will also consider one full time and two part time programmers. All work will be done remotely with no onsite presence at all.

These programmers will need to have experience working with Mach-II and/or Model-Glue. At least one of these applications will be based around Model-Glue and (maybe Service-Glue bonus points if you tell me what this is in your cover letter). The other application may be more ad-hoc in structure. Both applications will be primarily CFC based.

The contractors will be receiving instructions in the form of UML diagrams and requirements documents. The developers will be responsible for creating the needed classes and Unit Tests (using Paul Kenny’s CFCUnit).

The contractor must have a strong understanding, in particular, of data persistence design patterns. Currently the databases for these projects will be MSSQL.

I would like to find some of the best in the business for these projects! If you fit the bill, send me your resume at . When you send your resume, please include in your cover letter a few good reasons I should choose you over the rest of the competition and your required hourly rate.


At Last, MX 7!

Last night I did a little “Cowboy IT” of my own. I upgraded this server from ColdFusion MX 6.1 to 7. All in all, it went rather smoothly, except for two things.

First off, there is appears to be a rather important difference as to how ColdFusion 7 creates stubs for web services. Specifically, my fortune web service simply stopped working from ColdFusion after the upgrade. I could still use it from .NET, but CF kept giving me bizarre errors relating to errors in the Java stub. The strange part about this was that if I simply renamed the fortune.cfc file it would work. However, lots of people use that service, I can’t just change the name!

I ended up stripping out all the public methods and trying to rebuilt it method by method till I could find the part causing the problem. Oddly, when I had only one remote method I still got an error, but it was different from others. Unfortunately, I failed to make note of it. I think it was something like a duplicate file name error from Apache Axis. I spent some time pondering this before I stumbled across the answer to the problem!

My web service is named fortune.cfc. However, it’s <cfcomponent> tag looked like this:

<cfcomponent displayname="Fortune" hint="I am a web services version of the popular BSD/Linux amusement/game Fortune.">

On a whim, I decided to change displayname=”Fortune” to displayname=”fortune”. I stopped and restarted ColdFusion and suddenly (and much to my surprise) ColdFusion 7 was able to generate the stub for this web service!

My guess is that if you define the display name, and it’s the same as the CFC’s file name, it needs to be the same case.

I hope this helps someone else. What a pain!

The other issue, which I’m still resolving, relates to Verity but might be specific to how I tried to do the update. I was running on JRun. However, I wanted to be able to use the Enterprise manager in ColdFusion. From what I’ve seen, this only shows up in the default installation of ColdFusion on JRun. Subsequent instances of ColdFusion or instances created from EAR or WAR files do not provide the enterprise manager. So, I tried creating ColdFusion Archives (car files) for each of my server instances. I then uninstalled JRun completely and reinstalled fresh from the ColdFusion 7 installer.

This did give me the enterprise manager, as hoped. However, when I restored the car files to the new instances I had issues with one instance that had been using Verity. The problem is that ColdFusion MX 6.1 collections do not upgrade or are not forward compatible with ColdFusion MX 7 collections. However, MX 7 made a valiant effort to deploy the 6.1 collection anyhow. I received errors relating to deploying the collection. However, it made reference to the collection in the administrator.

Now the collection can’t seem to be easily deleted. I have to jump through hoops do to so. If I do manage to delete it and recreate it using the administration interface I can’t add documents to it. Very odd.

Luckily, it’s only one instance and one collection. I plan to export all the settings from the bad instance (not including verity collections) and create a whole new instance. I’ll then import the good settings from the old instance and point all the sites to that new (and hopefully correctly working) instance. If that works I should be able to recreate the collections and delete the bad instance all together.

It might be a pain, but I’ve got to say, JRun and J2EE are cool. + BlueDragon.Net = Good Business Sense, Not a Deal with the Devil!

I’ve been stewing recently on some nonsense I’ve heard people in the ColdFusion “community” spouting over the last week or so. Specifically, there’s been a lot of criticism of for using BlueDragon.Net.

A few people seem to be criticizing MySpace. The criticism seem to be following two routes:

  1. MySpace does not follow the critic’s (mis)conceptions of best coding and testing practices
  2. MySpace is stupid for using BlueDragon.Net and not ColdFusion MX.

From what I understood from last Friday’s CFUnited keynote speech, is a rapidly growing website. When I say rapidly, I mean in terms that no one I know in the ColdFusion “community” has ever dealt with. Consider that they were adding 10 to 20 machines every few weeks to keep up with their growth rate.

The problem MySpace ran into was that they were running into problems related to diminishing returns. A database server can really only handle a certain number of connections at a time. Also, connecting to a SQL server is an expensive operation. For this reason, web application servers such as ColdFusion will typically keep a certain number of connections open at a time. This is called pooling. Queries are directed to the connections in the pool which are not busy. In the case that all the connections are busy the request waits (or opens a new connection).

In MySpace’s case, they were adding ColdFusion servers so quickly that they reached the ultimate limits of their data layer in terms of concurrent connections. So, as they added more ColdFusion servers they actually had to lower the number of connections available in the pool for every single one of their ColdFusion servers. Eventually, this begins to detriment performance of the application.

This is the proverbial rock and a hard place. You’re damned if you add more servers and you’re damned if you don’t. Shit!

So, to stay in business, MySpace had to do something about this problem. I encourage you to remember that MySpace is a for profit business. They’re not going to just fold up because their favorite technology doesn’t meet their extremely demanding needs.

So, they began experimenting with .NET. They found that by deploying BlueDragon.Net and leveraging ADO.NET and CFML they could reduced the load on their servers by 50%. Presumably this also fixed their problems related to database connections.

Let’s pause at this point and think about what this means from a business perspective. Let’s say (conservatively) that spends $5000 for each server they buy. Now add to that the costs of delivery, installation, labor to maintain the servers, software licensing, electricity, racks, networking gear, datacenter space, cables, air conditioning, security, and more. Those servers are expensive!

I would guess in one-time costs alone, that MySpace spent an absolute bare-bones minimum of $250,000. I would bet it was well over $2,000,000.

Not to get off on a tangent, but I once set up a very small redundant array of web and SQL servers. There were about 8 to 10 web servers including live, staging, and dev. There were hardware load balancers. Their were two clustered SQL servers, a SAN and NAS, backup robots, and all the other domain servers required to simply provide a minimum level of redundant service. Not including software licensing costs, I spent more than half a million dollars of my employers money. And that was for less than 20 servers. Now, back to our normally scheduled ranting.

Throughout MySpace’s presentation, they made it clear that they were publishing updates to the site several times a day. Frequently these updates would break the site, or portions thereof.

I heard rumblings from a few people that MySpace wasn’t following best practices with regards to rolling out code. Admittedly, the speaker made it sound like people were using the live site as a form of testing. However, we, as the general public, don’t have any clue what their process for rolling code out really is. We don’t know if it was as hectic as the speaker made it sound, or if there is a staging and testing process they follow.

Keep in mind that only a very few and select number of people in the entire industry have any experience with a site that receives the amount of traffic that does. I’ll bet you’ve never worked on anything of this scale. I know I haven’t!

Consider this, how the heck do you effectively test code which will be under the load of hundreds of thousands of concurrent users? The answer is simple: Under the load of hundreds of thousands of concurrent users ! Or, to put it another way, if you think it works, publish the son of a bitch! If the servers start to smoke you’d better either roll back or fix the problem right away.

Think about this in another light: Let’s say there actually is a way a way to realistically simulate the load of hundreds of thousands of concurrent users in a staging environment. On what hardware do you place the staging site? If MySpace has 50 web servers (consider the cost we discussed earlier, please), are you really going to spend millions dollars just to test code before rolling it out?

Ok, let’s say you actually did have a way to effectively test the site under that level of load, how do you synchronize your developers? Presumably, MySpace has a reasonably large development team. Also presumably, each developer or a team of developers is working on different stuff concurrently. From the sound of it, code is completed and ready for testing several times a day. Given that, do you take all of the updates and push them out to the 50 to 100 staging servers (which takes time (which is money)) and test them all at the same time? If so, what if one group of developers is completed before another. Does that first group then have to sit idle (wasting time and money) while waiting for the other group to finish their code and deploy it for testing? Then what about the time it takes to test and interpret test results?

What I’m getting at is that, with the exception of crossing your fingers and pushing the code live, there’s no realistic way of testing a site of this size in a way that would be meaningful with the level of traffic it is receives. Furthermore, if there was, it wouldn’t be cost effective given the alternative of testing in the live environment.

So, yes, it might be CowBoy coding, but I’d bet my left leg they know a lot more about this than you or I do. So, cut them some slack and let them get on with their jobs. You shouldn’t be criticizing these guys, you should be amazed that they’re doing as good of a job as they are.

Now, onto the second complaint, which is much more amorphous:

They used BlueDragon.Net! (Gasp!)

The most intelligent argument I’ve seen against the use of BlueDragon.Net is that Microsoft is, well, Microsoft. A lot of people have a bone against Microsoft because they’re extremely competitive. Yes, they were even convicted of being a monopoly once, but that has no bearing on

Based on that argument, I’ll try extrapolate the true complaint. I believe that people are concerned ultimately about platform lock in. In other words, because MySpace elected to use BlueDragon.Net, they were forced to use Windows servers. (By the way, they were already using Windows. So this doesn’t really change the equation at all.)

Another concern may be that people think that Microsoft’s ultimate goal is not to support New Atlanta and BlueDragon.Net, but to convert ColdFusion programmers to .Net programmers and kill off ColdFusion as a platform.

This very well might be true. If it is, get over it. There are only two things which can come from that: Competition creates a better ColdFusion. Or ColdFuion dies and we are ultimately forced to find a new tool, .Net or otherwise.

Remember, a language is just a tool. You may select a tool because you prefer it, however businesses select tools because it does the best job to solve a specific problem and because it provides the company with the highest return on investment..

If Macromedia ultimately “losses” this battle and stops selling ColdFuison, you will not be forced to use .Net. You may choose to, but there are hundreds of other options out there. Business people are not emotionally attached to their tools. Business people are pragmatic about their tools. The tools do one thing, and one thing only: Solve a problem.

I’m going to ignore the emotional side of the argument as it has no bearing.

Let’s take a step back. First off, before MySpace decided to use BlueDragon.Net, they apparently evaluated lots of other options such as using Oracle for their data layer, moving to Linux, rewriting everything from the ground up, and more. In the end, they decided on BlueDragon.Net.

I can explain why in one word: Money .

I can also restate this in another word: Business .

Whether you like it or not, is hosted on Windows. (I’ll bet that this too was a business/money decision made when they first got started, but I’ll not get into that now.)

Either way, it didn’t matter.’s staff (who are already familiar with their application, the business model, and the market) are a Windows-based staff. To hire a set of Oracle DBAs and/or a set of Linux administrators would have a incidental costs in addition to the cost of the new employees salaries. New employees need to be to get them up to speed. If you consider that your pool of qualified candidates is in the probably less than 1000 because of the special requirements of this site, the employees salaries will not be inconsequential. Additionally, there is bound to be a lot of on the job training required to get up to speed. But it’s not just these new people. You’re going to need to retrain your existing staff or fire them and loose your investment in them along with all their unique expertise and knowledge.

The end of it is, moving to Linux and/or Oracle was too expensive and would have handicapped them during the transition. I’d guess the transition would have been a bare minimum of at least one year. Furthermore, there’s no guarantee that porting to those platforms would have actually provided any benefits in terms of performance. Porting to Linux and/or Oracle clearly would have had no bearing on their ability to test code outside of a live environment.

My ultimate point was that, when they did a cost vs. benefits analysis on the various options, the most effective option, with the highest return on investment was moving to BlueDragon.Net.

What are the tradeoffs? Well, they’re locked into Microsoft operating systems as long as they wish to continue using BlueDragon.Net. They’re somewhat locked into ADO.NET and Microsoft SQL server, so long a they choose to continue using those. I can’t think of any other trade offs. Please add some in the comments, if you can think of them.

What are the advantages? They don’t need to significantly shift or retrain their staff, saving significant amounts money. They don’t loose their investment spent to develop their current ColdFusion based code. They get better performance (this is apparently related to performance of ADO.NET vs. JDBC drivers). They have to put less money in new hardware and associated costs. They can leverage some of the power of ColdFusion and .NET as is appropriate for their company.

So, let me summarize.

All business are, ultimately, in the business of making money. It is MySpace’s job to make money, not to spend money questionably or irresponsibly. When all things were said-and-done, they didn’t favor BlueDragon.Net to provide free marketing to New Atlanta. They did it because it allowed them to receive the absolute most return on investment when compared to their other options.

If you think about it, the lesson from the keynote isn’t that you should use BlueDragon.Net, or Microsoft software, or that you shouldn’t use Macromedia’s products.

Here’s the lesson:

Think pragmatically and make the most cost effective decisions to help guarantee the success of your business.

Tag Cloud