I’ve been stewing recently on some nonsense I’ve heard people in the ColdFusion “community” spouting over the last week or so. Specifically, there’s been a lot of criticism of MySpace.com for using BlueDragon.Net.
A few people seem to be criticizing MySpace. The criticism seem to be following two routes:
- MySpace does not follow the critic’s (mis)conceptions of best coding and testing practices
- MySpace is stupid for using BlueDragon.Net and not ColdFusion MX.
From what I understood from last Friday’s CFUnited keynote speech, MySpace.com is a rapidly growing website. When I say rapidly, I mean in terms that no one I know in the ColdFusion “community” has ever dealt with. Consider that they were adding 10 to 20 machines every few weeks to keep up with their growth rate.
The problem MySpace ran into was that they were running into problems related to diminishing returns. A database server can really only handle a certain number of connections at a time. Also, connecting to a SQL server is an expensive operation. For this reason, web application servers such as ColdFusion will typically keep a certain number of connections open at a time. This is called pooling. Queries are directed to the connections in the pool which are not busy. In the case that all the connections are busy the request waits (or opens a new connection).
In MySpace’s case, they were adding ColdFusion servers so quickly that they reached the ultimate limits of their data layer in terms of concurrent connections. So, as they added more ColdFusion servers they actually had to lower the number of connections available in the pool for every single one of their ColdFusion servers. Eventually, this begins to detriment performance of the application.
This is the proverbial rock and a hard place. You’re damned if you add more servers and you’re damned if you don’t. Shit!
So, to stay in business, MySpace had to do something about this problem. I encourage you to remember that MySpace is a for profit business. They’re not going to just fold up because their favorite technology doesn’t meet their extremely demanding needs.
So, they began experimenting with .NET. They found that by deploying BlueDragon.Net and leveraging ADO.NET and CFML they could reduced the load on their servers by 50%. Presumably this also fixed their problems related to database connections.
Let’s pause at this point and think about what this means from a business perspective. Let’s say (conservatively) that MySpace.com spends $5000 for each server they buy. Now add to that the costs of delivery, installation, labor to maintain the servers, software licensing, electricity, racks, networking gear, datacenter space, cables, air conditioning, security, and more. Those servers are expensive!
I would guess in one-time costs alone, that MySpace spent an absolute bare-bones minimum of $250,000. I would bet it was well over $2,000,000.
Not to get off on a tangent, but I once set up a very small redundant array of web and SQL servers. There were about 8 to 10 web servers including live, staging, and dev. There were hardware load balancers. Their were two clustered SQL servers, a SAN and NAS, backup robots, and all the other domain servers required to simply provide a minimum level of redundant service. Not including software licensing costs, I spent more than half a million dollars of my employers money. And that was for less than 20 servers. Now, back to our normally scheduled ranting.
Throughout MySpace’s presentation, they made it clear that they were publishing updates to the site several times a day. Frequently these updates would break the site, or portions thereof.
I heard rumblings from a few people that MySpace wasn’t following best practices with regards to rolling out code. Admittedly, the speaker made it sound like people were using the live site as a form of testing. However, we, as the general public, don’t have any clue what their process for rolling code out really is. We don’t know if it was as hectic as the speaker made it sound, or if there is a staging and testing process they follow.
Keep in mind that only a very few and select number of people in the entire industry have any experience with a site that receives the amount of traffic that MySpace.com does. I’ll bet you’ve never worked on anything of this scale. I know I haven’t!
Consider this, how the heck do you effectively test code which will be under the load of hundreds of thousands of concurrent users? The answer is simple: Under the load of hundreds of thousands of concurrent users ! Or, to put it another way, if you think it works, publish the son of a bitch! If the servers start to smoke you’d better either roll back or fix the problem right away.
Think about this in another light: Let’s say there actually is a way a way to realistically simulate the load of hundreds of thousands of concurrent users in a staging environment. On what hardware do you place the staging site? If MySpace has 50 web servers (consider the cost we discussed earlier, please), are you really going to spend millions dollars just to test code before rolling it out?
Ok, let’s say you actually did have a way to effectively test the site under that level of load, how do you synchronize your developers? Presumably, MySpace has a reasonably large development team. Also presumably, each developer or a team of developers is working on different stuff concurrently. From the sound of it, code is completed and ready for testing several times a day. Given that, do you take all of the updates and push them out to the 50 to 100 staging servers (which takes time (which is money)) and test them all at the same time? If so, what if one group of developers is completed before another. Does that first group then have to sit idle (wasting time and money) while waiting for the other group to finish their code and deploy it for testing? Then what about the time it takes to test and interpret test results?
What I’m getting at is that, with the exception of crossing your fingers and pushing the code live, there’s no realistic way of testing a site of this size in a way that would be meaningful with the level of traffic it is receives. Furthermore, if there was, it wouldn’t be cost effective given the alternative of testing in the live environment.
So, yes, it might be CowBoy coding, but I’d bet my left leg they know a lot more about this than you or I do. So, cut them some slack and let them get on with their jobs. You shouldn’t be criticizing these guys, you should be amazed that they’re doing as good of a job as they are.
Now, onto the second complaint, which is much more amorphous:
They used BlueDragon.Net! (Gasp!)
The most intelligent argument I’ve seen against the use of BlueDragon.Net is that Microsoft is, well, Microsoft. A lot of people have a bone against Microsoft because they’re extremely competitive. Yes, they were even convicted of being a monopoly once, but that has no bearing on MySpace.com.
Based on that argument, I’ll try extrapolate the true complaint. I believe that people are concerned ultimately about platform lock in. In other words, because MySpace elected to use BlueDragon.Net, they were forced to use Windows servers. (By the way, they were already using Windows. So this doesn’t really change the equation at all.)
Another concern may be that people think that Microsoft’s ultimate goal is not to support New Atlanta and BlueDragon.Net, but to convert ColdFusion programmers to .Net programmers and kill off ColdFusion as a platform.
This very well might be true. If it is, get over it. There are only two things which can come from that: Competition creates a better ColdFusion. Or ColdFuion dies and we are ultimately forced to find a new tool, .Net or otherwise.
Remember, a language is just a tool. You may select a tool because you prefer it, however businesses select tools because it does the best job to solve a specific problem and because it provides the company with the highest return on investment..
If Macromedia ultimately “losses” this battle and stops selling ColdFuison, you will not be forced to use .Net. You may choose to, but there are hundreds of other options out there. Business people are not emotionally attached to their tools. Business people are pragmatic about their tools. The tools do one thing, and one thing only: Solve a problem.
I’m going to ignore the emotional side of the argument as it has no bearing.
Let’s take a step back. First off, before MySpace decided to use BlueDragon.Net, they apparently evaluated lots of other options such as using Oracle for their data layer, moving to Linux, rewriting everything from the ground up, and more. In the end, they decided on BlueDragon.Net.
I can explain why in one word: Money .
I can also restate this in another word: Business .
Whether you like it or not, MySpace.com is hosted on Windows. (I’ll bet that this too was a business/money decision made when they first got started, but I’ll not get into that now.)
Either way, it didn’t matter. MySpace.com’s staff (who are already familiar with their application, the business model, and the market) are a Windows-based staff. To hire a set of Oracle DBAs and/or a set of Linux administrators would have a incidental costs in addition to the cost of the new employees salaries. New employees need to be to get them up to speed. If you consider that your pool of qualified candidates is in the probably less than 1000 because of the special requirements of this site, the employees salaries will not be inconsequential. Additionally, there is bound to be a lot of on the job training required to get up to speed. But it’s not just these new people. You’re going to need to retrain your existing staff or fire them and loose your investment in them along with all their unique expertise and knowledge.
The end of it is, moving to Linux and/or Oracle was too expensive and would have handicapped them during the transition. I’d guess the transition would have been a bare minimum of at least one year. Furthermore, there’s no guarantee that porting to those platforms would have actually provided any benefits in terms of performance. Porting to Linux and/or Oracle clearly would have had no bearing on their ability to test code outside of a live environment.
My ultimate point was that, when they did a cost vs. benefits analysis on the various options, the most effective option, with the highest return on investment was moving to BlueDragon.Net.
What are the tradeoffs? Well, they’re locked into Microsoft operating systems as long as they wish to continue using BlueDragon.Net. They’re somewhat locked into ADO.NET and Microsoft SQL server, so long a they choose to continue using those. I can’t think of any other trade offs. Please add some in the comments, if you can think of them.
What are the advantages? They don’t need to significantly shift or retrain their staff, saving significant amounts money. They don’t loose their investment spent to develop their current ColdFusion based code. They get better performance (this is apparently related to performance of ADO.NET vs. JDBC drivers). They have to put less money in new hardware and associated costs. They can leverage some of the power of ColdFusion and .NET as is appropriate for their company.
So, let me summarize.
All business are, ultimately, in the business of making money. It is MySpace’s job to make money, not to spend money questionably or irresponsibly. When all things were said-and-done, they didn’t favor BlueDragon.Net to provide free marketing to New Atlanta. They did it because it allowed them to receive the absolute most return on investment when compared to their other options.
If you think about it, the lesson from the keynote isn’t that you should use BlueDragon.Net, or Microsoft software, or that you shouldn’t use Macromedia’s products.
Here’s the lesson:
Think pragmatically and make the most cost effective decisions to help guarantee the success of your business.
DougHughes.net 