I’ve been stewing recently on some nonsense I’ve heard people in the ColdFusion “community” spouting over the last week or so. Specifically, there’s been a lot of criticism of MySpace.com for using BlueDragon.Net.
A few people seem to be criticizing MySpace. The criticism seem to be following two routes:
- MySpace does not follow the critic’s (mis)conceptions of best coding and testing practices
- MySpace is stupid for using BlueDragon.Net and not ColdFusion MX.
From what I understood from last Friday’s CFUnited keynote speech, MySpace.com is a rapidly growing website. When I say rapidly, I mean in terms that no one I know in the ColdFusion “community” has ever dealt with. Consider that they were adding 10 to 20 machines every few weeks to keep up with their growth rate.
The problem MySpace ran into was that they were running into problems related to diminishing returns. A database server can really only handle a certain number of connections at a time. Also, connecting to a SQL server is an expensive operation. For this reason, web application servers such as ColdFusion will typically keep a certain number of connections open at a time. This is called pooling. Queries are directed to the connections in the pool which are not busy. In the case that all the connections are busy the request waits (or opens a new connection).
In MySpace’s case, they were adding ColdFusion servers so quickly that they reached the ultimate limits of their data layer in terms of concurrent connections. So, as they added more ColdFusion servers they actually had to lower the number of connections available in the pool for every single one of their ColdFusion servers. Eventually, this begins to detriment performance of the application.
This is the proverbial rock and a hard place. You’re damned if you add more servers and you’re damned if you don’t. Shit!
So, to stay in business, MySpace had to do something about this problem. I encourage you to remember that MySpace is a for profit business. They’re not going to just fold up because their favorite technology doesn’t meet their extremely demanding needs.
So, they began experimenting with .NET. They found that by deploying BlueDragon.Net and leveraging ADO.NET and CFML they could reduced the load on their servers by 50%. Presumably this also fixed their problems related to database connections.
Let’s pause at this point and think about what this means from a business perspective. Let’s say (conservatively) that MySpace.com spends $5000 for each server they buy. Now add to that the costs of delivery, installation, labor to maintain the servers, software licensing, electricity, racks, networking gear, datacenter space, cables, air conditioning, security, and more. Those servers are expensive!
I would guess in one-time costs alone, that MySpace spent an absolute bare-bones minimum of $250,000. I would bet it was well over $2,000,000.
Not to get off on a tangent, but I once set up a very small redundant array of web and SQL servers. There were about 8 to 10 web servers including live, staging, and dev. There were hardware load balancers. Their were two clustered SQL servers, a SAN and NAS, backup robots, and all the other domain servers required to simply provide a minimum level of redundant service. Not including software licensing costs, I spent more than half a million dollars of my employers money. And that was for less than 20 servers. Now, back to our normally scheduled ranting.
Throughout MySpace’s presentation, they made it clear that they were publishing updates to the site several times a day. Frequently these updates would break the site, or portions thereof.
I heard rumblings from a few people that MySpace wasn’t following best practices with regards to rolling out code. Admittedly, the speaker made it sound like people were using the live site as a form of testing. However, we, as the general public, don’t have any clue what their process for rolling code out really is. We don’t know if it was as hectic as the speaker made it sound, or if there is a staging and testing process they follow.
Keep in mind that only a very few and select number of people in the entire industry have any experience with a site that receives the amount of traffic that MySpace.com does. I’ll bet you’ve never worked on anything of this scale. I know I haven’t!
Consider this, how the heck do you effectively test code which will be under the load of hundreds of thousands of concurrent users? The answer is simple: Under the load of hundreds of thousands of concurrent users ! Or, to put it another way, if you think it works, publish the son of a bitch! If the servers start to smoke you’d better either roll back or fix the problem right away.
Think about this in another light: Let’s say there actually is a way a way to realistically simulate the load of hundreds of thousands of concurrent users in a staging environment. On what hardware do you place the staging site? If MySpace has 50 web servers (consider the cost we discussed earlier, please), are you really going to spend millions dollars just to test code before rolling it out?
Ok, let’s say you actually did have a way to effectively test the site under that level of load, how do you synchronize your developers? Presumably, MySpace has a reasonably large development team. Also presumably, each developer or a team of developers is working on different stuff concurrently. From the sound of it, code is completed and ready for testing several times a day. Given that, do you take all of the updates and push them out to the 50 to 100 staging servers (which takes time (which is money)) and test them all at the same time? If so, what if one group of developers is completed before another. Does that first group then have to sit idle (wasting time and money) while waiting for the other group to finish their code and deploy it for testing? Then what about the time it takes to test and interpret test results?
What I’m getting at is that, with the exception of crossing your fingers and pushing the code live, there’s no realistic way of testing a site of this size in a way that would be meaningful with the level of traffic it is receives. Furthermore, if there was, it wouldn’t be cost effective given the alternative of testing in the live environment.
So, yes, it might be CowBoy coding, but I’d bet my left leg they know a lot more about this than you or I do. So, cut them some slack and let them get on with their jobs. You shouldn’t be criticizing these guys, you should be amazed that they’re doing as good of a job as they are.
Now, onto the second complaint, which is much more amorphous:
They used BlueDragon.Net! (Gasp!)
The most intelligent argument I’ve seen against the use of BlueDragon.Net is that Microsoft is, well, Microsoft. A lot of people have a bone against Microsoft because they’re extremely competitive. Yes, they were even convicted of being a monopoly once, but that has no bearing on MySpace.com.
Based on that argument, I’ll try extrapolate the true complaint. I believe that people are concerned ultimately about platform lock in. In other words, because MySpace elected to use BlueDragon.Net, they were forced to use Windows servers. (By the way, they were already using Windows. So this doesn’t really change the equation at all.)
Another concern may be that people think that Microsoft’s ultimate goal is not to support New Atlanta and BlueDragon.Net, but to convert ColdFusion programmers to .Net programmers and kill off ColdFusion as a platform.
This very well might be true. If it is, get over it. There are only two things which can come from that: Competition creates a better ColdFusion. Or ColdFuion dies and we are ultimately forced to find a new tool, .Net or otherwise.
Remember, a language is just a tool. You may select a tool because you prefer it, however businesses select tools because it does the best job to solve a specific problem and because it provides the company with the highest return on investment..
If Macromedia ultimately “losses” this battle and stops selling ColdFuison, you will not be forced to use .Net. You may choose to, but there are hundreds of other options out there. Business people are not emotionally attached to their tools. Business people are pragmatic about their tools. The tools do one thing, and one thing only: Solve a problem.
I’m going to ignore the emotional side of the argument as it has no bearing.
Let’s take a step back. First off, before MySpace decided to use BlueDragon.Net, they apparently evaluated lots of other options such as using Oracle for their data layer, moving to Linux, rewriting everything from the ground up, and more. In the end, they decided on BlueDragon.Net.
I can explain why in one word: Money .
I can also restate this in another word: Business .
Whether you like it or not, MySpace.com is hosted on Windows. (I’ll bet that this too was a business/money decision made when they first got started, but I’ll not get into that now.)
Either way, it didn’t matter. MySpace.com’s staff (who are already familiar with their application, the business model, and the market) are a Windows-based staff. To hire a set of Oracle DBAs and/or a set of Linux administrators would have a incidental costs in addition to the cost of the new employees salaries. New employees need to be to get them up to speed. If you consider that your pool of qualified candidates is in the probably less than 1000 because of the special requirements of this site, the employees salaries will not be inconsequential. Additionally, there is bound to be a lot of on the job training required to get up to speed. But it’s not just these new people. You’re going to need to retrain your existing staff or fire them and loose your investment in them along with all their unique expertise and knowledge.
The end of it is, moving to Linux and/or Oracle was too expensive and would have handicapped them during the transition. I’d guess the transition would have been a bare minimum of at least one year. Furthermore, there’s no guarantee that porting to those platforms would have actually provided any benefits in terms of performance. Porting to Linux and/or Oracle clearly would have had no bearing on their ability to test code outside of a live environment.
My ultimate point was that, when they did a cost vs. benefits analysis on the various options, the most effective option, with the highest return on investment was moving to BlueDragon.Net.
What are the tradeoffs? Well, they’re locked into Microsoft operating systems as long as they wish to continue using BlueDragon.Net. They’re somewhat locked into ADO.NET and Microsoft SQL server, so long a they choose to continue using those. I can’t think of any other trade offs. Please add some in the comments, if you can think of them.
What are the advantages? They don’t need to significantly shift or retrain their staff, saving significant amounts money. They don’t loose their investment spent to develop their current ColdFusion based code. They get better performance (this is apparently related to performance of ADO.NET vs. JDBC drivers). They have to put less money in new hardware and associated costs. They can leverage some of the power of ColdFusion and .NET as is appropriate for their company.
So, let me summarize.
All business are, ultimately, in the business of making money. It is MySpace’s job to make money, not to spend money questionably or irresponsibly. When all things were said-and-done, they didn’t favor BlueDragon.Net to provide free marketing to New Atlanta. They did it because it allowed them to receive the absolute most return on investment when compared to their other options.
If you think about it, the lesson from the keynote isn’t that you should use BlueDragon.Net, or Microsoft software, or that you shouldn’t use Macromedia’s products.
Here’s the lesson:
Think pragmatically and make the most cost effective decisions to help guarantee the success of your business.
DougHughes.net 
Comments on: "MySpace.com + BlueDragon.Net = Good Business Sense, Not a Deal with the Devil!" (34)
One pint I offered during one of these debates was that this could open up CFML to .NET developers. Is it likely to bring a flock of converts…probably not. But it could happen.
What could the results of this be? Well BD has typically been at least a half (more like full) release behind ColdFusion. Imagine being given a taste of the forbidden fruit, only to find out you can’t have it all…at least until the next release…but by that time the forbidden fruit is better than what you have now…so you wait fo rthe next release.
Now…you get a .NET developer dabbling in CFML using BD. They start liking it…they investigate further..discover CFMX7 and all the wonders it can offer…now we have a convert.
Please forgive me if this is completely unrealistic..I try to look on the positive side of things.
LikeLike
nice to hear someone talk about pragmatism with regards to the whole picture rather than dogma.
LikeLike
Great stuff Doug! Always helps to look at this from the perspective of "the suits." I also like Scott’s notion that maybe, just maybe, some MS folks will at least take a look at CF now. (We can dream …)
The thing I keep going back to is when everything was on the table for myspace.com they *could* have dumped CFML altogether, and if BD .NET wasn’t available they might well have done so. If that had been the case we’d be looking at a huge black eye right now instead of a great success story for CFML as a technology (regardless of the specific implementation).
LikeLike
Matt, I doubt they would have thrown out the investment in CFML. They would have found some way to get this to work. BD.NET must have been the path of least resistance.
Think about this. What if they had 10 developers paid a (lowly) 60,000. Let’s also (wrongly) assume that’s the only cost (ignorning space costs, insurance costs, and other overhead). Those developers worked two years to build what they had now. Throwing that out is like throwing out a minimum of $600,000.
That’s not likely to happen.
LikeLike
True Doug, not likely but if they saw no options they would have had to go in another direction regardless of the cost involved. I’ve seen dumber decisions get made with fewer actual reasons behind them. Millions of dollars a year get spent on technology conversions because some manager who knows nothing about technology lays down a "We WILL Use Technology X" mandate. Happens all the time.
Yes, the cost of conversion would have come up in the discussions, but if they saw sticking with CFML as throwing good money after bad they would have gone in a different direction. Granted their decision would have been difficult but this certainly isn’t unheard of. They might have just seen it as a necessary expense, taken it on the chin, planned out their migration, and started the gradual move to another technology. My point is that I’m glad they had this option available.
LikeLike
Doug & Matt,
Totally agree with you two! This is a perfect example of a company being able to say "see, CFML (not Macromedia ColdFusion) was the right choice, we have options when writing in this language!" boosting confidence in other companies in that developing in CFML is a good investment, as opposed to them them saying "CFML was the wrong language so we went to .NET" and scare companies into using this technology. I see two camps here, the "Macromedia Developers" versus the "Software Developers" and that’s a personal decision, as for me I prefer to think of myself as a "Software Developer". To tie my career and future into one company’s technology is not only stupid – pardon my bluntness – but it also causes the kind of mass hysteria when mergers occurs. It’s also not a good personal investment since you are limiting yourself of what opportunities are out there! (like Doug said, passion aside "we are in business to make money!"). There’s a reason why CFML is not a proprietary language, imagine if it were? Isn’t that why some people grip about .NET? Like Scot Stroz said earlier in his comments (and Scott, I too hope that this is not too optimistic.. ) but I think that New Atlanta has built a very valuable bridge, not a one way for developers of CFML to go .NET but one that can have the opposite flow as well and help enrich our community with new people that were previously sitting on the opposite bank!!!
Cheers…
LikeLike
Doug & Matt,
Totally agree with you two! This is a perfect example of a company being able to say "see, CFML (not Macromedia ColdFusion) was the right choice, we have options when writing in this language!" boosting confidence in other companies in that developing in CFML is a good investment, as opposed to them them saying "CFML was the wrong language so we went to .NET" and scare companies into using this technology. I see two camps here, the "Macromedia Developers" versus the "Software Developers" and that’s a personal decision, as for me I prefer to think of myself as a "Software Developer". To tie my career and future into one company’s technology is not only stupid – pardon my bluntness – but it also causes the kind of mass hysteria when mergers occurs. It’s also not a good personal investment since you are limiting yourself of what opportunities are out there! (like Doug said, passion aside "we are in business to make money!"). There’s a reason why CFML is not a proprietary language, imagine if it were? Isn’t that why some people grip about .NET? Like Scot Stroz said earlier in his comments (and Scott, I too hope that this is not too optimistic.. ) but I think that New Atlanta has built a very valuable bridge, not a one way for developers of CFML to go .NET but one that can have the opposite flow as well and help enrich our community with new people that were previously sitting on the opposite bank!!!
Cheers…
LikeLike
Back in March I was working on what I thought was the biggest CFMX-FuseBox public site I had seen; 18 servers in a Cluster behind BigIP Hardware Devices in front of Oracle and DB2 RDBM’s. Then someone pointed out MySpace and the obvious size of the site blew my mind. It was bloody fantastic to have what at that time was the web’s seventh largest site running on CF and FuseBox. It has only got more fantastic since, in my opinion, MySpace have effectivlel blown away the belief that CF does not scale irrespective of the JDBC limitations.
Doug you put out a great post here and I do think there are possibilities in .NET warming to CF after all it is so bloody enjoyable. Thanks for this post.
LikeLike
So, do you think eBay, Yahoo, MSN etc just throw code up on the live site and see whether it breaks? Or do you think that they do have a way to perform realistic load tests and QA?
Do you think that large sites with huge investments in a given technology do everything possible to retain that technology? Or do you think that they sometimes just throw everything out and rewrite it all from scratch?
I personally know several companies that have invested many millions of dollars in a given technology and then decided it doesn’t meet their needs and just thrown it out, only to start from scratch with something new.
Let’s all just be happy that the 5th most trafficked site on the Internet is a ColdFusion site and leave it at that.
LikeLike
Sean. I suppose you’re right. At the same time, half of Google’s code is "Beta". Isn’t this the same as saying it’s not completly tested and that the user’s mileage may vary? To be fair, I’ve never ever worked for a company the scale of Google, MSN, AOL, MySpace, etc. Who am I to talk?
On top of that, I think both you and Matt make good points about a company throwing out investments. Once again though, I think it’s a business decision. If the costs involved in keeping your investment outweigh the cost of building something new, then the obvious answer is to build something new.
One other thing I havn’t seen discussed is how much effort they made to move to MX 6.1 or 7. Apparently they tried, but (because this is New Atlanta marking) they didn’t really discuss that.
LikeLike
"Let’s all just be happy that the 5th most trafficked site on the Internet is a ColdFusion site and leave it at that."
I think it’s more accurate to say that "the 5th most traficked site on the Internet is a CFML site", since it’s already partially running on BlueDragon.NET, and won’t be running ColdFusion at all after the conversion is complete.
LikeLike
I’m interested more in how .NET was able to make their servers more efficient.
"So, they began experimenting with .NET. They found that by deploying BlueDragon.Net and leveraging ADO.NET and CFML they could reduced the load on their servers by 50%. Presumably this also fixed their problems related to database connections."
Does anyone have any insight on how/why this is true?
LikeLike
Chris, I addressed this in my presentation at CFUNITED-05. I just put up a blog entry on this (the first in a series), which also contains a link to the original PowerPoint presentation:
http://blog.newatlanta.com/index.cfm?mode=entry&entry=C7ED7F00-8772-14C6-68A1F97022CAA85B
LikeLike
Being pragmatic about technology choices is always a wise choice.
I just visited myspace.com for the first time ever just now. It took quite a while to load up.
If you keep clicking on the two Cool New People, you’ll eventually get this error:
http://www.myspace.com/error.cfm?aspxerrorpath=/Pages/Profile.aspx
"Sorry! an unexpected error has occurred.
This error has been forwarded to MySpace’s technical group."
/shrug
LikeLike
Vince, I would have thought it was pretty obvious I meant "ColdFusion-the-language" rather than any particular product… We don’t talk about "CFML developers", we talk about "ColdFusion developers". We don’t talk about "CFML Components", we talk about "ColdFusion Components".
LikeLike
Please be kind to the newbie! And politely ignore if this is annoying.
I found this post via a google search for myspace technology. I’m trying to get enough of a sense of how such technologies work so that I can communicate effectively with techies on a project I envision that would have similar functionality as myspace.com.
If someone wouldn’t mind pointing me towards resources for basic understanding of things like Cold Fusion and .net, I would greatly appreciate it.
I do work with basic html and xhtml and use a lot of online services. So I know a little bit.
Thanks for your help:
clyde(at)prohiphop(dot)com
LikeLike
Looks like I missed the discussion. In case someone checks back, I’ve been browsing myspace a bunch over the last two days. I can’t go more than 10 or 15 minutes without it having some kind of problem but different each time. Anyway, I’ll save the details.
later
LikeLike
My 2 cents…
The possibility of more developpers using CFML because of BD.NET certainly doesn’t hurt (in my opinion) CF as a dev language… but offers a wider audience of developpers to ‘taste’ our beloved dev language.
That’s the conclusion that I’ve come to.
– Yves –
LikeLike
I know i’m a little late to the discussion – but I had actually never been to myspace.com until I read Doug’s post.
It doesn’t seem to be working correctly right now. The home page is generally blank. It has boxes drawn to hold content – but there is no content.
Is this the kind of problem they have been having with CF MX servers and the JDBC drivers – or are they already converted to BD? If so it doesn’t see to have resolved whatever problems they are having.. Unless the homepage is supposed to be empty.
LikeLike
myspace.com runs on BlueDragon so if there are any problems with the site now – and judging from some of the comments in this thread, there certainly are problems! – then those are BlueDragon issues…
LikeLike
That’s not true, Sean. Most of the site is still running CF5–it’s going to take several weeks (months, maybe) to convert the entire site. The problems you’re seeing are due to CF5 and are the reason the site it being upgraded to BD.NET. You’ll know the upgrade to BD.NET is complete when these problems go away.
LikeLike
Vince –
I think you can understand the consufion. I got the impression, from the keynote, that the conversion to BD .NET had already been completed. I can’t imagine I was the only one who walked away with that thought.
LikeLike
Yeah Scott, that’s definitely the impression they were giving! Such big news – the biggest CF site in the world runs BD.NET! Except it doesn’t.
LikeLike
Except that MySpace is running BlueDragon.NET. At the time of the CFUNITED announcement, the upgrade from CF5 to BD.NET had begun.
There are over 400 servers in their web server cluster. The process of taking each server out of production, installing BD.NET, uninstalling CF5, verifying the installation, and putting it back into production takes time. Especially when you consider they’re continuing to do new feature development at the same time (that is, they’re not stopping everything while they do the migration). If they did just one server a day, it would take over a year!
The upgrade to BD.NET is being done as a carefully planned, controlled, and managed process, and is proceeding smoothly.
LikeLike
Vince –
You can’t have your cake and eat it too.
New Atlanta cannot claim that BD .NET is running the largest CF site in the world, and then claim that some of the issues seen when visiting that same site are because the site is running CF 5.
In an earlier comment you suggested the process of migrating was ‘going to take several weeks (months, maybe)’. How can you migrate over 400 servers using the process you just described in several weeks, or even monhts?
LikeLike
Boy, you just have to love Marketing spin.
Hey, our stuff is great, running one of the biggest sites in the world…. well, except when the site breaks, then they aren’t using our stuff.
As John Stossel on 20/20 would say
"gimme a break."
LikeLike
Hi Scott,
The MySpace server cluster is segregated into "application pools." There are just over 30 separate applications that make up the MySpace web site. If you go to the MySpace web site, you’ll notice URLs for "mail.myspace.com", "forums.myspace.com", "music.myspace.com", etc.; these define the separate applications. For example, the "mail" application pool consists of 15 servers that only serve the mail application ("mail.myspace.com").
The upgrade from CF5 to BD.NET is being done one application at a time. Since I know which applications have been upgraded to BD.NET and which are still running on CF5, I can tell whether a problem is with a BD.NET server pool or CF5 server pool based on which application is exhibiting the problem (for example, in an earlier comment, "Bill" said he was having problems with the home page, which is still running on a CF5 server pool).
LikeLike
Vince –
That makes perfect sense. It may have helped if that explanation was given earlier.
Since I doubt you will be sharing the migration information, I guess we just need to take your word for it;-)
LikeLike
Vince, given this error (which isn’t a freak occurrence):
http://www.myspace.com/error.cfm?aspxerrorpath=/Pages/Profile.aspx "Sorry! an unexpected error has occurred.
I don’t see how you can say that all of the outstanding errors are CF related instead of BD, so you’re points are falling as nothing but marketing spin upon these ears…
LikeLike
Excellent, well-written article!
LikeLike
Well written! And quite correct.
LikeLike
=^ ^=
LikeLike
It’s great to see MySpace on NewAtlanta
products. Presumably an enterprise like
MySpace has enough oomph to be able to
test quality into New Atlanta products.
Our own experience was that they wanted to
charge us for licenses to debug their buggy
software, while violently resisting the
notion that maybe they should start
compensating us after ten or twenty hours
trying to recreate and debug egregious
problems that made the CF 5.0 debacle look
like a walk in the park.
Now, New Atlanta finally has a reasonable
test bed, and it’s called MySpace.com
LikeLike
JyN0oN brtsgstvuswz, [url=http://kcymvgdderkc.com/]kcymvgdderkc[/url], [link=http://nwqnztsaphiw.com/]nwqnztsaphiw[/link], http://vjayhlwwcixg.com/
LikeLike