Where Oh Where Did My Bottleneck Come From?
At one point in my career I thought I knew what performance tuning was… I would have described it as “writing tight code.” (Remember that phrase? “Tight code”?) And if I were to be tasked with fixing performance issues, I would have jumped on my computer, gotten a local copy of the app going, and started hitting pages to find the slow ones, then look at the code for ways to fix the issue. That, however, was before I knew what performance engineering really is.
The problem with that is the fact that no application runs in a vacuum… that’s the whole point of the web.Between your users and your application are myriad other systems: browsers and the hardware that run them, copper, fibre, switches and routers oh my! Even between the local gateway and your application, there is probably a web server, maybe a load balancer, a switch or two and probably some sort of box to hook your 1GB Ethernet backbone up to the real interwebs. Essentially, your application server is an island in the midst of a massive sea of technology, any part of which can involve itself in the performance and reliability of your application. If that weren’t the case, my old methods would work fine… but it ain’t that way.
Meanwhile, one of the burning questions we have to answer is “what is performance?” and its evil twin “what is poor performance?” What do we use to benchmark whether or not an application performs acceptably? That, however, is a pretty easy question to answer: User perception of response times. At least initially. There are other more technical issues to be resolved down the line, but the first sign of trouble is when a user calls the support desk and complains that the login is taking 30 seconds to process.
So, where to start? As I said before, originally I would have started with the code, run the processes being complained about, “tightened up the code”, and pushed it to prod without a great deal more thought. Only later would I start to consider the DB connections, the NIC, switches, etc., but if one of those items is the problem then I just wasted 3 days rewriting a block of code for, what? Nothing. That code performed well enough as it was… and I never solved the problem.
I have heard it said that, by the time you’re ready to fix the problem, you have 99% confidence that you know exactly what it is and it’s true. 1% is a reasonable margin of error… I mean, there’s always going to be times like when you installed the new lightbulb in the basement and flipped the switch but nothing happened. Who thinks to check the fuse panel when a lightbulb goes out? The thing is, that’s a cheap mistake. Spending days implementing a performance fix that misses the mark is not.
So we’re back to process, process, process. And it all starts with user perception. So we hook up JMeter to run a test plan that models real-world usage patterns (and we base those on web server logs, right?) and we do see a performance issue… what does it tell us? Heh, well that depends, doesn’t it?
Yes, I’m about to talk about jmeter again… because it gives you great info: min/max/mean processing times, error rates, etc., all aggregated across as many requests as your test plan made. So if you run a jmeter test plan against the application from the same machine that’s hosting the it, you should be seeing the application in its peak performance capability… your baseline benchmark.
It’s often enough to confirm that the application itself is or is not the culprit, but if you don’t see the problem at the server, just move to the next network connection closer to the internet until you’re hooked up to the WIFI at Starbucks. Have some cups of coffee, eat a scone or three or… well, bummer. You found a dying NIC and never made it to Starbucks. At some point you should definitely see a dramatic, unreasonable slowdown and, at the least, narrow the field of potential causes. Personally I think the most critical part, however, is to record findings so that you have real, hard numbers to work with. I’ve actually taken to using a spreadsheet to track the numbers with the added benefit that I can use charts and graphs to demonstrate the issue to stakeholders who often can’t tell a “long-running query” from a crescent wrench. Plus I can run many different sets of benchmarks, recording their results over time, and compare them all based on anything from time of day to season of the year.
Ultimately, though, this is about “reasonable performance”. It takes work to build a performant application delivery system… there’s no sense in building it to serve 10,000,000 hits a day with <20ms response times when it only needs to serve 10,000 requests and response times are allowed to be up to 100ms. Until you can’t do that, you don’t have a problem…. what we’re trying to prevent is a user going to your domain and suffering through long and painful response times and deciding that your website sucks. We need to concern ourselves with the perceptionthat your application is horrendous and should be replaced with a teletype in a small room full of well-trained monkeys, however unfair that may be. Which is why, if you know what you’re doing, a tool like jmeter is really the place to start.
The point is that jmeter has the capacity to place a realistic load on your application, simulating as many users as you need so you can see how your application performs not just under load but from any of several different locations. You can run a tool like jmeter from a remote machine (you can even take your laptop home and use it from a consumer internet connection), from the local machine (to get a baseline of your applications actual performance for comparison with other locations) and from pretty well anywhere in between.
Because the best application in the world running on a broken network is nothing more than a broken application (from the user’s perspective!)… know what I mean?
I set the server to listen on 8085 (any open port should work fine), and I chose the type of ‘HTTP Request HTTPClient’ 
Now you are ready to perform your actions that will represent a user in Scenario A. Browse the site of your choice (I will be picking on Adobe.com for this test, please don’t everyone run out and run 100 threads against Adobe, I may become un-popular). Carefully perform only the actions that you want to be replicated in your test, then switch back to Jmeter and turn off the proxy to ensure no additional rogue web requests get captured. You should now have something that looks like this (I named my controllers just to make it easier to follow my plan). 
Now go back into your proxy object, configure it to record actions to the controller ‘Scenario B’. Restart the proxy and perform some other browsing actions that a typical user may do. Now we are ready to apply some ‘realistic’ touches to our test plan here. Looking at your test plan, you now see several controller groups under your ‘Scenario A’, each of which represents a web action. As I am sure you know, when you click on a link or go to a web page, often times other pages or assets get loaded. Each sampler (they look like an eyedropper) under the controller represents an actual HTTP request that occured during the action you recorded. Now, we want to add some human randomness here, because a machine slamming in web requests every 1 ms is really not a realistic test of your servers!
Right click on an individual sampler object, choose ‘add timer’, then ‘add gaussian random timer’. Set your ‘constant delay offset’ to 2000 ms, then set the deviation to 1000 ms. This will add a random delay to each action, in the range of 1000ms – 3000 ms (one to three seconds), which will more accurately represent a real person browsing your site.Now all that is left to do is add your listener elements, and run your tests! Remember, for good testing you always want to first get a baseline with your test of current performance, then test after making any site modifications. One of the hardest parts of creating a test is accurately modeling your real traffic. Search through user feedback, apache logs, google analytics exit logs, any data you can get to help you decide how you construct your test plan. The plan will do you no good at all if its not ‘real world’!Next time, I will discuss adding csv config elements so that user actions can pull search or post criteria from a file. 
DougHughes.net 