We’ve been going through beta testing at Strangeloop, which means I’ve had the chance to do some serious scaling of ASP.NET. One of the interesting experiences that keeps coming up in this process is the reaction we get from customers when we’re helping them do load testing.
One of the things we can offer our early beta test customers is the opportunity to load test their site, with and without Mobius in the loop. We need the test data anyway, and quite a few candidates don’t really have much in the way of load testing resources ready to go. And then we test their site in our lab with our Spirent Avalanche, and they go “Wow! I need one of those!”
So what’s a Spirent Avalanche, you ask? Funny you should ask… It’s 3Us of load testing love.
Josh Bixby, our VP of product development, noticed it when he was at trade shows. One of the benefits of having our feet in both the development camp and the networking camp is that we naturally see things on the network side that a lot of developers don’t. Josh pointed out that virtually every company making networking appliances had one of these 3U boxes in their demo racks. But I’d never heard of it before. So we checked it out, and realized it was the best answer I’ve ever seen to doing load testing. I know that load testing isn’t something people want to think about unless they HAVE to think about it. But if you do have to think about it, you have to check this out.
I don’t need to emphasize how much of a pain load testing is. Typically, you have two options, both of which suck: If you’re doing it yourself, you may be spending literally a week setting up a load test farm, and you’re probably spending more energy making the configuration work than actually doing the test. Which is no surprise, since most likely you’re using any piece of junk you can find, trying to network together a bunch of machines with different NICs, different performance, different speeds, etc., before you even begin to configure the test. I had one customer that bought me ten identical, dedicated servers for load testing - for about the same cost as an Avalanche - but that’s the exception, not the rule. And it still gives you much less control, you have to do all your own analytics, etc.
It’s easy to think “Oh, I’ll just use Mercury Interactive (sorry, HP Mercury) to do my load testing.” Easy until you see the price. Paying six digits for load testing with a 20% annual maintenance contract isn’t so easy. And that’s just for software – you still supply the hardware. I don’t think anyone told Mercury that the Dot Com Boom was over.
So taking a page from the network guys, there’s a third way to do load testing: You get a Spirent Avalanche, hook it up, and let it do the job. One 3U box with four gigabit Ethernet ports that can generate nearly two million users by itself. So you’ve got the hardware and the software all in one box.
Of course, the Avalanche isn’t cheap either, although they’ve nailed the gradually pregnant business model well – you can rent the gear, and those rental charges get applied to a purchase. We spent less than $100,000 on our 2700 with all the features we needed to do web testing. It also uses TCL-based scripting, which is usually the realm of networking guys, not developers, and can be difficult to understand. TCL provides the Avalanche with the flexibility to do load testing on a lot more than just web stuff.
However, bundled with the Avalanche is a product called TracePlus/Web Detective (Spirent Edition), made by System Software Technology (SST). SST makes a variety of different TracePlus products for networking and web, including this version specifically for working with the Avalanche. TracePlus provides the classic capture mechanisms that you see with most load generating tools, where the tool captures your navigation of the web pages and captures them as HTTP commands. The Avalanche internally converts this to its TCL commands.
The Avalanche has some ability to do reporting internally (pretty graphs), but the main way we’re using it is in “Excel mode”, where it generates CSV files that we can load into spreadsheets for analysis.
We’re also finding that the Avalanche doesn’t understand ASP.NET things like viewstate very well, but then, neither does WAST. We’re using Visual Studio 2005 Team Edition for Testers to get really smart functional testing around specific ASP.NET features.
Even with these complications, it’s such a better way to do load testing than setting up servers, and infinitely better than letting your paying customers do the testing. So if you’re doing load testing, why aren’t you using one of these? Why don’t more people know about this? This is pretty standard equipment if you build networking gear. It’s not like the Avalanche is some new, earth-shattering product. It’s not even mentioned on the main page of Spirent’s Web site?!?
I have yet to find anyone else in the ASP.NET world using a Spirent Avalanche. I really think it’s just a cultural issue, where great stuff is getting lost in translation between the networking world and the Web development world.
Important lesson: If you’re not paying attention to the networking space, you should be. You may just be wasting your time wrestling with a problem that other smart people have already solved. That’s one of the cool things about working with Strangeloop; we really get to straddle the line between those two worlds.