Richard Campbell Blogs Too

Greg and I talked to Brien Posey about securing Exchange Server. We ranged all over the map - there's so much to talk about in this space, depending on your Exchange configuration. The sheer variety of connection methods for Exchange, from VPNs to RPC over HTTP, using SSL, OWA and Outlook Mobile Access... how many holes in the firewall can one product poke?

Happy Halloween! Send us an email at info@runasradio.com with your comments and suggestions.

And then suddenly, its over.

The tradeshow at Interop is only two days: Wednesday and Thursday. Each day the booth was open for six and a half hours. Doesn't seem like much.

So why am I so tired?

We talked to a lot of folks at Interop about Strangeloop - almost all were IT and/or network folks. Some had no ASP.NET or no web site at all, so there wasn't much to talk about. But many more were very conscious of the fact that they had challenges with performance and scaling of their web sites.

One of my favorite visits was a fellow who said "I'm not responsible for the web site, but my boss needs to know about this" after seeing the entire demonstration. He took a data sheet and all sorts of info. Half an hour later he was back with his boss in tow and I did the whole pitch again. At the end his boss looked at him and said "you've got their info? Good."

Its a great feeling, being in the right place at the right time. And that seems to be the reaction of the majority of folks we talked to at Interop.

As soon as the show was over the booth came down. Amazing how quickly it came apart, considering how long it took to set up. The server rack was packed up for shipping in no time. I grabbed one of the big banners to take back to the office in Vancouver.

Tomorrow I head home for about a week, then Las Vegas for DevConnections!

Not much to say about Day 1 and 2 of Interop - we were too busy getting ready for the tradeshow to get to see any sessions at all.

Our booth design for Interop uses a 20x20 island, we're right beside Riverbed. The booth itself has two columns kitty-corner from each other, supporting a central span. This cuts the booth in half. On one side is the reception counter with scanners, documents, etc. The other half is the presentation area, where we have a small podium, a big screen and a bunch of cubes to sit on.

This photo is during set up on Tuesday... things are just about finished. You can see the server rack on the right in one of the columns. Its controlled by the laptop on the podium - this is where we do the live demonstrations of the AS1000. On the other side of that column and on both sides of the other column are demo stations for showing how the AS1000 works.

Folks really like this booth design, I'm thrilled with it. Lots of other vendors were coming over and taking pictures of it. I feel like we really utilized our space well, the live demo packs people in, and when they have more questions they can move over to the other demo stations to get answers.

We're doing three demos on the live station at Interop. One is done by Stephen Forte, the CTO of Corzen. He's talking about how the challenges of Corzen's business in rolling out new features quickly and constantly to their customers. Spending time on optimization is just not an option, they're looking to the AS1000 to minimize that concern.

Kent and I are also doing demonstrations, using Visual Studio for Testers to do load tests against the rack with and without the AS1000 to demonstrate how it improves performance and scalability.

Carl Franklin is here as well and is video taping the demonstrations so that we can put them up on the Strangeloop site for future viewing.

Greg and I talked to Doug Toombs about some of the tools an IT Pro ought to know about.

Tried on another new format for this show - spending just a couple of minutes on each tool before moving on to the next.

It was lots of fun for us, I hope it worked for you. Let us know at info@runasradio.com.

Back to New York again, this time for Interop.

Interop is a trade show focused on networking and mobility technologies. Monday and Tuesday are set up days, the trade floor is open Wednesday and Thursday.

So far with Strangeloop we've been primarily focused on the ASP.NET community, since our product is aimed squarely at that market. But we also have a foot in the networking space, after all the AS1000 lives in front of the web farm, typically an area populated with load balancers and firewalls, which are totally the domain of the networking guy.

Its going to be very interesting to me to see how that audience reacts to the AS1000.

The most exciting part of the show is the demo rack - a shippable 10U rack with a pair of AS1000s and several Dell 1950 1U servers for doing load tests and demonstrating how the AS1000 can help ASP.NET applications scale.

Greg and I talked to Tony Howlett and Glenn Kramer about Information Security. We get beyond protecting credit card numbers, talking about the depth of protecting an enterprise's information from the public and its own employees as well from intentional and unintentional breaches.

Tony and Glenn work together, and you can tell they've done this duet before - always fun to chat with another duo.

Let us know how this show works for you at info@runasradio.com.

Finally, after two weekends and hours of work, I get to do what I started out trying to do - building a six drive RAID 5 array out of terabyte hard drives. Cartman's old 5U case was all cleared out, I had all the components, now all I had to do was assemble the beast. I had a little problem with the Adaptec 3805 controller.

The 3805 is actually an SAS controller, using the mini-SAS plugs that handle four drives each. On the web site the specification says that the board comes with a pair of mini-SAS to SATA cables, but there were no such cables in my box. Turns out I had ordered the OEM version of the board (the only one available), and it had no cables in it - which makes sense, its an OEM board, the OEM is always going to want to do something unique with the board.

Fine, I'll order my own cables. But NOBODY has stock on mini-SAS cables. I flip out at the supplier, and he calls Adaptec, and they offer to give me a pair of cables for free (which was mighty nice of them), if I'll pay the shipping. Totally worth it, I had ordered the wrong product and they were willing to fix it. A FedEx overnight shipment later, I had cables.

There's so much room in the 5U case that things came together rather quickly. The motherboard dropped in without a hitch, as did the drive array caddy. Then came the tricky bit...

Like Cartman, Butters has a separate pair of mirrored boot drives, although in this case the drives are 7200rpm SATA II drives, rather than the Ultra-160 SCSI drives of Cartman.

In the 5U case, the boot drives hang from the card retaining bar... and the first hitch of the build occurs. In a test hanging (shown to the right), the pair of drives hit the CPU fans. This is bad.

When a situation like this arises, first you curse. Then the full reality of the situation hits - all the work you've done for the past few days may have been for naught, this machine won't fit into this case.

I ran into the same issue with Cartman during his rebuild, I had to modify the cooling blocks to use lower-profile fans to avoid conflicting with the hanging hard drives. But I didn't have that option this time... no handy low-profile fans, no alternative cooling blocks. I needed a different solution.

And here's the solution - move the drives. It's not like the new machine is full of cards anyway, it has exactly one, the Adaptec 3805 raid controller. And that card is low-profile anyway.

So I removed all the card holders from the bar and moved the mounting bracket so that the drives would hang away from the CPU fans. Problem solved. 

That was really the only hitch in the assembly of Butters, and it only took me a few minutes to solve it. I like this new drive position better, it puts the drives right inline with the main fan, so there'll be plenty of cooling air coming over those drives.

 A little more fussing with wiring and I was on my way with a successful boot of the new motherboard...

Notice that I plugged one of the 1TB drives into the machine as well, getting ready for the transfer of all that data back onto a shiny new 5TB array.

Ah, if only it was that easy. First I had to get a server install done. Which you think would be easy - a brand new motherboard, it should be no problem to get things up and going with Windows Server 2003, right?

Wrong.

Since I was planning to use this machine to run virtual machines, of course I wanted a 64 bit operating system on it - there's 16GB of RAM in there, how else would I address it all?

So I installed Windows Server 2003 SP2 64 bit edition. And the installation went cleanly, but didn't recognize the pair of built-in gigabit NICs. I wasn't all that surprised, after all, brand new motherboard, I'd need to install the drivers separately. Now if only I could find them.

On the Tyan web site you can see all sorts of drivers for the S2927, including drivers for Windows 2003 Server 64 bit, so you'd think there would be NIC drivers there. In fact, under the heading "Driver Packs" there is a pack for Windows 2003 Server 64 bit which SAYS it has LAN/NIC drivers. However, if you actually download it, there's no NIC drivers in there. In fact, if you open up the zip file, the README doc lists everything in the driver pack and it does NOT include the NIC drivers.

I tried installing it anyway, but to no avail - the NICs were still unrecognized.

However, the Adaptec software worked great AND the 5TB array was able to be built. But it was going to take more than 24 hours to prep itself, so it was worth tinkering with other configurations before settling for this.

So I headed over to the nVidia site... perhaps the reference drivers for the nVidia chipset would handle the NICs better. The chipset on this motherboard is the nVidia nForce Professional 3600 series. And lo and behold, there ARE reference drivers for Windows 2003 Server 64 bit. But they TOO could not recognize the NICs.

I even tried the prerelease tool on the download page to detect what drivers to use, and it recommended the Vista drivers! Figuring it couldn't be any worse, I tried them too... and this time the NICs were recognized, but were not functioning.

So now I'm afraid - afraid that my motherboard is defective. But now that I have nothing to lose I thought "what the heck, let's try Windows 2008 Server!" I had Release Candidate 0 handy, it was worth a shot.

Windows 2008 Server RC0 is a massive 2.5GB, I had to make a DVD for the install. But it installed flawlessly and recognized the motherboard, including the NICs. I was fully operational. And Windows 2008 Server is beautiful... but its a release candidate!

So now my motherboard was working perfectly, I installed the Adaptec RAID controller software. It installed, recognized the controller AND the drives. For the first time I had everything working, admittedly on a release candidate. How could I resist? I configured the 5TB array and let it rip.

The build ran overnight and finished perfectly. I had a 5TB drive array!

I shutdown Butters, closed it up and stuck it in the rack.

Powered it up again, but when it booted, there was no drive array! I rebooted again, still no array. What was going on? Pulled Butters back out of the rack, opened it up, booted it again... still no array.

I went into the 3805 BIOS to configure the array and it didn't show up until I selected "Refresh Array." Then it showed the complete array, in perfect condition!

Baffled, I exited the BIOS settings which caused a reboot... and the array vanished again. This time when I finished booting into Windows, I opened up the Adaptec configuration manager... it showed a failed controller and failed drivers. I selected "Refresh Array", and it still showed as everything failed - but Windows suddenly found the array! The drive letter popped up and everything acted fine.

Oddly enough, I was a bit suspicious.

So I started loading data onto the array. I wasn't going to erase any backups, so I waited for it to fail.

Loading went much faster than backing up, since the drive was plugged directly into the machine. Within a few hours, I had everything reloaded.

I was still suspicious.

I configured the file shares and got both the music and television archives up and running. They worked perfectly.

Now I really had a problem - I was running a release candidate OS, the configuration software says the array has failed (although the BIOS says its fine, once you refresh), but Windows itself is perfectly happy with it. And my family was happy to have the music and video back online. I couldn't very well take it back down. As long as it didn't reboot, the array seemed to stay up. Scary.

I sent a tech support request to Tyan, hopefully they'll have something useful to say. I really ought to go back to Windows 2003 Server 64 bit, but only if I can get the NICs to work.

My parts arrived during the week, but it wasn't until the weekend that I actually had time to start putting things together.

However, before I could build the new machine with the new parts, I had to get Cartman out of the 5U case, which meant moving him into the 4U case I had. The 4U case was populated with a rather old Linux machine that I hadn't powered up in a couple of years. So all of that came out, leaving a clean 4U case, ready for loading:

Once the case was clear, out came the rack again and Cartman shut down and pulled. I put Cartman on a separate table from my regular service table and gradually disassembled it and moved the parts into the 4U case.

Here you can see Cartman's motherboard loaded into the 4U case. Its an old Tyan board with dual P3 processors and 512MB of RAM. A great board in its day, its terribly dated now.

Notice also a pair of PCI-X slots, both of which are normally occupied - one with the Adaptec 29160 SCSI controller, the other with the Adaptec 2820SA RAID controller. The RAID controller just runs the big SATA storage array, the boot mirror runs off the SCSI controller, as well as the DVD drive and the external tape drive.

The drive array sits in the big gap on the left side of the case (right side if you're looking in the front), and the pair of boot drives live in the little gap full of wires between the drive array and the DVD.

Mounting the motherboard is always the trickiest bit of the build, once that's done, the rest goes quickly. The only difference between Cartman in the 5U case and Cartman in the 4U case is a four drive RAID array instead of six drives.

This is what Cartman looks like fully loaded into the 4U case and booting up. You can see the SCSI ribbon cable running to the pair of drives in the center front of the case and the four blue SATA cables running from the RAID controller to the array chassis. Even the DVD is SCSI, although where the drives are SCSI-160 LVD, the DVD is SCSI-40.

The other two cards in Cartman are the video and gigabit ethernet. This is an old machine, very little is onboard. But you can appreciate why I have to replace Cartman. All those things we're used to having right on the motherboard have to be added card-by-card.

Cartman was none-the-worse-for-wear in the move, still working the same way, with just the RAID array being down.

 In fact, as you can see from this shot at the front of the case, I did not install the terabyte drives into the chassis, since I currently have backup data on three of the drives, and I need six for the new array in the 5U case.

When I'm finally able to clean off those backup drives (when I believe everything is stable), I'll build a RAID 1+0 array using the old Adaptec controller. That will stay under the 2TB limit with the 1TB drives and give me the reliability and performance I want.

Ultimately, Cartman will be retired, but really only the motherboard. All the drives are fresh, what's needed is a new multi-core, multi-processor motherboard with a ton of RAM.

I'm thinking that since the 5U machine has an AMD motherboard, I'll put an Intel board in this machine. Probably something from ASUS, we'll see.

Cartman went back into the server closet without incident... tomorrow the 5U case gets a new motherboard, and a new machine will be born: Butters!

Our discussion with Joel Oleson on Sharepoint Management was largely a fallout of Sleepless in New York. Hanging out with Sharepoint guys invariably means talking Sharepoint. We tend to focus on the development of Sharepoint rather than the management of it, talking to Joel we got the IT viewpoint.

As always, email us at info@runasradio.com with your comments and questions.

So if you didn't get the hint, I'm upgrading the capacity of my servers in my server closet.

Last time I upgraded capacity it was in Cartman, migrating from a 400GB six drive RAID array to a 2TB six drive RAID array. With the new 1TB hard drives, I was ready to move to a 2TB six drive RAID array.

I bought ten Seagate Barracuda ES.2 1TB drives, six for Cartman, and four to go into a different rebuilt server.

So stage 1, actually started back on Friday, was to back everything up... rather than taking the chance of pulling Cartman and pulling into him directly, I hooked up the 1TB drives, one at a time, to my Phillip, one of my workstations, and copied everything off. It took until Sunday to finish the copy across three drives.

Here you see one of the drives getting loaded across the network the slow, but low risk way. Even with gigabit ethernet, transferring data from the old array on Cartman through the network, into Phillip and then out via SATA takes a long time.

By the way - they may call them 1TB drives, but they format to 933GB. That whole 1000 bytes vs. 1024 bytes thing is getting out of hand. It was fine when we were dealing with smaller drive, but when you're talking 933GB vs. 1000GB, that's 7% of the capacity of the drive missing. At some point you have to call foul - this is not a 1TB drive.

So I had three drives filled with the contents of the old array, that left me with seven drives empty to build the new array... although I only needed six.

Once the backup was finished, it was time to pull Cartman, which meant opening up the server closet and pulling the rack.

Here's the server rack pulled, Cartman is near the bottom, just above the 2000VA UPS. The long grey 1U box is an Exabyte 1x10 SCSI tape backup unit. You can also see the power supply of my temporary Exchange rig that has been running some two years as just a power supply, motherboard and hard drive sitting on a towel. I'm tempting fate, I know.

You can see how the rack pulls out on the rails, using folding arms in behind with cables running across the arms.

Beside the server rack, the second shot is the network rack that has the dual internet connections, all the patch bay wiring for network, telephone and cable. The 1U console is pulled out to shut down Cartman, its wired back to the server rack where the KVM switch is. 

Here's a look into Cartman for the first time in a couple of years:

Looks about the same as last time.

That's the end of the photos, because things went downhill from here and I stopped thinking camera and started thinking much meaner thoughts.

I carefully extracted the six 400GB drives that have been the 2TB array for the past couple of years. I figured I could always go back to the original drives. I replaced those drives with six blank 1TB drives. Fired up Cartman and built a new array.

The Adaptec 2810SA controller recognized the drives fine, but wouldn't create an array bigger than 2.1TB. It appears to be a hard limit of the controller. I upgraded firmware on the controller, to no avail. I tried configuring it in Windows 2003 Server and directly in the firmware, hit the same limit either way.

So much for that - now I have to make a choice. I could build two three drive arrays of 2TB each, or replace the controller. I wasn't going to sacrifice an extra drive for this, I needed a new controller.

So now that I admitted I needed new hardware, it was time to revisit my thoughts of hardware migration in general.

The original versions of these servers go all the way back to 2000, with upgrades on the way. One of the issues I've run into again and again is that migrating to new servers is hard, so hard that old servers are tough to retire, they just go on and on until they fail and you're forced to give them up. Cartman, after all, is a dual P3 machine, still going on. I've upgraded the OS, replaced the CPU fans, swapped the drives a couple of time... but its still an old machine.

My new vision of the rack is to go to completely virtualized servers. I want to build a pair of high performance multiple processor servers with lots of RAM and 64 bit operating systems running multiple virtual machines. I need a pair so that I can fail between them - they will back each other up and each should be capable of running the entire server farm itself.

Cartman, obviously, is not qualified for this job. So Cartman will have to go away eventually.

Through a series of unexpected events that I shall not go into in detail, I ended up in possession of a Tyan S2927 motherboard with a pair of AMD dual core processors and a bunch of RAM. This was a motherboard able to take on my virtualization mission - it just needed to find a place to live.

The need for a new RAID controller capable of handling arrays bigger than two terabytes gave me the excuse to make the big move - migrate Cartman out of the 5U case with six drive bays into the 4U case with four drive bays, and move the new Tyan motherboard into the case with a controller able to get me my 5TB array.

However, that meant I had to wait for more parts to arrive, which meant waiting a few days. I'm ordering in more RAM for the S2927 board (might as well fill it) and an Adaptec 3805, which will go into a PCI-e slot and handle the big array.

I ended my day by putting Cartman back in the rack and back online again, although without the drive array. We could live without the big storage for a few days.

Got home from Bulgaria to discover that my new hard drives had arrived!

What sort of drives you might ask? Why Seagate Barracude ES.2 1TB hard drives, of course! Because we all need ten terabytes.

Greg and I talked to Mitch Garvis about Microsoft BDD. We really dug into all the different options for automating the deployment of software in the enterprise, including SMS and the like. We also talk about the difference between no-touch and light-touch deployment. It's interesting to think about how the culture of a business affects what tools are appropriate for managing the workstations.

Let us know how we're doing with RunAs at info@runasradio.com.

Carl and I grabbed an interview with Dino Esposito in a quiet room during the conference, his viewpoint on Silverlight and ASP.NET technologies is always interesting.

Dino's session on "What Partial Rendering is not AJAX" rang true for me as well - his point is that the essence of AJAX is pushing page rendering to the browser, rather than computing it on the server. But partial rendering still computes the HTML on the server and sends it to the browser to display. This undermines the goal of AJAX.

I had last session of the day (and conference) and a huge crowd for my load testing talk today, as usual there were relatively few folks in the audience that had done load testing before, so a lot of my talk focused on the fundamentals of why and where for load testing. The data we've gathered around Strangeloop is great stuff for getting people started.

Sold out! Yep, the show is packed. Its not the biggest show in the world, but the attendees are focused and excited to be here. The keynote speech today included the local Microsoft folks and Telerik and, of course, Tim Huckaby! Tim's stories around building great applications that change the world are hard to touch. The audience was spellbound.

My work came in the afternoon, I took the Scaling Habits of ASP.NET Applications out for a spin again, with lots of interesting questions and discussion afterward.

In the evening Carl and I ran a panel discussion on WPF with Tim Huckaby, Brian Noyes and Todd Anglin.

Tomorrow is the last day, then we're touring Sofia!