Richard Campbell Blogs Too

At Tech Ed Orlando a bunch of the RDs got together to record ten minute videos called GrokTalks.

The idea of the GrokTalk came from the thought that often in conferences we find a useful tidbit in a session that is only about ten minutes long... so why not do only those ten minutes?

My GrokTalk focused on the new error handling capabilities of SQL Server 2005 and how you can use them to recover from a deadlock inside of a stored procedure. You can take a look at my video and the others at http://www.groktalk.net/.

...like a wife who's computer is dead because the water pump stopped working.

Not your usual every day computer problem either, is it?

I had the electrician in here on Wednesday, I wanted the circuits for the workstation bays rewired so that they passed through the server closet. Why? I wanted to move the UPSes for the workstation bays into the server closet. This would accomplish two things: reduced noise and more space in the tiny 12U workstation bays.

Since my electrician wired the place during the renovation, he didn't have a problem with what I was doing, why I was doing it and why it had to happen. He's learned that with me, weird is the norm. The whole thing was done in just a few hours.

Of course, while the electrical work was going on, the workstation circuits had to be disconnected, which meant all the workstations were off. My wife and I worked from our laptops for the few hours that the work took. When the work was done, however, there was a casuality - the water pump.

Naturally, my machines both powered up again just fine. But my wife's workstation wouldn't power up at all. I figured it was the power supply, and I always have a spare, so I pulled the machine out, popped the cover and plugged the power supply into just the main power of the board to see if it would start, and it did.

Feeling smug at my immediate diagnosis, I pulled all the power plugs off the gear in the machine, unmounted the power supply and performed the swap out, plugging everything back in. And it didn't work. Doncha love it when that happens?

Fortunately, this had happened to me before, and I am blessed with a pretty good memory when it comes to stupid things happening to me. So I unplugged the pump and turned the machine on. It wouldn't start. I switched the power supply off for a few seconds, then turned it back on again, then tried to power up the machine and it worked. So I plugged the pump in - boom, dead machine again. Unplug the pump, power the machine, no workie. Turn off the power supply, turn it back on, power the machine, and it works. See the pattern?

What's happening is that the controlling circuitry in the pump is causing a dead short in the power supply. The power supply, to protect itself, effectively shuts off and won't power up. Until you cycle the power supply itself, its not going to turn on again.

So, I've fried another pump. How? Beats me, it sucks. I go to my favorite supplier of Innovatek gear and discover there's a shiny NEW version of the pump available, with improved electronics. Hmmm - maybe its not just me? So I order the pump immediately with overnight shipping. Admittedly, it was late at night on Wednesday when I did this, so the order wasn't filled until Thursday.

Lo and behold, on Friday the pump actually showed up! Its a miracle! So now the fun of retrofitting a pump comes into play. And you know what that means - time to drain the system.

Ah, how awful life would be without a bulb pump. You can see I'm using the pump to push air into the system and force the water out into the yogurt container.

After the bulk of the water was drained out, I turned the machine up on its side and removed the side of the case - the only way to extract the pump.

The reservoir is mounted to the pump, which is at the lowest part of the case, normally. Turning it up on its side drains the last bit of water out of it, trying to minimize the mess - I've learned this from experience.

The reservoir is pressure-fit onto the pump and takes some twisting to get off the pump. You can see in the photo that the reservoir is sitting on top of the case, its rubber-gasketed pump mount visible.

Here you can see the pump has been removed, its slide off base still in place. What I had forgotten was that the pump slides off backward (down) into the case. I had to pull the drive assembly out a few inches to get the pump loose.

The new pump dropped into place easily enough, its the updated model of the old pump, hopefully with this short-out problem resolved. I've had two pumps gone this way now, admittedly both were a couple of years old.

Once the pump is reinstalled, the reservoir is pressed into place, and then the hoses are fitted back on.

Just to complicate matters further, I swapped out the old GEForce4 video card for my more advanced ATI Radeon 9800XT with the double-sided water jacket.

With everything hooked back up, it was time to put the case back together and get things running again. These rackmount cases are awesome, but unfortunately no longer available anywhere. They're true workstation cases - no locking face plate, and there's no rivets or welds anywhere, the entire case is assembled with screws, so that every part can be removed.

Add in the lucky coincidence that the standard Innovatek radiator fits in the case along with a 120mm fan and you have, in my opinion, the best darn rack mount water cooled PC case possible. That's why I have three of 'em.

The old pump sitting outside the case now, you can see I have the pump bypass plugged into the power supply to run with pump without powering up the machine. Distilled water and a little Innovatek water conditioner are added, I discard the old water.

After some time tapping and burping lines to get all the bubbles out, the water loop ran steady, so it was time to power up fully. The machine came to life without consequence, recognized the new video card and everything was good to go.

For the moment the machine is back in its workstation bay, cover off while I check temperatures and keep an eye on things in general. Once you've breached a water loop, its worth keeping an eye on it for awhile to make sure its not leaking or anything stupid is happening.

Meantime, I still have to actually take advantage of the electrical changes and shuffle my UPSes around.

Wasted a good hour cleaning all the blog spam out of my blog.

Upgraded to the latest version of dasBlog as well to keep the crap out.

What is wrong with these people?

I hinted back before Tech Ed that Phillip, my big gaming machine, had died. Actually, it died only a couple of weeks after I put it together, I just didn't want to talk about it.

One thing I noticed about this machine when I built it was that it did run hot. My other machines can keep their temperatures under 35C with no load, this one with no load struggled with 45C. Add in SETI@Home working the processor at full bore and 45C requires at least 50% fan power. And then there are those darn video cards...

So one day I finally sat down to try out the full potential of the new machine with its lovely, top-of-the-line SLI video cards. I set up Half Life 2 running in 1920x1200 mode. It runs smoothly at around 100fps, in the really gnarly stuff it gets as low as 70... obnoxious, innit?

Enjoying myself immensely, I set off on a campaign of maximum destruction in Half Life 2, enjoying the view, when I feel the heat on my back. I turn around to see the temperature of the water loop hit 75C. Yeop, most of the way to boiling the water. I shut down HL2 to get rid of the load, but the damage was done. Within minutes the machine had died and wasn't coming back. Motherboard baked.

I went and did some math and discovered that each video card ran at about 80 watts. The processor generated only 55W! So when the video cards were working hard, the machine cooked.

Now I had two problems - the first was fixing the machine, which meant a motherboard transplant. This is not normally something I fear, but with watercooling its much more difficult. Especially the water cooling on this machine, with two video cards and a Northbridge chip right between them. The hoses are short and twist all over the place. And the last thing you want to do is breach the water loop.

Here's Phillip sitting on the service desk. Notice I plugged a speaker in to get a listen to any BIOS error beeps. Unfortunately there were none, supporting my belief that the motherboard was cooked.

My first attempt at extraction was to pop both video cards out of their slots. I powered up again at this point in the hopes that perhaps the video cards were dead and now I'd get a missing video card beep pattern - alas, no luck, no noise, no nothing. The machine is still dead. I'd have to peel all the water cooling gear off.

The motherboard extracted. What you didn't see is that I had to unscrew the motherboard from the case and pull it clear of the jacks in the back, then lift the board up with the water gear still on it. The problem was the Northbridge chip, which had a pair of nylon nut-and-bolt sets holding it on. The only way to get those off was to get to the nuts under the board.

The CPU was a bit tricky to remove just because there was so much surface area, but twisting and prying got the block off.

With the motherboard free, I cleaned everything up and transferred the RAM and processor to the new motherboard. In addition I removed the Northbridge fan from the new board (and put it onto the old board so it would be stock again and RMA-able).

Here the new motherboard is all prepped with fresh thermal grease, ready to be installed. And yes, I would remember to clean off the CPU block before I mounted it on the CPU again.

Remounting the water gear on board starts with the Northbridge chip block, since it once again has to be bolted down from the back. The board sits in the case at an angle so that I can get to the back of it, and I slid the block and bolts into the holes, then gently placed the nuts on the bolts until the threads catch. Then its a process of turning each nut a bit so that the block is squarely over the Northbridge chip.

Once the Northbridge is in place, everything else is lifted up so that the board can slide into place. The motherboard is screwed down and then the video cards went into place, allowing the CPU block to be replaced as well. Then the power/reset switch plugs, power/hard drive LED plugs, USB plug for the Matrix Orbital controller, SATA plug for the hard drive, IDE plug for the two DVD drives, and then all the power plugs for the motherboard (there are three: main plug, secondary 12 volt and a molex for the video) plus the additional power plugs for the video cards.

A quick top-up of water into the reservoir and I was ready to power up again for the first time in more than a month.

And the beast lives! If you look close, the screen is stopped on a BIOS error because there's no CPU fan. Which is a reasonable error since there is no CPU fan. Some quick BIOS tweaks took care of that.

So, remember when I said there was two problems? The first one is now resolved - the machine is back to life with a motherboard transplant. Problem number two is how to avoid cooking the motherboard again. Within minutes of powering up, running no high load software (like SETI@Home), the machine is already at 44C. Add SETI@Home and the temperature immediately rises a couple of degrees, causing the fan controller to turn up the fan to cool it back down again.

Fire up Half Life 2... well, I wasn't going to do that again.

I found the answer at Sprite - the guys I get most of my gear from. For whatever reason, they happened to have an Innovatek RADI-Dual in stock. I don't know why, they'd never sell the thing... well, okay, maybe not never.

This radiator is twice the size of the ones that I use in the case, and has mountings for two 120mm fans. It wouldn't fit in the case, but it would offer a whole bunch more cooling. Would it be enough? With it immediately available, it was too easy not to try it.

I mounted a pair of ultra-quiet Vantec 120mm fans, directly powered... I've burned up a couple of these lovely fans with controllers, so I didn't want to take the chance. And besides, even at full power these fans only generate 28dB of noise, so you can't hear 'em at all.

To connect the radiator into the loop I disconnected the top-side connector of the existing radiator and moved it to the bottom feed on the new radiator, then added a new hose from the top connector of the new radiator down to the old radiator. Powered up and started adding water to the pump reservoir as fast as I could to fill that new radiator. Several ounces later, everything was full and ticking along.

Its a little on the Mad Max side of things, but it sure does work!

Check out the front view of the machine, you can see the temperature of the water - just below 31C!

When I fired up SETI@Home, the temperature didn't move at all. So then the real test: play some Half Life 2. After one hour of play, the water temperature got to 32C. Methinks the fix is in!

Obviously, the system can't stay like this. But I'm afraid the only real answer to this problem is going to be much more radical: converting to central water cooling. That would involve putting a set of pumps, radiators and reservoir inside the server closet and running hoses through the walls to the two workstation bays in the office. The same way that you have a wall plate for power and a wall plate for network access, there would be a wall plate with water input and output. Then you'd just plug the machines in.

There are a bunch of advantages to the central water cooling solution. The first is that there will be a lot more water, and that water will be chilled. So the ability to cool will increase substantially. The machines will be even quieter having no fans (except the whisper fans in the power supply), no pumps and no radiators. Another huge bonus will be that the heat of the machines will actually be taken out of the room, being dumped into the server closet with its great big AC unit.

The downside is that the machines are no longer self-contained for cooling. When I have to service them, I'd need to use an external water cooling module, something like the CoolerMaster Aquagate or the Koolance Exos 2. All resolvable stuff.

So, for the moment, everything seems to be functioning here in water cooling land. I'm watching Phillip closely for any water leaks, I'm a bit concerned that the heat event might have damaged some seals. But so far, so good.

I had every intention of blogging through Tech Ed, but it didn't happen.

One week ago today (Monday), I was walking from my hotel room to Tech Ed in a really foul mood. Being grouchy is rather unusual for me, but you can ask Steve Forte and Cathi Gero, they were there, and boy, was I grouchy.

I guess its been a few years since I've done back-to-back conferences, having done the SDC conference in the Netherlands the previous week, my body was trying to tell me I should be at home by then.

All of this changed when I got on stage with Steve to do our Advanced Querying session.

Y'know, speaking at conferences is really a money losing proposition for the majority of speakers, myself included. I would be making more money staying at home and working. But its really, really fun. Really. Engaging a big group of people (and there was about 800 people in the room) is a challenge, its exciting, and when it goes well, you're in orbit for the rest of the day. And I think it went pretty well - lots of laughter and ooh-aahs.

So to my audience at my first session: THANKS! You made my week.

Some folks have been emailing me, unable to find the samples for the session. I've attached two files here, the first is the setup file which creates the sample tables.

Setup.sql (3.57 KB)

The second is the demo script itself with all the queries Steve and I showed.

SS2k-YukonSamples.sql (24.42 KB)

We got a ton of response on this session, and some cool new ideas for a new version next year.

My second session on Tuesday was the Profiler session, with my special guest Vipul Shah who jumped in to show off the cool new features of Profiler in SQL Server 2005. Its a cool session, but I think with Profiler 2005 coming, I'm going to have to rewrite it to really dig into the new capabilities of the tool.

As I explained in the session, my real focus on the Profiler session was to let developers know that things can happen to your queries between your code and SQL Server, and Profiler is really the only way to know. The big example I show is ADO 2.5 messing with a SELECT statement and stored procedure, wrapping them in cursors. I haven't found the same behaviour in ADO.NET, so I think the demo is getting moot.

I'm thinking next year I'll revise the session to make it more of a “Using Profiler as a Diagnostic Tool” type session.

On Wednesday Carl Franklin and I did .NET Rocks! in front of a live audience. I think there were close to a thousand people in the room, which was at the far end of the conference center. And I do mean the far end - I figured by the time we got there we'd walked to Cuba. We interviewed the Team System guys, I think the show went really well, it was fun to dig into more of the story behind Team System... and even better to have a bunch of fans watching the show!

Alas, my version of the standard DNR disclaimer (normally Geoff's domain) didn't make the cut of the show, but either way, a good listen.

Tech Ed may be over, but the Tech Ed Charity Auction isn't. 23 Tech Ed speakers, including me, are donating an hour of consulting time via phone, email or IM. You can bid on EBay at http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=5587400881. The auction ends on June 16, so bid soon, and bid often.

After the SDC conference I flew back to North America, but not home. Instead I've stopped off for a couple of days with Carl Franklin here in New London, Connecticut.

Last night we recorded Mondays, with everyone except Mark Miller actually here in the studios. We laughed til we cried, it was quite ridiculous and a wickedly fun show.

This morning we're recording an episode of DotNetRocks with Michele Bustamente, then Carl and I are packing up and heading down to Tech Ed in Orlando.

What can you say about New London? Its got a far greater sense of history than we have on the west coast, around here a 100 year old building is still considered pretty new, people are proud of pointing out structures that were built before the War of Independence. It strikes me as a fabulous place to raise a family, which is of course exactly what Carl and his wife Gretchen are doing.

When we say Beer Night, we mean it!

Near the beginning of this evening session, Kent Alstad and Remi Caron conspired to bring Steve and I these huge two liter beers in Heineken labelled boots. I finished mine, Steve didn't. Not that he wasted it, he filled a dozen regular sized glasses from the boot.

The session itself was a melage of SQL Server 2005 topics, our slide deck and five slides: the title slide called the session “Estaban, 'splain dis Jukon to me!” It got less serious from there.

We had a fine discussion on the horror and fear around using the CLR inside of SQL Server, Ted Neward and Markus Egger got into it before Steve put the brakes on what was turning into an entirely too serious discussion.

I have to thank Kim Tripp for firmly planting all the details of Snapshot Isolation into my head, I think I was able to deliver a coherent explanation to everyone when the topic came up.

The beer continued to flow after the session, but we still knocked out some good talks the next morning. The conference was a ton of fun, its nice to see the SDGN group growing bigger and better still.

The day after the conference we went on the traditional Holland Tour, although this year it wasn't traditional at all... we actually got to sleep in a bit, rather than leaving at 8am as we have many years, we weren't loaded up and gone until 10! First stop was the Airborne Museum in Arnhem, which showed the history of the northern-most (and unsuccessful) part of operation Market-Garden.

Suitably subdued, our next stop after doing several spirals around the Netherlands was a paintball center!

The paintball matches were about 10 on 10, doing various missions: capture the flag, deliver your flag, and the Blackhawk down scenario, where you have a VIP stuck in the helicopter and have to go in and rescue them.

Our team won three of four matches, and I have to say that even our loss was pretty sketchy. I'd attribute our success to some great players, including the crazy fire teams of Steve Forte and Arnot, Ted Neward and Kent Alstad, Remi and Rob and the implacable Cathi Gero, who knows exactly what a VIP should do - stay alive and book it down the trail!