Richard Campbell Blogs Too

So Patch Tuesday came by as usual, and machines were patched. I have one workstation, Terrance, that still isn't running Vista. Terrance is the giant triple screen machine, running a total of 4960x1600 worth of displays. Plugged into it is the MOTU Traveller that I do all my recording with for RunAs Radio and .NET Rocks. So needless to say, it needs to be completely reliable, since I record every week.

So, as I said, Patch Tuesday came by as usual. And as usual, Terrance didn't auto-install the patches, just downloaded 'em and let me know. And, as usual, I hit the button. And, as usual, the patches required a reboot. What could go wrong?

Terrance didn't come back.

Terrance sits at the bottom of the stack of two workstations, sitting underneath Phillip. Phillip is a far crankier computer, struggling with barely sufficient cooling. So I'm usually tinkering with Phillip to keep him happy, while Terrance is totally reliable.

Except that Terrance didn't come back.

I left Terrance off for a couple of days. I had already finished recording for the week, so I had until the following Tuesday to get things fixed up, and I figured I could wait for the weekend.

When the weekend came, I hauled Terrance out of the rack, which means shutting down and removing Phillip as well. Annoying. Moved Terrance up to the service counter and re-installed Phillip so at least I had one workstation up and running.

With the cover off and plugged into the test harness, Terrance still wouldn't power up. Well, wouldn't power up is a bit of an exaggeration... the motherboard power light is on, but the main power light wouldn't turn on, and it seemed like the power switch was useless. Sometimes I'd see the LEDs on the RAM chips sequence a bit, but the drives never spun up.

I suspected that my groovy new water cooled power supply might be the culprit. Fortunately, I have a power supply tester, so I plugged it in and powered it up.

Here's a look inside Terrance. You can see the dual video card set up (running a pair of nVidia 7800s) that are both water cooled, along with the CPU, northbridge, southbridge, hard drive and power supply. This is also the moment of truth, with the power supply tester on the left plugged into the water cooled power supply. The power supply is the blue thing on the lower right of the photo, the black block attached to it is the water cooled part. The heat sinks for the power supply are connected to the block and water passes through the block to cool it. No fan in the power supply.

So, turn on power supply, and the power supply tester should report voltages, all that good stuff.

Only it doesn't do anything.

Being the suspicious type, I pull out my spare power supply, a nice Enermax Whisper unit, not water cooled, but nice and quiet. Plug power supply tester into that, fire it up, and everything lights up. Ah ha, one dead power supply.

So, to replace the power supply, I need to breach the water loop. I hate breaching the water loop, its messy. But, Terrance is one of my external water loop equipped machines. That means I have a pair of hoses running out the back of the machine that can be connected together to be self-contained, or connected to a larger external cooling system. Ultimately my plan is to plug into the wall for water cooling, but goodness knows when I'm going to have time to finish that.

But, back to the problem at hand - I have hoses with self-sealing couplings running out the back. That makes draining the water system a whole lot simpler.

In my hand I have a bulb pump which is connected to one side of the external loop. The other side has an unsealed coupler connected to it and is stuffed into a clean yogurt container to catch all the water. 20-30 squeezes of the bulb pump later, all the water is drained out of the system into the yogurt container.

Now to actually breach the loop - doing the pump out isn't really a breach anymore because I use those self-sealing couplings to keep everything tidy. The trick to removing things from a water cooled machine is to realize they still have some water in them, so closing off the water connections is a good idea.

On Terrance, the power supply was added to the water loop between the southbridge block and the hard drive block. So I want to connect the southbridge hose to the hard drive and leave the power supply out. But to keep the power supply from leaking, I connected the existing hose running between the power supply and the hard drive to the other end of the power supply.

So you can see the hose loop on the two water cooling connectors of the power supply, and in my hand is the hose that used to run to the power supply from the southbridge chip. I just had to rotate that hose around and connect it back to the hard drive and everything was sealed up again, no mess.

So the rest was easy - extract the power supply, install the Enermax supply, reconnect the power supply and I'm off, right!

Here's a shot of the freshly wired in Enermax supply. Looks good, huh?

Except for the part where it doesn't power up at all. Motherboard light turns on, but the machine won't power up. I'm suspicious. So I retest the old water cooled power supply. Its still dead. But something else is wrong.

Experience has shown me that the one thing that can stop a water cooled computer in its tracks is a bad pump. So I unplug the pump and the machine powers up normally. I have a bad pump as well. Good news, I have a spare pump.

For whatever reason, I failed to photograph the pump installation. Its very tricky, I had to turn Terrance on his side, remove the side plate (five screws off the bottom, five screws off the side, four screws off each end), disconnect the hoses from the reservoir, wiggle the reservoir off (its press fit), slide the pump out, slide in a new pump, remount the reservoir, reconnect the hoses, put the side plate back on, reinstall all the screws. Apparently I was so busy I never snapped a photo the whole time.

This is a shot immediately after finishing the pump swap-out. The dead pump is out of the machine, the new pump is in the machine. Everything else is the same, and Terrance now works.

Refill of water was pretty simple, just power up and keep pouring water. Once the lines had been burped (leaving it running for a half hour or so with the reservoir lid off), I buttoned everything up and put Terrance back in place.

Total service time, about two hours.

So, questions: did the pump kill the power supply, or the power supply kill the pump? And what killed either one or both?

And I need to restock on spare parts.

So here's a funny story.

Not counting the post from June 2006 where I provided my sample code for TechEd US in Boston, it was about a year ago since my last blog post.

I was the middle of some serious water cooling upgrades, PWOP had a bunch of new shows under way and things were just hopping.

And I got the fish tank in my office rebuilt and up and running with a new tank, new plumbing and some lovely salt water fish.

It lasted about 12 hours, then it burst. 120 gallons of salt water all over the floor.

Took six months to clean up.

I didn't want to talk about it. It was very difficult to get work done, to do much of anything.

But that's all over now, and I miss you guys.

And there's a TON to talk about! New machines, new companies, new shows! All good stuff.

I'm in Orlando at the moment at DevConnections. Carl and I are going to interview Scott Guthrie in front of an audience tomorrow. That'll be fun.

I'll keep you posted. Promise.

The nVidia 6800 Ultras that used to inhabit Phillip were the bane of my water cooling plans. I water cool for the quiet, not the performance. But these 6800s are so hot, my quietness plan was all messed up.

And its entirely my fault, too. I bought waaay too much video card. I wanted to experiment with SLI, using two video cards to run one display. Granted, the one display was a Samsung 243T with a native resolution of 1920x1200. The cards performed amazingly well, except that they were so hot, I ended up adding an additional radiator and two fans to the system to keep it cool.

The latest refit retired these toaster ovens, and I figured rather than let them sit there and rot, I'd let me friendly neighborhood computer store resell them for me. In fact, the owner came and picked them up, he had them sold before I was ready.

Unfortunately, the new owner wasn't into water cooling, so I had to convert these water cooled video cards back into air cooled ones. I had kept all the fan equipment in the original boxes, so it wasn't tough to find. Reassembly, however, is tricky.

Freshly removed from Phillip and drained of water, one water-cooled nVidia 6800 Ultra.

Four center plate screws, six spring loaded edge screws and two voltage regular mount screws later, the water cooling block is removed from the board. Notice the less than perfect impressions on the cooling block from the RAM chips of the video card. The system was never unstable, but it sure looks like this block wasn't as tightly fitted as it could be.

Deploying all the air cooling hardware. The copper block goes onto the GPU (along with the black backing plate), the angled aluminum block with the white blobs on it goes onto the RAM chips, the voltage regulator heat sink is the bottom right hand corner of the picture, and the fan assembly itself is in the top right hand corner.

After cleaning off the old thermal paste, I applied new stuff to the GPU, used the original white contact pads for the RAM, and carefully put everything together. The copper GPU plate goes on first, using spring loaded screws that go through the plate, board and into the black backing plate. Then the RAM cooler goes on with six different spring loaded screws. The comes the heat sink for the voltage regulators, held on with a pair of spring loaded clips. Finally, the fan itself is held on by three screws and plugged into the board.

Innit purdy? I like the water cooled version better myself.

That was the first one, the second one was even easier. Then a careful repack into the box, including power cables and DVI-VGA adapters for each.

These video cards were not a great investment for me - I think they were worth about 20% of what I paid for them a year later. Not counting the water jackets, which I still have and I can't imagine what I'll do with them. Maybe EBay.

Before I could get into rebuilding Terrance, my triple-screen system, there was an obstacle that had to be resolved first: Phillip, the SLI system that sits on top of it.

This is what my workstation rack looked like - at the bottom, barely visible, is a Minuteman 1000RM E rackmount UPS. I had my electrician rewire the outlets in my workstation bays so that the power passed through the rack closet, so this UPS could move into the rack closet, saving me 2Us and two fans.

Above the UPS is the Terrance, triple-screen system. It was the quietest thing in the stack, a P4 based system with a Matrox Parhelia to drive the three Viewsonic 18" displays. And sitting on top was the problem child: Phillip, the AMD-based gaming system with a pair of nVidia 6800 Ultras configured for SLI. This is plugged into a Samsung 243T. This rig put out 100 frames per second in Half Life 2 at 1920x1200. It also nearly melted in the process. The 6800 Ultras are just too hot. I ended up strapping a big radiator to the top of the case with a pair of Vantec 120mm Stealth fans mounted on it. Yes I know: fan bad. But melting worse.

Here's a top view of the SLI system, you can see the size of the additional radiator. This kept the system cool even under heavy SLI use. In exchange, of course, for ugliness and noise. Whenever I would record .NET Rocks! I'd have to turn this machine off.

The solution was to get rid of the 6800 Ultras. I considered going with later model SLI cards, say a pair of 7800GTs. These are actually cooler than the 6800s, and have more horsepower. Then I got a look at ATI's X1900. A 512MB video card with comparable performance to many SLI systems. In one card. How great is that? So I switched - trade in the 6800 Ultras for one Sapphire X1900XT (and a bunch of money).

I'm very much of the mindset that anything worth doing is worth doing excessively. And since I was going to totally overhaul Terrance, why not do the same for Phillip? The problem was, there really wasn't much better than the existing gear. The ASUS A8N SLI motherboard is great. The AMD processor in it, granted a single-core 4000+, but still a great processor. 7200rpm hard drive, dual burners... what could I really do to improve it? The new video card gets rid of the heat problem, so other than that, a couple of gigs of stinky fast Corsair RAM is all I could come up with.

The upside to this is that it meant I had one machine that would stay operational - it didn't need to have a scratch re-install because I wasn't changing the motherboard, just the video card.

However, it also meant breaching the water loop of the biggest, ugliest water cooled machine I've ever built.

The number one problem you face when breaching a water loop is how to do it without making a mess. The first thing I always do is take the cap off the reservoir, which allows air in to the system. The water loop is more or less air tight, so creating some pressure relief lets water leave the lines. Next I open up the highest point in the loop, which is normally the top of the radiator. In this case (look at the photo above) the top of the radiator is quite high up, and the line is essentially dry when the pump isn't running.

Ordinarily I'd use my little bulb pump to force all the water out of the system, but since its fatal encounter with the resident terrier, it was up to my lungs. So I pulled the line from the top of the upper radiator, then added a bit of hose onto the radiator connector and aimed it at ye olde yogurt container. Then I blew into the other end. And blew, and blew. There's a lot of water in the system.

Eventually I drained enough that the lower line of the upper radiator was also dry, and then I pulled that off as well, and reconnected it to the upper connector of the lower radiator.

That got the upper radiator out of the loop. I was careful in actually removing it because it still had a lot of water in it. I had to rotate it a bunch of times to get the majority of the water out.

I took a break at this point, you can see in the above photo the now completed water loop without the additional radiator. This is how I originally configured the system until I discovered that 6800 Ultras run at sun-like temperatures.

Next step, extract the 6800s.

The 6800s were plumbed into the system between the processor block and the Northbridge block, which is between the two video cards. And boy, was that ever fun to get together the first time. However, getting them out wasn't so bad - the connectors for the water blocks sit relatively high up, so with most of the water out, the lines were pretty much high and dry. I had to cut a new segment of hose to run between the processor block and Northbridge block.

Once I got the cards out, this is what I found.

The second video card in the SLI pair had sprung a leak. That goop is from the water loop dripping down into and beside the PCI-E slot. Beats me why the thing still worked. I wasn't all that concerned, since all this was all on the second PCI-E slot, and I was switching to a single card. Notice I've already turned the ASUS Patent Pending SLI mode card over, although I don't think its actually inserted correctly...

You can see where the leak came off the water jacket and dripped down onto the motherboard. I strongly suspect I melted the seals on this water jacket when it overheated... before I realized I needed a second radiator for it.

I'd worry about the 6800 Ultras later. Now it was time to fit the new video card and get things back up and running again.

You can see the new card and the new hose running from CPU to Northbridge. It looks too high in this photo, but it wasn't. Unforunately, Innovatek hasn't made a water jacket for the X1900XT yet, so I'm going to have to leave the fan on the video card for now. The good news is that its speed sensitive, so when I'm not running anything graphically intense, its pretty quiet.

One interesting problem was that the power adapter cord that came with the video card was only a four prong cable, and there's a six prong plug on the board. I tried it, and it didn't work - the machine kept coming up with a BIOS level error on the display saying "plug power into the video card." I used one of my six prong spares and it powered up fine. Price of being first with one of these cards, I guess.

What isn't in the above photo is the 2GB matched pair of Corsair 3500LLPro I stuffed in, fast response RAM with lots of head room and blinky lights.

Phillip powered up fine in this new configuration, and Half-Life 2 plays great on it.

One machine down, one to go.

...like a wife who's computer is dead because the water pump stopped working.

Not your usual every day computer problem either, is it?

I had the electrician in here on Wednesday, I wanted the circuits for the workstation bays rewired so that they passed through the server closet. Why? I wanted to move the UPSes for the workstation bays into the server closet. This would accomplish two things: reduced noise and more space in the tiny 12U workstation bays.

Since my electrician wired the place during the renovation, he didn't have a problem with what I was doing, why I was doing it and why it had to happen. He's learned that with me, weird is the norm. The whole thing was done in just a few hours.

Of course, while the electrical work was going on, the workstation circuits had to be disconnected, which meant all the workstations were off. My wife and I worked from our laptops for the few hours that the work took. When the work was done, however, there was a casuality - the water pump.

Naturally, my machines both powered up again just fine. But my wife's workstation wouldn't power up at all. I figured it was the power supply, and I always have a spare, so I pulled the machine out, popped the cover and plugged the power supply into just the main power of the board to see if it would start, and it did.

Feeling smug at my immediate diagnosis, I pulled all the power plugs off the gear in the machine, unmounted the power supply and performed the swap out, plugging everything back in. And it didn't work. Doncha love it when that happens?

Fortunately, this had happened to me before, and I am blessed with a pretty good memory when it comes to stupid things happening to me. So I unplugged the pump and turned the machine on. It wouldn't start. I switched the power supply off for a few seconds, then turned it back on again, then tried to power up the machine and it worked. So I plugged the pump in - boom, dead machine again. Unplug the pump, power the machine, no workie. Turn off the power supply, turn it back on, power the machine, and it works. See the pattern?

What's happening is that the controlling circuitry in the pump is causing a dead short in the power supply. The power supply, to protect itself, effectively shuts off and won't power up. Until you cycle the power supply itself, its not going to turn on again.

So, I've fried another pump. How? Beats me, it sucks. I go to my favorite supplier of Innovatek gear and discover there's a shiny NEW version of the pump available, with improved electronics. Hmmm - maybe its not just me? So I order the pump immediately with overnight shipping. Admittedly, it was late at night on Wednesday when I did this, so the order wasn't filled until Thursday.

Lo and behold, on Friday the pump actually showed up! Its a miracle! So now the fun of retrofitting a pump comes into play. And you know what that means - time to drain the system.

Ah, how awful life would be without a bulb pump. You can see I'm using the pump to push air into the system and force the water out into the yogurt container.

After the bulk of the water was drained out, I turned the machine up on its side and removed the side of the case - the only way to extract the pump.

The reservoir is mounted to the pump, which is at the lowest part of the case, normally. Turning it up on its side drains the last bit of water out of it, trying to minimize the mess - I've learned this from experience.

The reservoir is pressure-fit onto the pump and takes some twisting to get off the pump. You can see in the photo that the reservoir is sitting on top of the case, its rubber-gasketed pump mount visible.

Here you can see the pump has been removed, its slide off base still in place. What I had forgotten was that the pump slides off backward (down) into the case. I had to pull the drive assembly out a few inches to get the pump loose.

The new pump dropped into place easily enough, its the updated model of the old pump, hopefully with this short-out problem resolved. I've had two pumps gone this way now, admittedly both were a couple of years old.

Once the pump is reinstalled, the reservoir is pressed into place, and then the hoses are fitted back on.

Just to complicate matters further, I swapped out the old GEForce4 video card for my more advanced ATI Radeon 9800XT with the double-sided water jacket.

With everything hooked back up, it was time to put the case back together and get things running again. These rackmount cases are awesome, but unfortunately no longer available anywhere. They're true workstation cases - no locking face plate, and there's no rivets or welds anywhere, the entire case is assembled with screws, so that every part can be removed.

Add in the lucky coincidence that the standard Innovatek radiator fits in the case along with a 120mm fan and you have, in my opinion, the best darn rack mount water cooled PC case possible. That's why I have three of 'em.

The old pump sitting outside the case now, you can see I have the pump bypass plugged into the power supply to run with pump without powering up the machine. Distilled water and a little Innovatek water conditioner are added, I discard the old water.

After some time tapping and burping lines to get all the bubbles out, the water loop ran steady, so it was time to power up fully. The machine came to life without consequence, recognized the new video card and everything was good to go.

For the moment the machine is back in its workstation bay, cover off while I check temperatures and keep an eye on things in general. Once you've breached a water loop, its worth keeping an eye on it for awhile to make sure its not leaking or anything stupid is happening.

Meantime, I still have to actually take advantage of the electrical changes and shuffle my UPSes around.

I hinted back before Tech Ed that Phillip, my big gaming machine, had died. Actually, it died only a couple of weeks after I put it together, I just didn't want to talk about it.

One thing I noticed about this machine when I built it was that it did run hot. My other machines can keep their temperatures under 35C with no load, this one with no load struggled with 45C. Add in SETI@Home working the processor at full bore and 45C requires at least 50% fan power. And then there are those darn video cards...

So one day I finally sat down to try out the full potential of the new machine with its lovely, top-of-the-line SLI video cards. I set up Half Life 2 running in 1920x1200 mode. It runs smoothly at around 100fps, in the really gnarly stuff it gets as low as 70... obnoxious, innit?

Enjoying myself immensely, I set off on a campaign of maximum destruction in Half Life 2, enjoying the view, when I feel the heat on my back. I turn around to see the temperature of the water loop hit 75C. Yeop, most of the way to boiling the water. I shut down HL2 to get rid of the load, but the damage was done. Within minutes the machine had died and wasn't coming back. Motherboard baked.

I went and did some math and discovered that each video card ran at about 80 watts. The processor generated only 55W! So when the video cards were working hard, the machine cooked.

Now I had two problems - the first was fixing the machine, which meant a motherboard transplant. This is not normally something I fear, but with watercooling its much more difficult. Especially the water cooling on this machine, with two video cards and a Northbridge chip right between them. The hoses are short and twist all over the place. And the last thing you want to do is breach the water loop.

Here's Phillip sitting on the service desk. Notice I plugged a speaker in to get a listen to any BIOS error beeps. Unfortunately there were none, supporting my belief that the motherboard was cooked.

My first attempt at extraction was to pop both video cards out of their slots. I powered up again at this point in the hopes that perhaps the video cards were dead and now I'd get a missing video card beep pattern - alas, no luck, no noise, no nothing. The machine is still dead. I'd have to peel all the water cooling gear off.

The motherboard extracted. What you didn't see is that I had to unscrew the motherboard from the case and pull it clear of the jacks in the back, then lift the board up with the water gear still on it. The problem was the Northbridge chip, which had a pair of nylon nut-and-bolt sets holding it on. The only way to get those off was to get to the nuts under the board.

The CPU was a bit tricky to remove just because there was so much surface area, but twisting and prying got the block off.

With the motherboard free, I cleaned everything up and transferred the RAM and processor to the new motherboard. In addition I removed the Northbridge fan from the new board (and put it onto the old board so it would be stock again and RMA-able).

Here the new motherboard is all prepped with fresh thermal grease, ready to be installed. And yes, I would remember to clean off the CPU block before I mounted it on the CPU again.

Remounting the water gear on board starts with the Northbridge chip block, since it once again has to be bolted down from the back. The board sits in the case at an angle so that I can get to the back of it, and I slid the block and bolts into the holes, then gently placed the nuts on the bolts until the threads catch. Then its a process of turning each nut a bit so that the block is squarely over the Northbridge chip.

Once the Northbridge is in place, everything else is lifted up so that the board can slide into place. The motherboard is screwed down and then the video cards went into place, allowing the CPU block to be replaced as well. Then the power/reset switch plugs, power/hard drive LED plugs, USB plug for the Matrix Orbital controller, SATA plug for the hard drive, IDE plug for the two DVD drives, and then all the power plugs for the motherboard (there are three: main plug, secondary 12 volt and a molex for the video) plus the additional power plugs for the video cards.

A quick top-up of water into the reservoir and I was ready to power up again for the first time in more than a month.

And the beast lives! If you look close, the screen is stopped on a BIOS error because there's no CPU fan. Which is a reasonable error since there is no CPU fan. Some quick BIOS tweaks took care of that.

So, remember when I said there was two problems? The first one is now resolved - the machine is back to life with a motherboard transplant. Problem number two is how to avoid cooking the motherboard again. Within minutes of powering up, running no high load software (like SETI@Home), the machine is already at 44C. Add SETI@Home and the temperature immediately rises a couple of degrees, causing the fan controller to turn up the fan to cool it back down again.

Fire up Half Life 2... well, I wasn't going to do that again.

I found the answer at Sprite - the guys I get most of my gear from. For whatever reason, they happened to have an Innovatek RADI-Dual in stock. I don't know why, they'd never sell the thing... well, okay, maybe not never.

This radiator is twice the size of the ones that I use in the case, and has mountings for two 120mm fans. It wouldn't fit in the case, but it would offer a whole bunch more cooling. Would it be enough? With it immediately available, it was too easy not to try it.

I mounted a pair of ultra-quiet Vantec 120mm fans, directly powered... I've burned up a couple of these lovely fans with controllers, so I didn't want to take the chance. And besides, even at full power these fans only generate 28dB of noise, so you can't hear 'em at all.

To connect the radiator into the loop I disconnected the top-side connector of the existing radiator and moved it to the bottom feed on the new radiator, then added a new hose from the top connector of the new radiator down to the old radiator. Powered up and started adding water to the pump reservoir as fast as I could to fill that new radiator. Several ounces later, everything was full and ticking along.

Its a little on the Mad Max side of things, but it sure does work!

Check out the front view of the machine, you can see the temperature of the water - just below 31C!

When I fired up SETI@Home, the temperature didn't move at all. So then the real test: play some Half Life 2. After one hour of play, the water temperature got to 32C. Methinks the fix is in!

Obviously, the system can't stay like this. But I'm afraid the only real answer to this problem is going to be much more radical: converting to central water cooling. That would involve putting a set of pumps, radiators and reservoir inside the server closet and running hoses through the walls to the two workstation bays in the office. The same way that you have a wall plate for power and a wall plate for network access, there would be a wall plate with water input and output. Then you'd just plug the machines in.

There are a bunch of advantages to the central water cooling solution. The first is that there will be a lot more water, and that water will be chilled. So the ability to cool will increase substantially. The machines will be even quieter having no fans (except the whisper fans in the power supply), no pumps and no radiators. Another huge bonus will be that the heat of the machines will actually be taken out of the room, being dumped into the server closet with its great big AC unit.

The downside is that the machines are no longer self-contained for cooling. When I have to service them, I'd need to use an external water cooling module, something like the CoolerMaster Aquagate or the Koolance Exos 2. All resolvable stuff.

So, for the moment, everything seems to be functioning here in water cooling land. I'm watching Phillip closely for any water leaks, I'm a bit concerned that the heat event might have damaged some seals. But so far, so good.

Yep - that happy time again. I got some new parts in and decided it was time to rebuild one of my water cooled machines. And, since I was gonna dig into one, I figured it was a good time to do check ups on the others.

First up was my wife's system. Its a P4 with a 533Mhz FSB, rock stable and the first machine I ever water cooled.

Doesn't the water look nasty? This is what happens when you let the water get too low - it cooks. A quick top off with distilled water fixed it up fine. It needs topping every three months or so. If you look really close at the pump, you'll see a white stain - this is a bit of leakage, caused by overheating. Note that at no time did the machine ever fail: like I said, rock solid.

After the refill, I ran the system for awhile and then used my handy-dandy Raytek infrared thermometer to check the processor block temperature. Look closely at the processor water block, you'll see a little red dot beside the word X-Flow, which marks the spot where the thermometer is reading the temperature. 101.5F is a good operating temp.

Next up is the development workstation, the machine with three monitors attached.

This machine, like the last, has a nice, tidy water cooling rig. The video card in question is a Matrox Parhelia. Notice there are three SATA cables? Two for my RAID 1 drive array and one for my Plextor PX-716SA SATA DVD-RW! This machine was a little low on water, not as bad as the other, but still needed to be topped up.

The gaming system had been running for awhile with air cooling while I awaited the arrival of several parts: a water cooling block for the Athlon 64, as well as new nVidia 6800 Ultra video cards and their cooling blocks. Everything showed up during the week, this was my first chance to put it all together.

I had to strip the motherboard out of the machine to replace fans with water blocks - both the processor and Northbridge chip had mounts that had to be accessed from the back of the mother board.

Here's a look at the motherboard with fans installed. That little Northbridge fan is particularily squealy noisy. On the right is the little Northbridge cooling block (which I had stashed away) plus the new Socket 939 water cooling block and mount.

Here's a look with the fans stripped off and the processor and Northbridge cleaned and ready for water block mounting.

The processor mounted up with no problems at all - screw in the new support frame, put a fresh coat of thermal paste on the chip, polish up the block, place it on top of the chip and snap the locks in place. Alas, the Northbridge chip wasn't so easy.

The first problem was the alignment on the mounting holes. The water block sits right between the two video cards, so the hose connectors have to face directly toward the front of the motherboard. The way the water block went together, there was no way to mount it that way. I had to disassemble the block to flip the mounting plate over.

In the shot you can see the bits of the water block, from the clamping nuts, to the actual hose connectors, the block itself and the mounting bracket. I didn't pull the copper base out of the block, there was no reason to (except to show you), and I risked damaging a water seal.

After flipping the mounting bracket, I discovered that the mounting posts that crappy little fan used were too short for the water block. Off to Home Depot for some nylon nuts and bolts. A quick rub-down with thermal paste, a bit of fiddling and the Northbridge block got mounted.

You can see the hose connectors are facing forward, if not exactly square to the front of the motherboard. I was willing to favor the first video card, since it wasn't quite as close to the Northbridge water block. Notice also in this shot the EIGHT SATA connectors on this ASUS A8N-SLI Deluxe motherboard. Of which I'm using only one.

With the motherboard square away it was time to put my sights on retrofitting the video cards, a pair of ASUS Extreme N6800 Ultras.

This is the card, still fully intact, with the Innovatek water block sitting beside it, along with back plate and mounting hardware. Notice on the water block there's the center plate for cooling the GPU, four contact points for RAM and the left-most edge screws down over the voltage regulators. All those holes in the block get filled with various kinds of screws.

The video card stripped, ready for cleaning and water block mounting. Note the four screws from the GPU block, five screws from the RAM block, three screws for the fan and two plastic posts for the voltage regulators. Did I mention this video card is much, much lighter with all that crap removed?

After cleaning the chips off, applying new thermal paste and assembling all the bits very carefully, you can see the back plate with its four screws, the five spring loaded screws for the RAM mounts and two screws holding down the block on the voltage regulator. The video card is all heavy again.

Did I mention there was two of them? SLI video, doncha know.

With all the water block appropriately installed, it was time to reassemble the machine.

This is the assembled version. The blue cables are power cords, the video cards take two molex connectors each, plus there's another molex plugged into the motherboard, in addition to its normal main and secondary power plugs. The red/black wires are temperature sensors (four), black/red/yellow are fan connectors, of which there is two - one for the radiator fan, the other is plugged into a water speed meter.

All those sensors and fans are connected to a Matrix Orbital display, which is wired to the system via USB, which is the silver braided cable (you have to look real close for that one). Oh, and the red braid cable running over the top is the single SATA drive.

The water plumbing is as follows:

• Radiator
• Reservoir
• Pump
• CPU
• Video Card 1
• Video Card 2
• Northbridge
• Hard drive
• Water meter
• Back to radiator

One temperature sensor is inside the case, the others are in the water loop, between the pump and CPU, Northbridge and hard drive, meter and radiator.

The controller is set up to vary the radiator fan speed automatically based on water temperature. While I'm still playing with the tuning, right now its set to keep the system at 45C (113F). At 44C or below it'll slow the fan down to 25% of maximum speed. Above 46C it'll increase the speed of the fan to 100% to bring the temperature down.

And if you're wondering why I'm posting this on a Monday... well, it took longer than planned to get everything finished. As usual.

When last we left the Water Cooling War, I was battling unreliability of my 800Mhz FSB machines. All along I thought it was the RAM overheating, although I was thinking the heat spreaders I added would do the trick. For one of the systems (the one from my last blog entry on water cooling) has been pretty darn stable. But the other machine (the triple screen beastie) was still getting BSODs every few days.

The interesting thing about these failures is that the hard drives would disappear afterward. It actually has two Samsung 160GB SATA drives in it, which I had planned to use as mirrored drives with the onboard RAID 1 Promise controller. After all, this is my main development machine, its worth while making sure the data doesn't go anywhere. But one of the drives had disappeared, I figured it had failed, so I was running the other drive solo.

And with the BSODs, both hard drives would be gone. I'd wait a half hour or so (I do have other machines to work from after all) and the drive would come back. I knew I'd have to replace the hard drives eventually, they have a three year warranty, but extricating water cooled hard drives isn't a lot of fun.

Well, it all came to a head last night when the drive just wouldn't come back. So I hauled the machine out and ripped both hard drives out... very carefully, so that I wouldn't cause any leaks.

As you can see in the photo, the hard drives have blocks on either side, connected by a plastic tube. This plastic tube is just press fit into place to handle the variations in width between hard drives, pull too hard and it would pop off and water will go everywhere.

I replaced both drives with identical models, put the whole thing back together and viola, two hard drives, mirrored and happy. I fired up Acronis to restore the image backup I have of my workstation (the easiest way to recover a system - you DO have a backup strategy, doncha?) and left the machine whirring away til morning.

When I returned in the morning, the machine had failed, both hard drives disappeared. Guess it wasn't the drives failing after all.

So, what could it be? The onboard controller? These ASUS P4C800-E motherboards have TWO different SATA controllers on them, could they both be bad? I think not. So what would knock out both drives?

Gosh, lookie there... the SATA power adapter plugs a 4-pin molex connector and provides connectors for BOTH SATA drives. Could there be a connection problem?

I replaced the adapter with a new one, and fired the system up - both drives recognized, no problem. I have to guess that once the system heated up, the connectors got loose and deprived the drives of power. This, naturally, would cause Windows to BSOD... and then there'd be no drives left.

So in a way, overheating is still the culprit, but I suspect that water cooling is not responsible for this. Of course, I won't know for sure til its been running for a few months.

Well, one night stretched into four, as usual...

After my main workstation freaked out and stuff stopped working reliably, I pulled the motherboard and had it replaced under warranty. It meant peeling all the water cooling off very carefully to extract the board without making a mess, but I got it done.

Took a couple of days to get the motherboard actually back home. Sprite got it in late Thursday night, but I didn't actually swap it til Saturday morning. Then there's always the fiddly bits of getting it mounted again, and getting the water cooling blocks cleaned up and re-installed.

Along the way I had blown the drives off, thinking that maybe the network and raid problems might be fixed with a clean install, and I'd always wanted to re-install that system in the first place, when I had done the initial installation a few months back, somehow I'd left the Hyperthreading off, which actually installs a different kernel in XP. And I never found a way to fix it, everywhere I looked they said “re-install from scratch.”

I like a scratch re-install - it feels fresh. So off I went on my happy re-install path. Unfortunately, the built-in NIC is a late generation Intel gigabit jobbie, so the XP disk doesn't recognize it, so all the network stuff has to be done post-installation after I can get the driver loaded.

Once I got the point of getting the network stuff set up, I couldn't register with my domain - the error message said the user or password was wrong. I went around in circles for a couple of hours on this one, interspersed with a little stress relieving Doom 3 when I couldn't figure out the problem (only ID has really figured out the therapeutic value of shotgun blasts).

I could surf the Internet from the machine, but I couldn't activate and I couldn't register with the domain. And the error messages were hopeless. Then I noticed the date was set to January 1, 2002. So I opened up a command window and used net time /set to grab the time from my domain servers, and everything was fine - activated and domain registered. I think kerberos is a fine idea, I just wish they'd give some hints in the error messages, “bad user name or password” doesn't cut it.

Oh, and I threw on Windows XP Service Pack 2, of course. Then I loaded a couple of SETI@Home work units to run and went to bed. In the morning, the screen was covered in white bars and I had no user interface... although SETI@Home was still doing its thing. I used my handheld infrared thermometer to check the temperature of the Matrox Parhelia... the water loop was showing 110F (remember, this is running under full load all night), the water block on the video card was at 120F, the uncooled RAM chips were at 160F. A wee bit too hot.

So I happened to have some Tweakmonster BGA RAM sinks, mix in a little Arctic Silver Thermal Adhesive, and I'm cooling video RAM.

Notice I've STILL managed to avoid breaching the water loop!

Fresh off the victory of my water cooled secondary workstation, I decided to finish off my primary workstation. The primary is the triple-screened machine running a Matrox Parhelia video card. I do my development work across its lovely 3840x1024 set of displays.

I had a couple of minor things outstanding on it to do - adding in a monitoring display for the front of the case that shows temperatures and fan speed. Of course, in this situation, I'm monitoring the temperature inside the case and in the water loop using a water loop sensor. And the fan I'm monitoring is actually a water loop impeller.

Harmless revisions, right? So off I go, fitting the bits into the case, breaching the water loop carefully and inserting the new components, and everything goes fine.

Until the motherboard (an ASUS P4C800-E) freaks out and stops working. Well, just the network adapter, and the RAID controller, and... well, I don't want the motherboard any more. Its all under warranty, but the price of building your own machines from parts is that you have to fix 'em too. So I called up my buddy Mike Neilsen at Sprite Computers (Mike supplies most of my components) and he ordered up a replacement motherboard for me.

Didn't take me long to pry the motherboard out of the case, and I managed to do it without breaching the water loop!

So I guess tomorrow I'm rebuilding my main workstation... a night without my beloved triple screen display! Horrors!

There's a certain contingent of folks (and you know who you are) who are living their water cooling dreams vicariously through me, and they’ll all be happy to know that I put aside some time to try and clean up my water cooling problems and finish some machines.

When last we left my intrepid bout of fiscal and productivity irresponsibility, I had hit a wall with reliability regarding my 800Mhz FSB machines. These are the high performance units that get damn hot. The problem, near as I could figure, was in the RAM. The two machines I set up for water cooling would periodically hang, and careful study of the occasional Blue Screen of Death seemed to indicate RAM problems.

The first time I got a hint that the RAM might be trouble; I stuck my finger on one of them and left the fingerprint behind. Yeah, they were hot all right. My theory (and I mentioned it on DotNetRocks some time ago) is that with the reduced airflow in the case caused by removing all those fans and replacing them with water cooling blocks, the RAM no longer gets enough air circulation.

My first attempt at a solution is to put heat spreaders on the RAM. These are copper plates at strap to either side of the RAM. Presumably they reduce the amount of heat in the RAM itself, giving the heat more area to wander to. So far I’ve notice that the heat spreaders are just as bloody hot as the RAM. I’m going to try adding a low velocity slot fan in the back of the case to try and draw more air through, seeing if that will help.

Of course, the wonder of periodic failures is that you never know if you got it right or not. You’re waiting to see if a presence is absent… in this case, the fact that I don’t get a Blue Screen of Death for six months will be my only proof that I was right.

Also, the crazy water cooling machine, the one with the Orbital Matrix controller in it to vary fan speed by temperature, had a leaky pump. I slathered the pump with silicon, but it leaked again. Then it stopped leaking. Then it started again. Then it stopped. Then I realized I was fighting with a hundred dollar part on a two thousand dollar machine and bought a new pump. So that had to go in.

And finally, since I had been noodling with this bloody machine for so long, a new cooling plate had come out for the ATI 9800 XT Pro I had, so I had to buy that too.

So, here’s a detailed record of the entire process:

1. Shut down machine.
2. Remove the cap from the reservoir.
3. Disconnect the highest point in the water loop, which for this machine is the top plug of the radiator, being careful to make sure the water line is drained (which is why you remove the cap from the reservoir, so some air can get in).
4. Unplug the ATX power connector from the motherboard.
5. Plug the bypass plug into the ATX power connector.

6. Squirt water around the room as the power comes on for the pump.
7. Switch off the power supply so the pump turns off.
8. Get water catcher (aka – the yogurt container), position so that removed hose is aimed into container.
9. Switch power supply on again.
10. Stare in confusion as the pump does not turn on.
11. Fiddle with bypass plug, switch position, try to find combination that works.
12. Go play Unreal Tournament 2004 for a half hour because blowing up digital stuff is less expensive than beating this stupid machine with a sledgehammer.
13. Return post-destruction to discover that no combination of power switch and plug will make the pump turn on.
14. Test with another power supply (What? You don’t have a spare power supply? What kind of geek are you?)
15. Pump still won’t start. Unplug pump.
16. System powers up without pump using either power supply.
17. Test power supply with replacement pump, the replacement pump works fine.
18. Devise a bulb pump to remove the water from the system without the pump.
19. Put cap back on reservoir before water comes flying out from the pressure of the bulb pump.
20. Drain majority of water from the system using bulb pump.

21. Stand machine up on its side, drain balance of water.
22. Remove side of the case in order to extricate pump.
23. Disconnect hoses, pull off reservoir, remove pump.

24. Mop up additional water spills from pump removal.
25. Pull video card for additional cooling block installation.

26. Remove existing water connect from A side cooling block already mounted to video card.
27. Add water interconnect to A side cooling block to allow B side to plug in.

28. Remove existing nylon screws holding A side cooling block on.
29. Put conductive goop on the backside RAM chips of the video card.
30. Place B side cooling block onto card, pressing interconnect into place.
31. Insert nylon screws through B side block, card, and into A side block.

32. Discover that video card will no longer go back into the AGP slot; the B side cooling block hits the hoses coming off the Northbridge chip.
33. Play some more Unreal Tournament 2004.
34. Install pump.
35. Install reservoir on pump.
36. Re-install hoses onto pump.
37. Remove the water block from the Northbridge chip.
38. Rotate Northbridge water block 180 degrees.
39. Reinstall Northbridge water block.
40. Install video card.
41. Reroute plumbing to deal with rotated Northbridge and moved connector on video card.
42. Replace pressure fit water flow sensor with screw down type.

See? Easy. Just follow this simple 42 step process!

Man, does the inside of that machine look like R2D2 barfed or what?

On June 24th I was a guest on DotNetRocks... but we didn't talk about .NET, we talked about my favorite subject, TOYS! Actually, the focus was on water-cooled computers, which is definitely toy-ish, although we digressed into a number of equally entertaining topics.

There were a variety of questions, so I figured I'd best answer them here. First off, I put together a little photo-pictorial of one of my water cooling conversions.

One of the gizmos I used in that photo-pictorial but didn't take a photo of is this little motherboard power adapter that I plug into my power supply so that I can fire it up without having to actual turn the machine on. It's very useful for being able to run the pump without heating anything delicate up.

Its just a female 20 pin ATX plug that connects pins 13 (ground) and 14 (power supply on) together. So there ya go Geoff, don't say I never did nuthin fer ya.

If you looked at the water cooling page above, you may have noticed I'm using rackmount cases for my workstations. I have a server closet that's all rackmounted, but I also had my desk custom built with rackmount bays as well.

This is one of the bays being fitted out while the office was still under construction. The rack itself is a 12U Middle Atlantic SRSR Rotating Sliding Rail System rack. This rack actually slides out of the bay and then rotates once fully extended so you can get at the back of the case without digging around blind. I have two of these in the office, one for each main workstation bay. There's enough room on the rack for a UPS, two PCs and other sundry gear.

Although we didn't talk a whole lot about it on the show, for rackmount junkies, here are a couple of links to my server rack set ups. The first link is to my old rack, which ran from September 2000 to December 2002. After that, my new rack server closet was up and running, which is how it continues to this day.

This shot was taken today... the rack is essential the same as it was December 19, 2002, except that it's a whole bunch messier. Over the summer I'll be rebuilding most of the systems in here, after all, some of the hard drives are now four years old an essentially ticking time bombs well past their MTBF (Mean Time Before Failure).

You may recall I had a little water problem with my PC... well, I managed to find and fix the leak.

The challenge of fixing the leak was finding it. I slid some paper underneath the machine to see where the water dripped out (after removing the covers from the machine) and left it run for awhile. That led me to the pump. But even knowing the water was somehow coming from the pump, I couldn't actually see the leak. So the pump had to come out.

Now, removing the pump means breaching the water loop (technically, its already breached with the leak, but still). So now I had to figure out how to get the water out of the system without getting it all over the office. The trick is to find the highest point of the water loop, where the water naturally drains out when its not in use. Or create a high point by lifting a hose as high as possible until its full of air. Hopefully, this is on a nice, long hose that you can unplug, lift out and stick into a bucket (or in my case, a large, empty yogurt container). Now you need to fire up the pump again, preferrably without burning up your system.

I have this little plug that I stick onto the main power supply connector that does two things - it makes sure that the motherboard is unplugged so the board won't power up, and it lies to the power supply so that it will turn on while not being plugged into the motherboard. The result is that the pump fires up (and the hard drive, and anything else plugged into the molex connectors).

This is the moment where you realize whether or not you unplugged the right end of the hose - either water is going to pump out into bucket, or shoot all over the case (ask me how I know). I took a shot of the pump extracted from the case, you can see the four bolts that hold the pump in on their rubber bushings.

Now that the pump was extracted, I made a little closed loop solution, hooking a hose from the output to the input of the pump. Poured a bit of water back into the pump and fired it up.

You may notice the water looks rather white and foamy - it is. The pump is under so little pressure, the water is just ripping around the loop and swirling in the reservoir. Good thing I didn't fill the reservoir all the way up, it would have shot it all over the place like an overflowing blender.

After a few seconds, I could see drips coming out of the pump housing, right beneath the reservoir mount. Turns out I actually had two leaks. The output mount sticking out of the top of the pump (which already has silicon on it) was still leaking, and there was a crack in the pump housing around the pump pickup from the reservoir. It took several tries and lots of silicon to actual get all the leaks plugged. Pulling the pump was definitely the right solution - I would have liked to have fixed it in place, but this was the only way to get it right.

Once the pump had run over night without a leak, I drained it, put it back into the machine and refitted all the hoses. After refilling the system, running it briefly to burp air from the lines and topping it up, I left it go for the night with the paper in place to find more leaks.

And this morning it has a clean bill of health. Nothing on the paper, water temperature holding steady at 38C.

Not a good hardware day.

I like water-cooled computers. Why? Because they're quiet. No 5000rpm high velocity fans on the processor, video card, etc, etc. Just a couple of slow, silent fans on the radiator and the power supply. The way it oughta be.

Putting water cooling into a computer system isn't a trivial task, but its not rocket science either. I've built three of them so far, and noise level in my office is better for it.

Unfortunately, now I have to deal with a new kind of problem... leaks.

See if you can see where this machine is leaking from.

See it? Me neither.