So Patch Tuesday came by as usual, and machines were patched. I have one workstation, Terrance, that still isn't running Vista. Terrance is the giant triple screen machine, running a total of 4960x1600 worth of displays. Plugged into it is the MOTU Traveller that I do all my recording with for RunAs Radio and .NET Rocks. So needless to say, it needs to be completely reliable, since I record every week.
So, as I said, Patch Tuesday came by as usual. And as usual, Terrance didn't auto-install the patches, just downloaded 'em and let me know. And, as usual, I hit the button. And, as usual, the patches required a reboot. What could go wrong?
Terrance didn't come back.
Terrance sits at the bottom of the stack of two workstations, sitting underneath Phillip. Phillip is a far crankier computer, struggling with barely sufficient cooling. So I'm usually tinkering with Phillip to keep him happy, while Terrance is totally reliable.
Except that Terrance didn't come back.
I left Terrance off for a couple of days. I had already finished recording for the week, so I had until the following Tuesday to get things fixed up, and I figured I could wait for the weekend.
When the weekend came, I hauled Terrance out of the rack, which means shutting down and removing Phillip as well. Annoying. Moved Terrance up to the service counter and re-installed Phillip so at least I had one workstation up and running.
With the cover off and plugged into the test harness, Terrance still wouldn't power up. Well, wouldn't power up is a bit of an exaggeration... the motherboard power light is on, but the main power light wouldn't turn on, and it seemed like the power switch was useless. Sometimes I'd see the LEDs on the RAM chips sequence a bit, but the drives never spun up.
I suspected that my groovy new water cooled power supply might be the culprit. Fortunately, I have a power supply tester, so I plugged it in and powered it up.
Here's a look inside Terrance. You can see the dual video card set up (running a pair of nVidia 7800s) that are both water cooled, along with the CPU, northbridge, southbridge, hard drive and power supply. This is also the moment of truth, with the power supply tester on the left plugged into the water cooled power supply. The power supply is the blue thing on the lower right of the photo, the black block attached to it is the water cooled part. The heat sinks for the power supply are connected to the block and water passes through the block to cool it. No fan in the power supply.
So, turn on power supply, and the power supply tester should report voltages, all that good stuff.
Only it doesn't do anything.
Being the suspicious type, I pull out my spare power supply, a nice Enermax Whisper unit, not water cooled, but nice and quiet. Plug power supply tester into that, fire it up, and everything lights up. Ah ha, one dead power supply.
So, to replace the power supply, I need to breach the water loop. I hate breaching the water loop, its messy. But, Terrance is one of my external water loop equipped machines. That means I have a pair of hoses running out the back of the machine that can be connected together to be self-contained, or connected to a larger external cooling system. Ultimately my plan is to plug into the wall for water cooling, but goodness knows when I'm going to have time to finish that.
But, back to the problem at hand - I have hoses with self-sealing couplings running out the back. That makes draining the water system a whole lot simpler.
In my hand I have a bulb pump which is connected to one side of the external loop. The other side has an unsealed coupler connected to it and is stuffed into a clean yogurt container to catch all the water. 20-30 squeezes of the bulb pump later, all the water is drained out of the system into the yogurt container.
Now to actually breach the loop - doing the pump out isn't really a breach anymore because I use those self-sealing couplings to keep everything tidy. The trick to removing things from a water cooled machine is to realize they still have some water in them, so closing off the water connections is a good idea.
On Terrance, the power supply was added to the water loop between the southbridge block and the hard drive block. So I want to connect the southbridge hose to the hard drive and leave the power supply out. But to keep the power supply from leaking, I connected the existing hose running between the power supply and the hard drive to the other end of the power supply.
So you can see the hose loop on the two water cooling connectors of the power supply, and in my hand is the hose that used to run to the power supply from the southbridge chip. I just had to rotate that hose around and connect it back to the hard drive and everything was sealed up again, no mess.
So the rest was easy - extract the power supply, install the Enermax supply, reconnect the power supply and I'm off, right!
Here's a shot of the freshly wired in Enermax supply. Looks good, huh?
Except for the part where it doesn't power up at all. Motherboard light turns on, but the machine won't power up. I'm suspicious. So I retest the old water cooled power supply. Its still dead. But something else is wrong.
Experience has shown me that the one thing that can stop a water cooled computer in its tracks is a bad pump. So I unplug the pump and the machine powers up normally. I have a bad pump as well. Good news, I have a spare pump.
For whatever reason, I failed to photograph the pump installation. Its very tricky, I had to turn Terrance on his side, remove the side plate (five screws off the bottom, five screws off the side, four screws off each end), disconnect the hoses from the reservoir, wiggle the reservoir off (its press fit), slide the pump out, slide in a new pump, remount the reservoir, reconnect the hoses, put the side plate back on, reinstall all the screws. Apparently I was so busy I never snapped a photo the whole time.
This is a shot immediately after finishing the pump swap-out. The dead pump is out of the machine, the new pump is in the machine. Everything else is the same, and Terrance now works.
Refill of water was pretty simple, just power up and keep pouring water. Once the lines had been burped (leaving it running for a half hour or so with the reservoir lid off), I buttoned everything up and put Terrance back in place.
Total service time, about two hours.
So, questions: did the pump kill the power supply, or the power supply kill the pump? And what killed either one or both?
And I need to restock on spare parts.