I finally finished piecing together and tweaking my three new mining rigs.
Here's to hoping they can fund my FPGA Mining research!
I will disclose my adventures and learnings here, for the benefit of fellow Bitcoiners.
Original Setup (per rig)
* Video Cards: 3x Sapphire 5850
* Motherboard: Assrock 890GX Extreme4
* CPU: AMD Sempron 140
* Memory: 1GB Kingston DDR3 1066 (PC3 8500)
* PSU: Rosewill RBR1000-M 1000W
* 16GB flash drive
* A mystery case ...
SoftwareUbuntu 11.04 64-bit - Catalyst 11.4, SDK 2.4
Phoenix 1.4 with phatk kernel
StatsOverclocking: 800/285
Hashing: 313MHash/s per card, 939MHash/s total
Temps: 68C, 66C, and 61C
Things That Went Well 
---The motherboard is quite nice for its price. On-board power, reset, and Clear CMOS buttons. On-board GPU. Three slots for graphics cards, and an extra PCI-e 1x for future expansion.
---NewEgg's shipping
---The MtGox price jumped to ~$9 as my parts were arriving. It's back down to $7 now, but that's still better than the numbers I did my cost analysis on initially.
Things That Went Okay :/
---The CPU and 1GB of RAM runs fine, but installing software through the GUI can sometimes throw stuff into cache and make stuff a bit sluggish. However, once you have everything installed, mining works just fine.
---The case I got is nice. 140mm on top, 120mm in front and back, and 2x120mm on the side over the GPUs. Pretty cheap too.
Things That Were Difficult 
---I wanted to use the front USB plugs on the case, and so plugged that cable in. One loud screeching noise from the video cards later I realized the USB plugs are too close to the bottom video card
This caused the cable to hit the fan. No damage done, but I have to use the plugs in the back.
---Initially the middle card was running very hot. >80C, sometimes touching 90C. I built my rigs one by one, so I could test them out and apply updates to the design on the next two. So for the second rig I shoved the PSU in the CD/DVD drive bays. I had to bend the metal a bit, but it fit.
For the second rig I also used some bits of plastic to push the video cards apart. On this motherboard they are right next to each other with very little room. That seemed to help a little; allows more air to get to the GPU fans.
And finally I also experimented with memory underclocking. From stock 1000MHz down to 285MHz. Runs stable, and that final tweak seems to be what got me my final numbers. 68C, 66C, and 61C.
---Wiring was a huge pain in the butt. I have a bunch of velcro strips that I use to tie wiring together, and those were
great. I keep all the paths for airflow wide open, but everything else is jammed full of wires.
---The biggest wild-card in my entire setup was the PSU. It had great user reviews, it was cheap, and had plenty of power. Only problem was, it had four separate 12V rails. I figured, if I was careful, I could use one of each GPU, and one for the motherboard and fans.
Long story short, it worked out just fine and the machines are running fine. But it was nerve-racking worrying about whether the rails could handle everything. The motherboard was especially worrying, because it supplies up to 75W through the PCI-e ports. 75 * 3 = 225W, or 18.75A. On a 20A rail ...
Also, the PSU had modular cables, but because I was so worried about power balancing, I put the case fans on the two 30A rails. This required attaching two of the PSU cables, and added clutter. What a mess!
---Another thing that ended up being difficult was my purchase from TigerDirect. Apparently they don't like money and decided I had to wait all weekend for them to verify my CC. This is despite me ordering from them a few times before. They called me, but when I called back they were closed. Then when I called Monday they said everything was fine ... so, I appreciate the waste of my time. Also, my items got back-ordered. Didn't get here until Friday (ordered the weekend before, 2day shipping).
NewEgg, on the other hand, had that stuff out and in my hands the second I pushed "Confirm." Well ... not
that fast, but I've always loved NewEgg's speed. Also, they don't seem to sell back-ordered items like TigerDirect.
Things That Went Poorly --- I initially chose to use 16GB flash drives as the hard drives for all these machines. I figured they would use less power, generate less heat, and not use any space inside the case (better airflow). I mean, Live Linux CDs run just fine off flash drives ... right? ... Well, after building the first rig, it felt like I had taken a step back in time to the 90s. With only 1GB of RAM, stuff was getting thrown into cache a lot. Guess where cache is? Yup, it's on the flash drive.
It didn't make a difference to mining, but it made installation take hours. It also meant longer boot-up times, which would exaggerate any downtime.
So I grabbed 3 of my old SATA drives and used those instead. Works great!
--- The on-board GPU worked fine for the first mining rig. The other two, however, refuse to use it. They can't even see it. It's not a big deal, but it's nice to have in case the cards lock-up and to keep GUI load off them. Still not sure why this is ... (the option to use on-board doesn't even show up in the BIOS).
Things I Would Change 
---In retrospect I would have dropped the case and bought a more expensive, single-rail, PSU. I wanted the case, because it was so cheap (like $50), and would provide easy mounting for fans and good airflow control. Well, I'll bet I could have bought a big, cheap shelf from the hardware store; mounted all the machines to it, and twisty tied all the fans around it. That would also have made wiring up the equipment easier.
Without a case, I would also have gotten a different motherboard with 4 slot capability. I didn't do that for this setup, because the 4th slot was at the very edge of the motherboard. That wouldn't fit the two-slot 5850s (fan on second slot). Without a case, though, that wouldn't have been an issue. Regardless, with a PCI-e 1x adapter I can get a 4th card on this motherboard.
Other Factors---The first night of running the machines, they were all in my living room with the windows shut. I woke up to an oven, and decided to fry some eggs on the cases. While it was neat to cook breakfast on my mining rigs, the clean up was a tad messy, and ultimately I wanted the apartment to be less than sweltering. Also, I used the last of my green peppers so I need to buy some more.
I ended up moving the machines next to a window. A big appliance fan will be purchased later to push all the heat out the window. Like a Pro.
---The last ordeal of my story is something I had completely forgotten to take into account. Circuit breakers. Like most typical homes, my apartment has several circuits for all the 120V outlets, each circuit with a 20A or 15A fuse. These fuses blow if the instantaneous current exceeds 20A, or if the average current exceeds ~17A. That's
about 1800W. The power strips I use are also rated for 1800W. If each rig draws 1000W at max, that's well over what my apartment can handle on a single circuit.
The 5850s draw ~180W each, which is 540W total. +100 for the rest of the computer and you have 640W. That's 1920W for all three machines.
I spent thirty minutes mapping out my entire apartment's circuit-to-outlet chart. With many thanks to the engineer who put this place together, the majority of the apartment's outlets are on a single circuit. Yay! So much fun! The rest are sporadic, with no logic to their placement. Also, I have no idea where two of the circuits are. They aren't hooked up to any outlets or lights ...
For now, I've got all 3 rigs running on one of the oddly placed circuits, which has nothing else on it. So far so good. I have one more nearby empty circuit I can use if things go awry.
It's all a huge pain. Hopefully something comes of my FPGA tinkering and I can replace these rigs with clean and efficient FPGA mining.
Closing ThoughtsWorth it? Yes! Mostly because it was like Christmas Day getting all those shipments, tearing open the boxes, and piecing everything together. It's also a total nerd fest, and warms my geeky heart to have a mini-datacenter churning away
Will it make me rich? Probably not. I'll be happy to break even and walk away with the experience. If it does churn out some cash, I'll pump that into my FPGA mining research.
