Bitcoin Forum
December 08, 2016, 08:13:59 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 »  All
  Print  
Author Topic: What I Learned from Building 3x5850 Rigs; May 2011  (Read 8177 times)
fpgaminer
Hero Member
*****
Offline Offline

Activity: 546



View Profile WWW
May 16, 2011, 03:11:57 AM
 #1

I finally finished piecing together and tweaking my three new mining rigs.  Cool Here's to hoping they can fund my FPGA Mining research!

I will disclose my adventures and learnings here, for the benefit of fellow Bitcoiners.

Original Setup (per rig)
* Video Cards: 3x Sapphire 5850
* Motherboard: Assrock 890GX Extreme4
* CPU: AMD Sempron 140
* Memory: 1GB Kingston DDR3 1066 (PC3 8500)
* PSU: Rosewill RBR1000-M 1000W
* 16GB flash drive
* A mystery case ...

Software
Ubuntu 11.04 64-bit - Catalyst 11.4, SDK 2.4
Phoenix 1.4 with phatk kernel

Stats
Overclocking: 800/285
Hashing: 313MHash/s per card, 939MHash/s total
Temps: 68C, 66C, and 61C

Things That Went Well Smiley
---The motherboard is quite nice for its price. On-board power, reset, and Clear CMOS buttons. On-board GPU. Three slots for graphics cards, and an extra PCI-e 1x for future expansion.

---NewEgg's shipping  Grin

---The MtGox price jumped to ~$9 as my parts were arriving. It's back down to $7 now, but that's still better than the numbers I did my cost analysis on initially.


Things That Went Okay :/
---The CPU and 1GB of RAM runs fine, but installing software through the GUI can sometimes throw stuff into cache and make stuff a bit sluggish. However, once you have everything installed, mining works just fine.

---The case I got is nice. 140mm on top, 120mm in front and back, and 2x120mm on the side over the GPUs. Pretty cheap too.


Things That Were Difficult  Huh
---I wanted to use the front USB plugs on the case, and so plugged that cable in. One loud screeching noise from the video cards later I realized the USB plugs are too close to the bottom video card Sad This caused the cable to hit the fan. No damage done, but I have to use the plugs in the back.

---Initially the middle card was running very hot. >80C, sometimes touching 90C. I built my rigs one by one, so I could test them out and apply updates to the design on the next two. So for the second rig I shoved the PSU in the CD/DVD drive bays. I had to bend the metal a bit, but it fit.

For the second rig I also used some bits of plastic to push the video cards apart. On this motherboard they are right next to each other with very little room. That seemed to help a little; allows more air to get to the GPU fans.

And finally I also experimented with memory underclocking. From stock 1000MHz down to 285MHz. Runs stable, and that final tweak seems to be what got me my final numbers. 68C, 66C, and 61C.

---Wiring was a huge pain in the butt. I have a bunch of velcro strips that I use to tie wiring together, and those were great. I keep all the paths for airflow wide open, but everything else is jammed full of wires.

---The biggest wild-card in my entire setup was the PSU. It had great user reviews, it was cheap, and had plenty of power. Only problem was, it had four separate 12V rails. I figured, if I was careful, I could use one of each GPU, and one for the motherboard and fans.

Long story short, it worked out just fine and the machines are running fine. But it was nerve-racking worrying about whether the rails could handle everything. The motherboard was especially worrying, because it supplies up to 75W through the PCI-e ports. 75 * 3 = 225W, or 18.75A. On a 20A rail ...

Also, the PSU had modular cables, but because I was so worried about power balancing, I put the case fans on the two 30A rails. This required attaching two of the PSU cables, and added clutter. What a mess!

---Another thing that ended up being difficult was my purchase from TigerDirect. Apparently they don't like money and decided I had to wait all weekend for them to verify my CC. This is despite me ordering from them a few times before. They called me, but when I called back they were closed. Then when I called Monday they said everything was fine ... so, I appreciate the waste of my time. Also, my items got back-ordered. Didn't get here until Friday (ordered the weekend before, 2day shipping).

NewEgg, on the other hand, had that stuff out and in my hands the second I pushed "Confirm." Well ... not that fast, but I've always loved NewEgg's speed. Also, they don't seem to sell back-ordered items like TigerDirect.


Things That Went Poorly Sad
--- I initially chose to use 16GB flash drives as the hard drives for all these machines. I figured they would use less power, generate less heat, and not use any space inside the case (better airflow). I mean, Live Linux CDs run just fine off flash drives ... right? ... Well, after building the first rig, it felt like I had taken a step back in time to the 90s. With only 1GB of RAM, stuff was getting thrown into cache a lot. Guess where cache is? Yup, it's on the flash drive.

It didn't make a difference to mining, but it made installation take hours. It also meant longer boot-up times, which would exaggerate any downtime.

So I grabbed 3 of my old SATA drives and used those instead. Works great!

--- The on-board GPU worked fine for the first mining rig. The other two, however, refuse to use it. They can't even see it. It's not a big deal, but it's nice to have in case the cards lock-up and to keep GUI load off them. Still not sure why this is ... (the option to use on-board doesn't even show up in the BIOS).


Things I Would Change  Wink
---In retrospect I would have dropped the case and bought a more expensive, single-rail, PSU. I wanted the case, because it was so cheap (like $50), and would provide easy mounting for fans and good airflow control. Well, I'll bet I could have bought a big, cheap shelf from the hardware store; mounted all the machines to it, and twisty tied all the fans around it. That would also have made wiring up the equipment easier.

Without a case, I would also have gotten a different motherboard with 4 slot capability. I didn't do that for this setup, because the 4th slot was at the very edge of the motherboard. That wouldn't fit the two-slot 5850s (fan on second slot). Without a case, though, that wouldn't have been an issue. Regardless, with a PCI-e 1x adapter I can get a 4th card on this motherboard.


Other Factors
---The first night of running the machines, they were all in my living room with the windows shut. I woke up to an oven, and decided to fry some eggs on the cases. While it was neat to cook breakfast on my mining rigs, the clean up was a tad messy, and ultimately I wanted the apartment to be less than sweltering. Also, I used the last of my green peppers so I need to buy some more.

I ended up moving the machines next to a window. A big appliance fan will be purchased later to push all the heat out the window. Like a Pro.

---The last ordeal of my story is something I had completely forgotten to take into account. Circuit breakers. Like most typical homes, my apartment has several circuits for all the 120V outlets, each circuit with a 20A or 15A fuse. These fuses blow if the instantaneous current exceeds 20A, or if the average current exceeds ~17A. That's about 1800W. The power strips I use are also rated for 1800W. If each rig draws 1000W at max, that's well over what my apartment can handle on a single circuit.

The 5850s draw ~180W each, which is 540W total. +100 for the rest of the computer and you have 640W. That's 1920W for all three machines.

I spent thirty minutes mapping out my entire apartment's circuit-to-outlet chart. With many thanks to the engineer who put this place together, the majority of the apartment's outlets are on a single circuit. Yay! So much fun! The rest are sporadic, with no logic to their placement. Also, I have no idea where two of the circuits are. They aren't hooked up to any outlets or lights ...

For now, I've got all 3 rigs running on one of the oddly placed circuits, which has nothing else on it. So far so good. I have one more nearby empty circuit I can use if things go awry.

It's all a huge pain. Hopefully something comes of my FPGA tinkering and I can replace these rigs with clean and efficient FPGA mining.  Cool

Closing Thoughts
Worth it? Yes! Mostly because it was like Christmas Day getting all those shipments, tearing open the boxes, and piecing everything together. It's also a total nerd fest, and warms my geeky heart to have a mini-datacenter churning away  Cool

Will it make me rich? Probably not. I'll be happy to break even and walk away with the experience. If it does churn out some cash, I'll pump that into my FPGA mining research.

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
ataranlen
Hero Member
*****
Offline Offline

Activity: 843


The One and Only


View Profile WWW
May 16, 2011, 03:19:37 AM
 #2

Very nice. Only thing you forgot to mention was total cost of your hardware Grin

MineTexas.com Minecraft Server We accept Bitcoin and Dogecoin.
Deepbit on Facebook: http://www.facebook.com/pages/Deepbit/151108048294815
SgtSpike
Legendary
*
Offline Offline

Activity: 1344



View Profile
May 16, 2011, 03:46:13 AM
 #3

3300MH/s is triple what I currently make (looking at your sig).  I wouldn't be complaining about that!  That's like... $120/day at current rates.  $3600/month... yeah.

Total cost would probably be in the neighborhood of $2300?
1bitc0inplz
Member
**
Offline Offline

Activity: 112


View Profile
May 16, 2011, 03:54:40 AM
 #4

Pictures, pictures, we need pictures  Grin

Mine @ http://pool.bitp.it - No fees, virtually 0 stales, what's not to love!
Chat with us @ #bitp.it on irc.freenode.net
Learn more about our pool @ http://forum.bitcoin.org/index.php?topic=12181.0
fpgaminer
Hero Member
*****
Offline Offline

Activity: 546



View Profile WWW
May 16, 2011, 04:05:06 AM
 #5

Thanks for commenting on my post!

Quote
3300MH/s is triple what I currently make (looking at your sig).
My total hashing power is 3500MH/s right now, but 720MH/s of that is from my original mining rig (cheap 5970 with existing hardware). So I gained an extra 2700MH/s from these new slaves.

Quote
Total cost would probably be in the neighborhood of $2300?
$2600 is the total cost after shipping and taxes. The taxes and shipping account for the $300 difference in your estimate Tongue

That's about the $1 per MH/s figure I was aiming for, so I'm happy with it. Smiley

I'll actually get a little bit of that money back from cashback and bitcoinbonus, rebates, and hopefully selling the free Steam codes that came with the video cards.

chaud
Newbie
*
Offline Offline

Activity: 17


View Profile
May 16, 2011, 04:20:05 AM
 #6

How much is this costing you in electricity / home cooling load increase electricity?
Kick
Jr. Member
*
Offline Offline

Activity: 45


View Profile
May 16, 2011, 04:42:12 AM
 #7

youre over hyping single rail design.

it's not that good. multi rail psus are better.

in case of a problem, everything connected to that one rail on the psu would probably go down.

in multi rails- if 1 rail fails, that rail would take what ever is connected to it down with it ( hypothetically ) saving your other parts
rickrollin
Jr. Member
*
Offline Offline

Activity: 41


View Profile
May 16, 2011, 05:31:13 AM
 #8

I'm curious if anyone has tried buying a server rack for their rigs and just stack them with a kvm switch and a massive fan / built in fans instead of placing things by the window and or their kitchen (like the one youtube video I saw lol)
SgtSpike
Legendary
*
Offline Offline

Activity: 1344



View Profile
May 16, 2011, 05:34:33 AM
 #9

I'm curious if anyone has tried buying a server rack for their rigs and just stack them with a kvm switch and a massive fan / built in fans instead of placing things by the window and or their kitchen (like the one youtube video I saw lol)
Server racks (and 4U cases) cost mucho bucks, unless you happen to find the right stuff used.  It'd be nice to have, but from a financial standpoint, it just doesn't make sense when you can make do with having fans by the windows.  In my case, I have A/C, so it's just a bit higher electrical bill than I'd normally see during the summer.
rickrollin
Jr. Member
*
Offline Offline

Activity: 41


View Profile
May 16, 2011, 05:49:40 AM
 #10

well in a server rack you can stack your stuff as long as you keep expanding to a point. You have more control over airflow too and can probably swoop one up for $400
bcpokey
Sr. Member
****
Offline Offline

Activity: 462


View Profile
May 16, 2011, 06:00:56 AM
 #11

One thing I will disagree with. That motherboard is stupid. Incredibly stupid. The layout is such that the 3 PCI-E slots are packed together incredibly tightly, with the horrible "push-clip" securing style, and to top it off, the placement of the case connectors and SATA ports are right next to the 2nd and 3rd lanes. This is ok for the sapphire 5850 xtreme, as it is a very small profile card, but a slightly larger, more powerful card will be completely unable to fit due to this stupid placement.

I have some other complaints, but I can't remember them at the moment, as I just threw my hands up in the air after I managed to finagle some things together that wasn't totally awful. So, it can work, but it's really not ideal.

My favorite board thus far is the MSI 890FX-GD70, you can find it a little cheaper than a new assrock if you're lucky and it gives you a whole lot more niceities, even if its a slightly older configuration.
fpgaminer
Hero Member
*****
Offline Offline

Activity: 546



View Profile WWW
May 16, 2011, 08:18:08 AM
 #12

Quote
with the horrible "push-clip" securing style
Oh man, I forgot about those. Yeah, they're terrible! I took them off (which was easy).

And yes, you're right about the case connectors. Those wouldn't fit a long video card. If you don't have a case, though, you don't need those connectors, so not a huge deal.

But...
Quote
My favorite board thus far is the MSI 890FX-GD70
Yeah, that's a nice board  Cool. It's a shame the last slot is unusable if you have a case.

But, as I said, I probably wouldn't use a case again. I'd just buy a wire shelf from the hardware store, cardboard, and twisty ties. Heck, you could even use extenders and dangle video cards from an upper shelf with the mobo below it Tongue

Bitsinmyhead
Sr. Member
****
Offline Offline

Activity: 376


View Profile
May 16, 2011, 08:39:31 AM
 #13

Great post. Thank you!

Trade Bitcoin using credit card and with 4:1 leverage at plus500.
Sign up through: http://www.plus500.com/?id=67176&pl=2 for €25 free, or become an affiliate here: http://www.500affiliates.com/?id=67176
keybaud
Member
**
Offline Offline

Activity: 112


View Profile
May 16, 2011, 07:53:51 PM
 #14

As I've built a 3 card rig, I'll tell you what I did to cool the middle card:

1. I put insulated spaces to separate the cards as much as possible.

2. I drilled 2 sets of holes in the side of the case right next to the 3 video cards and installed 2 fans blowing directly onto the middle card.

3. I set the front and rear fans to both exhaust from the case.

I had all the bits to do this lying about and it resulted in a temperature drop of 25 degrees on the middle card.
MoonBuggy
Newbie
*
Offline Offline

Activity: 12


View Profile
May 16, 2011, 09:22:01 PM
 #15

Interesting post, thanks. Do you need to make any specific allowances for CrossFire compatibility to get the best performance out of a build like this, or is it just a matter of throwing the cards in and letting them run independently?

1E2CuHoVHF3kbbXJmPwSVMrvFv8Vf7fko3
xenon481
Sr. Member
****
Offline Offline

Activity: 406



View Profile
May 16, 2011, 10:09:27 PM
 #16

I woke up to an oven, and decided to fry some eggs on the cases. While it was neat to cook breakfast on my mining rigs, the clean up was a tad messy, and ultimately I wanted the apartment to be less than sweltering. Also, I used the last of my green peppers so I need to buy some more.

With many thanks to the engineer who put this place together, the majority of the apartment's outlets are on a single circuit. Yay! So much fun!

Best parts of the post!  Grin

Tips Appreciated: 171TQ2wJg7bxj2q68VNibU75YZB22b7ZDr
VastLite
Jr. Member
*
Offline Offline

Activity: 32


View Profile
May 16, 2011, 10:55:08 PM
 #17

The "mystery case" sounds a lot like my Rosewill CHALLENGER. I find it to be a very nice case, in fact I have two of them. Others recommend cases such as the Antec 100 or one of the CoolerMaster HAF flavors, but for me (I'm only using single cards) the CHALLENGER is a very nice case. I've modded one of mine and mounted two 8w fans to the outside. They won't fit on the inside because they are industrial; they are full metal body and about twice as thick as a normal plastic 120mm fan. I'm not quite sure of the CF/M, but they are loud as hell and keep my card very cool, even when heavily overclocked. This is why I did not go with the cheapest CoolerMaster HAF, because it only have one 120mm side mount.

As far as the Antec 100 recommendation, it was from Diablo-D3, and the more I researched it, the more it makes sense for 3-4 card setups. It can fit two PSUs and has 2 or 3 120mm front intake fans (can't remember off hand.) - It is quite cheap, and I don't just mean in price; the metal looks like an extremely thin gauge and also very sharp. The shipping also kind of kills it, on Newegg it's about $12 to ship it, where-as the CHALLENGER is free and so is the HAF iirc.

Just my two cents; congrats on the builds and I definitely wish you luck in FPGA R&D. I would love to buy an add-in card specifically designed to run Bitcoin that got 30MH/w Cheesy
Littleshop
Legendary
*
Offline Offline

Activity: 1316



View Profile WWW
May 16, 2011, 11:12:16 PM
 #18

The "mystery case" sounds a lot like my Rosewill CHALLENGER. I find it to be a very nice case, in fact I have two of them. Others recommend cases such as the Antec 100 or one of the CoolerMaster HAF flavors, but for me (I'm only using single cards) the CHALLENGER is a very nice case. I've modded one of mine and mounted two 8w fans to the outside. They won't fit on the inside because they are industrial; they are full metal body and about twice as thick as a normal plastic 120mm fan. I'm not quite sure of the CF/M, but they are loud as hell and keep my card very cool, even when heavily overclocked. This is why I did not go with the cheapest CoolerMaster HAF, because it only have one 120mm side mount.


The Coolermaster HAF 912 is for me the best mining rig case for a 2 card miner.  You can even properly mount a 2.5" laptop sata drive (my favored drive now) properly without any additional pieces.

I used the ultra-blaster case for other rigs because I got them cheap, but seeing the HAF 912 I just wanna re-case them all now. 

grndzero
Sr. Member
****
Offline Offline

Activity: 392


View Profile
May 16, 2011, 11:32:54 PM
 #19


Yeah, that's a nice board  Cool. It's a shame the last slot is unusable if you have a case.


You just need a better case. Look for a PC-K57 or Lanboy. Vendors are slowly coming out with cases with 8 expansion slots in the back and will fit 4 double spaced cards.

Ubuntu Desktop x64 -  HD5850 Reference - 400Mh/s w/ cgminer  @ 975C/325M/1.175V - 11.6/2.1 SDK
Donate if you find this helpful: 1NimouHg2acbXNfMt5waJ7ohKs2TtYHePy
VastLite
Jr. Member
*
Offline Offline

Activity: 32


View Profile
May 16, 2011, 11:56:04 PM
 #20


Yeah, that's a nice board  Cool. It's a shame the last slot is unusable if you have a case.


You just need a better case. Look for a PC-K57 or Lanboy. Vendors are slowly coming out with cases with 8 expansion slots in the back and will fit 4 double spaced cards.

Or if you're very cheap, you can fit 4x dual-slot cards in the Antec 100.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!