Data Center Mining Garage and Man Mining Cave

[citronick]

@ groovy1962 - It really depends.  If you have to use box fans, those will be louder than the fans used on the Server rack rigs.  So it wouldn't matter.  If you're not using box fans, open air is of course much more quiet.  The fans are what makes all the noises so you can easily control that variable.  IF heat is not the issue, I would do Open Air All the way.  If I were to build a long shed, it would be open air 100%.  Cheaper, faster to install, faster to troubleshoot, etc.  Server rack is not Cheap.  Its' only necessary to fixed an issue...................HEAT.  The extra side benefit is SEXINESS.  But it's very expensive and time consuming.

P.S I can't wait for VEGA 10................Going to be such a huge Game changer with those HBM memories!!!  That's the same ridiculous fast memory that allow the slow Nano card to be able to kill everything out there for ZEC at low wattage.  June can't come soon enough, I have 1 entire Rack just saving up for VEGA 10.  It better live up to the hype!  No more GPU shopping for me.  Time to save up for VEGA 10!

My problem in the warehouse is heat. With full on XMR mining now, not so bad. But for "mobility" reasons - my warehouse is on lease so I will need a good strategy to move the gear when I need to; I have been itching to rackmount all of the current rigs (all open cage btw)... after reading yun's project, it finally inspired me to get a few Rosewill's chassis, server racks and start building.

I almost sold off my Nanos to GPUshack a few months ago to buy 470s -- I am so glad I did not do that.

These Nanos with HBM are absolute speedster at stock 150w. For such a small card, its easily the fastest cards I have in the farm for any memory intensive algos like ZEC and ETH.

[citronick]

It absolutely would.  ReLive drivers don't play well with mod BIOS.  It put back the whole signature mess.  DONT USE IT.

If I recall .... its because of this signature signing mess that prompted me to use smOS ..... I sleep better now with Linux.

If you ask me.... I dont even know what the most recent version of the Crimson driver... 

[yun9999]

Best to save up for Vega 10.  Less rigs to manage if budget is not an issue.  Less cases to buy / build, MB, CPU, SSD, Mem to buy and build and they should keep their value easily for at least 12-18 months.  Worst case scenario, one would always be able to mine for at least 3 month with the high end cards and then dump them for almost free mining.  This is granted you're able to get them at launch.  This have been true for most high end GPUs.  My entire 6th Rack is reserved for Vega 10.   HBM IS THAT GOOD!  Let's hope it lives up to the hype but so far we're already hearing it's going to kick 1080's butt.  HBM memory unlike normal memories can be tweak to use very very little power without effecting much mining hashpower.

P.S if one is to look carefully to the left of the MSI GPU box rack, you will see an entire array of FURY boxes.  Yes, they also uses HBM memories.  =)

[vapourminer]

@ vapourminer  - Square D makes a good one too.  I can have this side mounted as well but it would require drilling thru the metal panel which I don't want.  The wires are very short and won't reach top.  My Electrician assures me it's high quality and I don't need to look at it daily.  He said he will be happy to come open the panel and check for me every quarter if it makes me happy.  =) 

@yun9999

chose the square d as my electrician recommended it. it has 2 green "Im OK" lights on it visible from the outside of the case, if either goes off its time to replace the MOVs (which are user replaceable).

my 200 amp panel has no real room in it for an internal unit, so..

i love the whole house suppressor bit as it also protects my 220 v hybrid hot water heater (air sourced heat pump based) and my 220 v geothermal (ground sourced heat pump) system.

[bit off topic follows, unless you will get backup power in your setup)

also put in a 7000 watt generac transfer switch to go with the 8500 watt key start briggs and stratton generator while we were at it. peace of mind is worth it. winter gets COLD here in the northeast with no power. not to mention having A/C in the summer when the power drop out. we have several power failures per year here, some several hours long. thats what we get for living in the sticks.

best part about the generator and transfer switch: my wife can switch the whole thing over safely start to finish with no help needed.. no risk to her or linesman working out on the poles.

[vapourminer]

It absolutely would.  ReLive drivers don't play well with mod BIOS.  It put back the whole signature mess.  DONT USE IT.

If I recall .... its because of this signature signing mess that prompted me to use smOS ..... I sleep better now with Linux.

If you ask me.... I dont even know what the most recent version of the Crimson driver... 

gotta love the relive drivers.. NOT. i have a strap modded 470 and tried the relive 17.1.x driver. installed to the end with NO error message but didnt actually install the driver. like wtf?? least it could of done is toss a message saying "driver not installed".

back to 16.10.x for me.

[DebitMe]

It absolutely would.  ReLive drivers don't play well with mod BIOS.  It put back the whole signature mess.  DONT USE IT.

If I recall .... its because of this signature signing mess that prompted me to use smOS ..... I sleep better now with Linux.

If you ask me.... I dont even know what the most recent version of the Crimson driver... 

gotta love the relive drivers.. NOT. i have a strap modded 470 and tried the relive 17.1.x driver. installed to the end with NO error message but didnt actually install the driver. like wtf?? least it could of done is toss a message saying "driver not installed".

back to 16.10.x for me.

Sounds good, I will give this one a try this evening.  Thanks for the help guys!

[m1n1ngP4d4w4n]

*********** STORING BOXES!!!!!!!!!!!*********************

The Larger your farm, the bigger your headache is with proper storage and organization to quickly find that GPU box in time for reselling, match SN# and all.  I don't have a big house so space is a premium.   This being said, most of my storage are in the Attic and it's VERY HOT up there.  So my goal is to be able to make space capable of storing over 2000 boxes such as GPU, PSU, MB, CPU, packing material etc.  I also built a shelf in the garage for Older Gen cards as I know those will be the ones that I need access to quicker for RMA and reselling.  Thank Goodness I don't need to keep the Rosewill boxes!!  That would fill my entire house!

The Plan
1) Quick access to any GPU or hardware box, I need to sell quickly and minimize heat exhaustion
2) Segregate my stuff from the stuff I'm hosting for my buddy

The Solution
1) For things I don't anticipate in selling anytime soon (PSU, CPU, MB, etc).  I put them in a corner that is further away.    Each section is separated (PSU, MB, shipping boxes etc)
2) Oldest Gen GPU boxes are stored in the Garage shelf that can hold over 200 GPU boxes organized by GPU model (390, Ref RX480, etc)
3) Private VIP section for my hosted buddy
4) For GPUs that I just bought and need quick access to in case of returns or whatever reason,  I repurposed my 48" wide rack for this.  I can split 1 rack into 2 halves and each halves can easily hold over 80 GPU boxes so 3 of them in a U shape when I open Attic access can hold over 240 boxes for very quick Access and verification of serial #.  I also Mark the box A for Amazon, J for Jet.com, and N for Newegg.com.  The reason for this is that Newegg is the toughest Vendor to do returns.  Amazon is the easiest.  If you accidentaly send a Newegg.com GPU to Amazon, they would accept as long as its' the same product.  If you sent a Jet.com or Amazon GPU to Newegg, the UPC won't scan or something and they will always reject.

This solution allows me to just quickly pop my head up and locate the GPU box very quickly and outside of the heat and then takes less than 1 minute to grab it.  This avoid Heat Exhaustion digging for that particular box!  I also put them stack 6 or 7 high based on the same rig where they are installed so if I locate 1 GPU for a rig to sell, the rest are nearby.  Small effort up front but man with hundreds if not thousands of boxes, this is like trying to find a book in the library after a storm without this system!

@bughatti - Don't sweat it bud.  No one is upset at ya.  Who would ever hate a Bughatti?

Have a suggestion for you, to find easily stuff, you could have it QR-coded (serial/partnumber) you would be able to setup a camera in front of boxes and find the one you wanted before even going. I use this same system for cataloging always moving hardware racks @ work, make it easy on everyone You could also just print a wider and much bigger part of the serial (like the last X numbers or letter to put beside the box, to be able to find it with the same system.

[Finksy]

Looking at that motherboard, I see an unused M.2 slot next to the far left PCIe slot.   Why not get a M.2 to PCIe adapter and another riser and that motherboard could then run 8 GPUs?  I have seen these adapters for $10.  Works for PCIe M.2 slots, not SATA ones, of course, but the PCIe X4 m.2 slot is more than enough to drive a X1 riser.

I'm considering trying this but have not had a chance to do so.  Wanted to know if others have done so, and if there's some reason it wouldn't work.  (Might require a Z270 MB to have enough PCIe lanes in the chipset.)

I have definitely considered doing this, but was advised against it by people MUCH smarter than me  To be honest, getting a 7-GPU rig stable can be challenging enough at times, and limits how much OC'ing you can do on individual cards as it is, so I am happy with them as they stand without another avenue for frustrations, instability or failures. If you do implement it on a larger scale let me know how it works for you!

[m1n1ngP4d4w4n]

I have definitely considered doing this, but was advised against it by people MUCH smarter than me   To be honest, getting a 7-GPU rig stable can be challenging enough at times, and limits how much OC'ing you can do on individual cards as it is, so I am happy with them as they stand without another avenue for frustrations, instability or failures. If you do implement it on a larger scale let me know how it works for you!

I can't agree more with your finksky, 7+ gpus rigs are very hard if not impossible to make stable, and usually require a much higher end motherboard (to have more pci-lane) so you lose quite a bit of the cost you save, not counting the hundreds headaches. 6x is the way to go IMHO 

[yun9999]

@Vapourminer - All 3 are good. Leviton, Eaton Ultra or Square D so whatever is available at good prices is a winner.  Any brand is still better than zero protection.  My family members just started adding them too.  For $100 for the hardware, if you can DIY, it's a no brainer.

@ Debitme - That's definitely the issue so no worries.  

@ m1n1ngP4d4w4n - Great idea but that method does add to the cost (labels) and more time to implement but the larger FONT would definitely help many people with poorer vision which will be me soon.  However, I prefer the K.I.S.S  method.  Keeping things Simple also help reduce cost.   Labels are not cheap.  I end up buying multiple label makers as they were selling them for $9.99 to $15 when on sale with the sample roll vs new rolls costing $27.95 to 34.95!!!  Its ridiculous what these companies are doing.  Much worst then normal printers.  



@ Finksy - Yeah, it's never a good idea to do 8 GPU unless it's direct attached and no risers needed like Pandas.  More components especially crappy risers = higher chance of 1 GPU taking the entire rig down.  I have the M.2 and you don't need very expensive MB to support 8 GPU.  The MSI Gaming 5 can do that.  You will need Linux for that I believe.  

My M.2.  I was going to mess with it for fun but decided against it.  It's hard enough keeping a rig stable w 7 GPU.  

[m1n1ngP4d4w4n]

@ m1n1ngP4d4w4n - Great idea but that method does add to the cost (labels) and more time to implement but the larger FONT would definitely help many people with poorer vision which will be me soon.  However, I prefer the K.I.S.S  method.  Keeping things Simple also help reduce cost.   Labels are not cheap.  I end up buying multiple label makers as they were selling them for $9.99 to $15 when on sale with the sample roll vs new rolls costing $27.95 to 34.95!!!  Its ridiculous what these companies are doing.  Much worst then normal printers.  

Oh i wasn't thinking doing proper labels we print them on very cheap paper, like 5 labels per sheet of paper, and print in eco/gray, the cost is barely anything , can't say anything about time, it does consume a bit of it, but we save 10x that time when we look for stuff. We bend the paper and insert it in the box opening, this way also no need to fix it or doesn't damage the box.

[bughatti]

@Yun, I have no plans for an 8 gpu rig but can you tell me what m.2 to pcie card you got to work on the msi z97, I bought one but it did not work.  Would like the ability to give it another try in case I have a bad pci slot on a board

This one?
http://www.ebay.com/itm/Q13025-WBTUO-LM-141X-V1-0-Drive-M-2-NGFF-to-PCI-E-X4-Adapter-Card-for-Desktop-PC-/351333328902

[yun9999]

@ bhugatti -  I never even tried further as I couldn't get Windows to see it or power the GPU even with the M.2 powered from Molex and my riser is powered by Molex.  My buddy told I need to disconnect another riser, do this do that, change Bios, Linux, blah blah.  Then I thought about instability and just say better to not even waste my time.  It does work with Linux, there are people who have it running with it on Gaming 5.  

I am easily side tracked.  Might be fun to re investigate at a later date for 1 of my Play rigs but right now I just don't have any cycles.  The one you show on ebay doesn't appear to have the Molex power cable, you will need that.  

@ m1n1ngP4d4w4n - Yeah for the most part, 6 GPU rigs are the way to go for Open air.  Very easy to troubleshoot and lower your chance of having 1 GPU with Crap ASIC that can't handle the ROMs.  However for Server Rack and for large deployments, we have to squeeze the max out of each Case due to higher cost per rig and space, network, rack, power cables, etc required.  For most optimal temperature in server cases, 6 GPU is also better.

However, for my Setup the difference for 60 rigs when using 7 GPU config vs 6 GPU is over $5K in savings plus 1 RACK less.  Very surprising right?, Considering that the MSI is $65 more expensive then the ASROCK.  However you're able to squeeze in 1 more GPU and the underlying MB set to support each rig (MB, PSU, CPU, SSD, RAM, Server case, power cable, Rack etc) makes it more efficient to use 7 vs 6 GPU configs.   So it's definitely a great value and I get a more premium MB with the super valuable BIOS Board Explorer feature.  I can also reboot any of the rigs without needing Emulators on MSI MB vs ASROCK.  If I do need high rez, I just connect the monitor cable and it will auto switched to High rez.  The benefit of 6 GPU config is better stability and cooling due to less heat and more spacing. 

Unfortunately I do love Furies so for those rigs, I still use ASROCK H81 with Emulators.......  The reason for that is spacing required for the Furies. They are longer cards so the thinner H81 helps it fit better.  I can squeeze a Gaming 5 in but again I can only fit 6 GPU so it's a waste.   My Open Air config is mostly H81 as the saving there is less substantial and I started off with a ton of H81.  As usual, there's never a 1 size fit all solution.  =)  I can't wait for the Biostar MB to come as we do need a good alternative to the H81 that is always sold out and the Gaming 5 which have went up from $90 used and $119 new to over $110-119 used and over 149 new.   

P.S I love ASROCK RMA, they are very good and fast unlike what you read in the Reviews.  MSI RMA..........................hmmmmmmmmm..................Caveats Emptor.

[citronick]

I have definitely considered doing this, but was advised against it by people MUCH smarter than me  To be honest, getting a 7-GPU rig stable can be challenging enough at times, and limits how much OC'ing you can do on individual cards as it is, so I am happy with them as they stand without another avenue for frustrations, instability or failures. If you do implement it on a larger scale let me know how it works for you!

I can't agree more with your finksky, 7+ gpus rigs are very hard if not impossible to make stable, and usually require a much higher end motherboard (to have more pci-lane) so you lose quite a bit of the cost you save, not counting the hundreds headaches. 6x is the way to go IMHO  

I think the 7-GPU rig really depends on the GPU type.
More forgiving cards like R7-370s - no so bad and this is the GPU config that I got working on H81 pro BTC mobo with 3-to-1 PCI splitter. This was one of my first rigs after 390s,290s rigs - 370s were on sale cheap back then. The rig is still in production and although it's not really that much of speedster vs 480s, Nanos, 390s, 290s... it's still a decent rig and it has amazing uptime life - very stable and doesn't even go beyond 65c temps.

Attempting 7 or 8 GPUs under Windows or Linux for 480s or Nanos -- I didn't get past first attempt. Too many variables and hacks to consider to make it to work. Then came Pandaminer w 8x480Ms-- now thats a beast. Suddenly there is no challenge to build 7/8-riggers anymore...

[rjg]

Wow...  Thanks for sharing all of your experiences and great ideas.  I am learning quite a bit by reading and following along.

I do have a question that seems appropriate based on your garage experiences.  I am in Little Rock, AR, so a bit further north than you but seemingly just as hot, just not nearly as humid (speaking from my experience visiting Houston in July  ).

I currently have 48 GPUs in the garage (double car) in an open air config and will be adding another 24 GPUs before my power capacity is tapped out.  I am using metal bladed fans blowing on the GPUs which seems to be doing the job at this point with the garage door cracked open about 8" or so and the attic access opened up.

I have ordered and received 4 of the Windeevent (https://windeevent.com/) units and I am trying to determine the sizing for a fan in the ceiling of the garage to exhaust the hot air into the attic.  I have been looking specifically at the QuietCool (https://quietcoolsystems.com/) whole house fans due to their easy installation, warranty, and very efficient motors.

The information that I have found suggests that I will need about 1.5 sq ft of intake for each 1500CFM, at this point, with the 4 vents, I have just over 4 sq ft so I have enough intake to support a bit over 3000CFM.

Can you offer suggestions on how many CFM I should be shooting for, I am thinking at least 1500 and more likely something around 3000 for the estimated 3500 cu ft of garage volume.

Thanks in advance for your help!

Robert

[DebitMe]

@ Debitme - That's definitely the issue so no worries.  

HAZZAH!!!  It is working!  Thank you for the help!

[yun9999]

@ Citronick - With Windows 10 and Linux, no hack is required unlike with Windows 7. I have over 40 rigs running on MSI Gaming 5 in a 7 GPU config.  No issues.  If there's stability issues, it's usually a problematic riser or GPU.  Just need to isolate and remove the issue and it will work fine.   The same if you're using ROM and 1 out of the 7 cards ASIC can't handle the OC settings.  Hence the more GPU we add into a rig, the higher the chance something will fail and cause issues.  RX470 and 480, Furies are stable enough to handle 7.

@ rjg -  @ 72 GPUs in a 2 car garage should be manageable.  Put all 4 of the Windees where you want the cooler air to come in.  72 GPU is roughly 12 Rigs.  Make a U shape type wall so that the intake will blow in 1 direction and the hot air will go toward that u type shape wall.  You can buy very inexpensive heat wall foam material at Home Depot.  They're also very good fire resistant material and very light / easy to cut.  

Your rigs will enjoy nice cool air and all the hot air will be extracted to the Attic.  Since you're blowing into a smaller area within the heat wall.  You have less CFM to cover then the entire garage.  This will allow you to use cheaper lower CFM fans.  My friend have the QC fans.  They're very good.  I would go with the 3K CFM version as they're very inexpensive.  12 Rigs in a garage is very manageable if they're RX 470/480.  Watts is also a measurement of heat so it really depends on total wattage.  So it's not about how many GPUs or rigs but how many total watts you're pumping out.   I also have friends that use 3 windees to push out the heat and 3 for intake.  All depends on what is the best config for you but should be no issues for 12 rigs.  Each rig above that = extra challenges.   When I first started out, it's just with 4 windees and 1 exhaust fan.

@Debitme - Yes.  very common issue, DAMN U AMD for tricking us modders with RELIVE drivers.

[arielbit]

i bought 5 pcs socket 775 motherboard with 4x pcie for 80$
7 pcs 80gb HDD for 35$

still looking at processors at 16-20$ each (quad cores)

i still have some extra ddr2 rams here, will buy more at 7-10$ each

that's 20 GPU potential expansion for only ~300$  (GPU and PSU not included)

the motherboards are like the one i've been using since 2013 and all are still fine today...the other thing i like about that board is they have 4 LED that if it all turns green-i know it will boot to windows except if there is a HDD failure, it also has LED for PCIe connection.

[DebitMe]

@ Debitme - That's definitely the issue so no worries.  

HAZZAH!!!  It is working!  Thank you for the help!

Well crap, it fixed one of the cards, but the other is still giving the same error message.  Not sure what to do.

[onefinecoin]

i bought 5 pcs socket 775 motherboard with 4x pcie for 80$
7 pcs 80gb HDD for 35$

still looking at processors at 16-20$ each (quad cores)

i still have some extra ddr2 rams here, will buy more at 7-10$ each

that's 20 GPU potential expansion for only ~300$  (GPU and PSU not included)

the motherboards are like the one i've been using since 2013 and all are still fine today...the other thing i like about that board is they have 4 LED that if it all turns green-i know it will boot to windows except if there is a HDD failure, it also has LED for PCIe connection.

You should also look into these used HP & DELL core 2 duos PCs, they are sold at a very cheap price and have 3-4 PCIe slots. As electricity is not a big concern where I m located, i am planning to expand using these. So the CPU will run as independent system and the 3-4 GPU can run over extenders and a separate PSU.