Data Center Mining Garage and Man Mining Cave ( PART 2 !!!!!!!!)

[yun9999]

*********DRAFTING!!!!!!!!!!!!!!!!!!!!!!

My original thread was getting over 22K Views in less than a month but I forgot 1 important thing.  I need to make myself the MODERATOR.   So I'm redoing / continuing the thread with this extension so I can make any edits / changes where it will be Easier for new Readers to follow as well as for original followers to be able to continue where we left off.

Link to Original Thread.
https://bitcointalk.org/index.php?topic=1761303.0

MY FARM SETUP

I have been mining for a very long time.  Most of you would know me as Nhan Do from my old Nick that was Hacked, hence why I'm using this new Nick.  I have been thru the many ups and downs of Crypto as well as hundreds of designs and layouts.  What I learned from my experience / ]mistakes, is that it's always best to keep things SIMPLE, EFFICIENT and use Quality components and only buy / support reputable vendors / brands.  I want to have this thread so that many other readers can hopefully learn from my many mistakes and have a better start / experience then I did when I started.














Please SUPPORT my Buddy's Store if you love this THREAD.

www.mining4less.com

[1713569318]

1713569318

[1713569318]

1713569318

[1713569318]

1713569318

[1713569318]

1713569318

[yun9999]

Mining Motherboard Review

The Baseline 6 GPU LGA 1150 Motherboard (DDR 3 memories)

ASROCK H81 BTC Pro (V2 is the same)

This is the go to Card for miners.  6 GPUs works no issues.  HDMI Emulator or monitor cable needs to be connected to the 16X GPU in order for the system to boot.  Molex connection to the Motherboard is optional.  Hit the N button to not get that message anymore.  I don't use the Molex to MB, never had any issues.  Don't believe the Rumor, ASROCK RMA support is Phenomenal and replacement speed is super fast.  They even gave me brand new V2 when V1 was not available.  Seems like all the replacement boards I got so far which is over 6 are all brand new.  This board is designed with mining in mind so it also have the Dehumidifier feature which is great for Houston.  



ASROCK H97 Anniversary

Make sure not the Mobile version which have less PCIE slots.  This is similar to the H81, but add additional RAM slots, more USB connection and a few more things.  It just adds to the cost, miners don't really need any of the additional stuff.  The reliability seems spottier based on my Sample Miner group vs H81.  It is also "REQUIRED" to plug in the Molex to the MB to get more than 4 GPUs working on this board, something I find annoying.  This was the go to MB after the H81 BTC pro was out of stock.



Biostar TB85

This is a good alternative to the ASROCK H81 BTC Pro.  You should have no issues doing 6 GPU with this Motherboard and prices is close to H81 and more readily available.  It is a big wider than the H81 so keep that in mind when building for your Rack.  Yellow slot MB is the Biostar.





The Baseline  GPU LGA 1150 Motherboard

MSI Gaming 5

This my premiere Sexy and go to Board for everything.  I love the BIOS Board Explorer feature so I can spot which GPU inside my Server case is having issues without needing to take it apart.  It can do 7 GPUs no issues.  Don't listen to what others are saying.  I have over 35 rigs working with 7 AMD GPUs no issues.  I have very limited experience with NVIDIA, so I can't say as I never did more than 6 NVIDIA cards 1070 ever and even that was a challenge when it first came out.  




See Slot 2 not showing up as connected / RED?  That's the one I need to fix.  Open Case cover, receded or replace that riser.  FAST / EASY!  For other MBs, I would have to do isolations testing which is more time consuming.  Oh yeah, that's my Long Island Ice Tea.  Got to have a nice drink while working on Rigs.





LGA 1151 Motherboards reviews  (Requires different CPU and DDR4 memories)

Have 7 PCIE Slots but will it be able to support 7 GPU via RISERs???   (T.B.D!!!)

This BioStar board is ABSOLUTELY GORGEOUS and looks even more incredible in real life with it's Carbon Fibre and LED lights.  BUT WILL IT SUPPORT 7 GPU??? 




To allow this Motherboard to support more than 4 GPUs.  Please follow the following steps in the BIOS.  This MB is problematic for Mining.  I can't recommend it.

(1) set all PCI-E at gen2

(2)go to BIOS " Chipset" > System Agent (SA) Configuration > Max TOLUD > Set to [3.5GB]



Motherboard testing in the LAB......................

MSI Gaming Carbon ( 7 GPU LGA 1151)






*********************TO BE CONTINUED***************************

[yun9999]

*******************Simple Rig Resetter (SRR) REVIEW **********************************

//DRAFTING//////

How many of us wake up to Rigs being Hung or have RIGs that is constantly requiring reboots?  I'm sure everyone does.  Not only does this cost us money but it also makes us worry about taking vacations, or be away from home.  I have over 65 rigs between myself and a friend I'm hosting for.  Embarrassed to say but it's the reality we live in, I usually wake up to at least 1 rig that is Hung or having mining issues.  That's a few bucks a day which doesn't seem like much but compound that to a few rigs x 30 days a month and you can see it can get very EXPENSIVE.  This is why SRR is so important to me.

Switched APC vs SRR

APC Switched PDUs

Pro
* You need a PDU anyways.  But it doesn't have to be a pricey APC.  ($15 Dell PDUs)
* Simple to use Web interface
* You can name the devices

Con
* Pricey at $100-150 used
* Max is 4-5 Rig support per PDU due to amperage
* Difficult to configure if you don't know.  You must buy the APC serial cable to connect via Hyperterm to set it up.  Most people don't know this or don't bother and end up using it as an overpriced Metered PDU.  I know I did in the beginning as my APCs didn't come with this cable.
* Not tied into the OS for automatic rig rebooting when a rig is hung
* DOES NOT save you money as there's no automation, you have to know a RIG needs to be rebooted and manually do so
* It's difficult and more expensive to run longer high AWG power cable then SRR signal cables
* Documentations are limited and my guess is many people will run into issues

SRR

Pro
* Very cost effective compare to APC for more than 5 rigs
* Simple to setup and use.  You can be up in less than 15 mins and most of the time required is just for plugging in Cables!
* Built in OS agents when using Simplemining.net OS
* Software to detect SRR and configure is very simple to use and Fast scanning.  
* Very SEXY and compact designed with attactive LEDs status.  
* Cables used are super high quality.
* Very well documented PDF on the how to steps
* Very simple and intuitive interface

Con
* Expensive for small setup of less than 5 Rigs
* Pricey cabling (Tytanick is working on a more cost effective cabling solutions)
* Limited to 8 port for the base unit (Expansion boards available soon)
* When you reboot the Status for the RIG still show up as Green vs RED ( Can be fix with updates)
* Single point of failure
* Can't name the ports ( Can be fix with updates)



***********The HARDWARE***********************

Lots of attention to Details!!






The unit is very Compact and the Cabling is Super High Quality!  At first I was not too happy about the pricy cabling as I thought it was normal cheapy on/off cablings but wow am I impressed.  Very happy with quality and I don't have to worry about some skinny cable snapping when I'm working on my rigs.  I was going to do a disconnect cable setup for my server cases config, but I like this so much, I think I will make the extra effort just to disconnect from the base if I need to.











The LEDs and sexy cabling makes me want to show off my SRR on the side vs hiding it in the rear







The product is super easy to setup and Tytanick's PDF for the step by step is very well done.  I finished all 8 of mine in about 15-20mins and most of that time was for running the cable as I like them organized.  





If you're using Simplemining.net OS, his agents are already built in.  You just have to choose SRR and point to it.






The scanning and management software is very easy to use, FAST and intuitive.  Just download, install and choose Scan and it will find the SRR device immediately.  I DO NOT recommend changing the IP addresses at this time.  When you do so, depending on what other services you have running, you may run into issues like I did.  There's no reset option to bring it back to factory default.  It's best to leave it as is as it scans based on other info and will auto detect, no need to memorize what port and IP addresses.





The Software is very intuitive and all the buttons speaks for itself.  The only thing that is important and this is why it's repeated many times in the PDF is to set your watchdog timer for at least 120 seconds.  This is the time out value where it would see no response from the Rig and send out a reboot / power on / off request.  You can make it longer if you want.  I tested the reboot and the RIG button was still showing as Green.  Minor issue that can be fixed via future updates but it worked as designed.  For windows rig, you have to install the SRR agents if you want the Watch dog auto restart features.




NOW I can feel more comfortable going on Vacations!!  Also this gives my Hosting partner more control / automation so they don't need to bug me as often for reboots.  I spent over $800 on 6 used APC 7911 but to date it have not saved me a dime.  Why?  Because I have to identify the issue first and then reboot it.  There is no automation.  One could argue that part of that is not wasted as you're still using it as a PDU. For me, I have my beefier Dell PDU that have more Amperage per Block so it wasn't necessarily needed.  


FAST ROI
This unit pays for itself super FAST.  I always recommend my friend to invest in Smart monitoring and automation tool as it will save you money in the long run.  Most of them tried to avoid the cost but eventually end up doing it.  So it's better to do it from the start and make your life easier.  This is based on my own example but the larger your farm and the more you OC, the more likely this relates to you.

Assuming you have 1 Rig down per day while sleeping or at work
8 hrs = $2-3 per Rig x 30 days = $60 - 90 opportunity loss.  

Now if you're on Vacation for 1 week and you have  1-2 rigs down.  That essentially pays for the entire SRR and ruin your entire vacation which cost even more.  I also know many people that use hosting facility and many times their SLAs are crap.  Those idle time waiting for someone to fix your rigs add up very fast.  I know at least 2 friends that are dying to get the SRR to their hosting site as they lost hundreds if not thousands due to idle and issues.

I get zero compensation for pushing SRR and I can't wait for the Expansion board to be available as I'll be the first person ordering.  This Product is Amazing.  You simply can't go wrong with this device.  I would even argue that it may be silly to be mining with 8 rigs or more without this.  The reason for that is there will always be HOT new coins and those new mining software will always be flakey.  If you have automation, you're up more and are able to be making much more.

[yun9999]

************PICO PSU*****************************************

Finksy's version will be SEXIER and more POWERFUL.  This is just to give everyone an idea of how PICOs work with the Server PSU.  In order for Finksy to get us attractive prices, he have to buy in good volumes.  This is where I urged Hawkfish to partner up with Finksy to make this happen.  It's not part of their core business so both of them don't want to be stuck with huge inventory of these things.  I said "DO IT FOR THE COMMUNITY, there WILL BE a huge demand for it once people understand the value it brings.

The goal is to get a complete turn key solution including all custom cables for around $30 while saving everyone from the headache of buying crappier / more expensive solution like I did and then hunt for specialized cables since the PICO was designed for mini PC not mining rigs.

Why a standard PICO PSU won't work for us
* The CPU cable won't reach a normal Motherboard CPU slot.  
* It requires an expensive and Bulky AC power adaptor.  Who needs another thing to plug in or waste another outlet?
* It will require a custom 6 Pin to 2.5MM barrel cable to power the Pico if you dont want to use the AC adapter and power it via server PSU
* It will require a 6 pin to 4+4 ( pin EPS CPU cable if you want to make it 100% server PSU powering or a 4 pin to 4+4 (8)Pin EPS cable if you want the PICO to power the CPU.  

What I got from Ebay

$40 for this weak 80W combo with no cables that I need.  I still had to go and find the EPS cable for $10 on Amazon and it was very difficult to get the 6 pin to 2.5MM barrel adapter cable.  I ordered 2 different version and only 1 worked.  More money wasted





6 Pin PCI to 2.5MM barrel adapter to power the PICO via server PSU.  Another cable was used to power the CPU via the server 6 Pin to 8 Pin EPS.  This waste 2 valuable PCI slot from the server PSU breakout board.







WHAT WE WANT................
Who wants to search on Ebay for separate items just to wait a month from China, overpay and get wrong stuff like I did?  No one.  I had to do many trial and errors and the complete solution worked and is better than the above but still not cost effective nor SEXY as I want it.  We want 1 solution that is tested and have everything so there is no guessing game.  We at least 160W of Pico power for the same price as 80W.  We want to use 1 6pin cable that splits and power the PICO and can also power the CPU.  This will reduce the PCI slot wasted per rig.  Most importantly, WE WANT IT CHEAPER!




WHY WE WANT PICOs.

* It's super Compact and sexy.  The entire PSU is just a the size of your MB plug adapter.  You hardly see the thing once it's plug into the MB slot.  I purposely left the cables outside just so you can spot the PICO!  If I hide it's tiny 4 pin CPU and SATA cable, you won't be realized a PICO PSU is plugged in.



* It's very cost effective if we can get it for $30.  You can't get a quality baby ATX PSU for under $50 raising your cost per RIG
* Less Cable clutters and space needed in the rear.  This help improve air flow and cable management



* PCI Powered Riser is also much higher quality build then normal Molex to SATA riser.  No more issues with those cheap Molex connectors coming lose and shorting via contacts and causing a small spark / fire.  The Capacity and everyhing on the new PCI version Hawkfish007 is one of the best quality version to date.  Do not be fool with the cheapy ribbon version, those won't work well.

Picture Powering 4-5 x 6 GPU rigs this solution easily when paired with Finksy 4kw PSU bundle.  SO FRESH FRESH...........SO CLEAN CLEAN!



NOW HERE IS THE KEY WHY YOU SHOULD want it.  On top of being SEXY, Cost effective per RIG, you now move 98% of your Power load to the more efficient GOLD to Platinum rated Server PSU.  These things are TANKS and are designed to be run for years 24x7 vs consumer grade ATX which are not.  When you're using the old way with 4-5 Baby PSU, those are generally cheapy 80+ or less efficiency so this solution pays for itself very quickly via the efficiency.  P.M Finksy and Hawkfish007 and tell them.  WE WANT PICOs.............WE WANT PICOs!!!!!!!!!!!!!!!!!!

[yun9999]

************  THERMAL CAMERA *********************************

Got this recommendation from m1n1ngP4d4w4n.  What a wonderful device!  I previously used the Laser Heat gun which was very cheap and great at quick spot checking but nothing like this THERMAL CAMERA!

You can quickly view all the HOTSPOTs.  

This is my Mining Shed.  You can even see the Floor concrete blocks heating up from the Sun



You can see that the Patch Panel are getting very warm.





It starts to get more interesting when you add the spot temperature indicator which let's you know if it's hot enough to be of a concern or just hotter than the other areas.

Here I'm comparing the different GPU temp to find out which one is having issues.  Usually idle or failing GPU will have lower temp but not always.  I can also use the same feature to see if any of my SATA to Molex cables are of poor AWG and are overheating (ready to melt, cause fire, etc).




I also use it to look from my intake air side of the Rack to see where some of the hot air is recirculating back and how I can improve it.  As expected the top side where I have 3 U of empty space is recirculating back some heat.  The very top side where you see all red is just the temp or the Heat wall.



I use the same camera to see my hot zones in the Shed Rack to work on the flow redirection to fix it from going from bottom to higher racks.





There's a million use for this camera.  Here's another one.  Instead of pulling out a thermometer.  I can just scan my kids to see if they're ok or is having a fever.  That's how I justify the cost with the wife.  =)

[yun9999]

Reserved for Content 5

[yun9999]

Reserved for Content 6

[yslyung]

holla...

[philipma1957]

Good you got it with delete option.

Please leave this one in this place.


The link to your old thread

https://bitcointalk.org/index.php?topic=1761303.0

[yun9999]

@ Phil - Great idea and added.  Thanks.


@ Yslyung - Holla

************My Uncles ************
As Usual, my uncles are very supportive.  As soon as they hear I started a new Self Moderated thread, they said GREAT IDEA!.  

Here's some Free ETH for u.

[melfice123]

Hello, any tips for use Pico psu and server psu to power on at same time? have 1,200w server psu using for cards and 700w gold psu for motherboard, proccesor, hard drive, etc, and 1 Pico psu on the way to replace 700w atx psu.

Greetings! 

[hawkfish007]

@ Phil - Great idea and added.  Thanks.


@ Yslyung - Holla

************My Uncles ************
As Usual, my uncles are very supportive.  As soon as they hear I started a new Self Moderated thread, they said GREAT IDEA!.  

Here's some Free ETH for u.

The best way to start a thread with a visit from uncle, congrats. I am drooling looking at your setup...

[citronick]

OK ... first new topic from me...

I have been meaning to ask the team on their network infrastructure - I am no expert in this field but enough skills to supply Internet access to all my rigs.

I dont use any network analysis tool - not needed because broadband speed is quite good.

But.... I think I am beginning to have some network congestions in the farm.

I have divided my warehouse into 5 zones.

Broadband Internet
As provided by ISP
            I
            I
MASTER: TPlink 48-port Switch (Zone 2, Zone 3, test area)
            I
            I
---> Zone 1 - TPlink 16 port switch x 1 + TPlink 8 port switch x 2 (basic cascade) - 100% used
                                                                        
---> Zone 4 - TPlink 16 port switch x 1 - 90% used
                                                                        
---> Zone 5 - Cisco 24 port router x 2 (basic cascade) - 90% used

Now the issue is Zone 1 and Zone 4 will intermittently disappears from network at random intervals at least once a day.

All switches and routers are 100/10 - no Gigabit and all switches are cheap unmanaged ones.
Got most of them during sales and recycled from previous projects.

All rigs are wired. No rigs have wireless connection even though WIFI is available.

Any good simple troubleshooter app out there?                                                                                                      

[cavaliersrus]

commenting to continue to watch the dev

[yun9999]

@ melfice123 - The Solution I'm getting from Finksy, the PICO will be powered by the server PCI cable via barrel pigtail so both will always be powered on at the same time.  I'm just waiting for the unit.  It's going to be at least another 2-4 weeks before I get them.

@ Citronick - I had the same exact issue.  My network spans out multiple directions from my office to living room switch, to kids play room switch, to garage and then split again in the garage etc.  The more switches I added, the more likely 1 of them wont' play nice with each other.  They are all dumb switches so you can't hard set the port at 10/100 or gigabit.  My guess is 1 of them are dumb and don't auto negotiate well and creating a lot of unnecessary traffic.  Surprisingly I found my 16 and 24 ports caused more issues then my cheap 4 and 8 port switch. 

My problematic switches.

Netis - 24 ports
Linksys - 24 ports. 

This is when I upgraded to the following switches

NetGear ProSafe

The Cisco (OLD) I got for free also don't play well with some switches as it does need a Crossover cable to work with some switches.  I just use the NetGear to act as the crossover for it. I would recommend not using any Linksys / Dlink / Netis or non Business class switch as they work in the beginning but will have interaction issues or congestion issues later.  Reduce the number of switch and invest in Business class one, they seem to have bigger backplane and better traffic control even though both would be dumb not managed switch.  I like the NetGear Prosafe, no issues so far and it seems to play well with all the switches.

[bughatti]

OK ... first new topic from me...

I have been meaning to ask the team on their network infrastructure - I am no expert in this field but enough skills to supply Internet access to all my rigs.

I dont use any network analysis tool - not needed because broadband speed is quite good.

But.... I think I am beginning to have some network congestions in the farm.

I have divided my warehouse into 5 zones.

Broadband Internet
As provided by ISP
            I
            I
MASTER: TPlink 48-port Switch (Zone 2, Zone 3, test area)
            I
            I
---> Zone 1 - TPlink 16 port switch x 1 + TPlink 8 port switch x 2 (basic cascade) - 100% used
                                                                        
---> Zone 4 - TPlink 16 port switch x 1 - 90% used
                                                                        
---> Zone 5 - Cisco 24 port router x 2 (basic cascade) - 90% used

Now the issue is Zone 1 and Zone 4 will intermittently disappears from network at random intervals at least once a day.

All switches and routers are 100/10 - no Gigabit and all switches are cheap unmanaged ones.
Got most of them during sales and recycled from previous projects.

All rigs are wired. No rigs have wireless connection even though WIFI is available.

Any good simple troubleshooter app out there?                                                                                                      

With the ability that we all can buy expensive GPU's I would suggest looking at something like this

http://www.ebay.com/itm/Refurbished-Cisco-WS-C3560G-48TS-S-Ethernet-Switch-90-Day-Warranty-QTY-Available-/322205858210?hash=item4b04f739a2:g:ZZgAAOSweXhXl3uy

It is a good gigabit switch and cheap from ebay or amazon.  It is what I use.  I have a 16 port tp link off of it and I have 4 netgear 8 port switches off it.  I have had no issues for 3 to 4 years running.  I would never go more than 1 unmanaged switch deep off of a business managed switch.  You also want to make sure you are not creating any network loops.  Sometimes having that many switches it can get confusing what is plugged into what.

If you want to get deeper into troubleshooting I suggest find a free network monitoring tool, here are a few from a 2015 blog just doing a random search
https://techtalk.gfi.com/the-top-20-free-network-monitoring-and-analysis-tools-for-sys-admins/
I have used Zenoss and nagios before.  If you really wanted to go crazy you could setup something like rsyslog on each rig (Windows and Linux) and send data to a syslog server.  When you have an outage you could parse the logs for the timestamp and see what is causing the issue.

[melfice123]

@ melfice123 - The Solution I'm getting from Finksy, the PICO will be powered by the server PCI cable via barrel pigtail so both will always be powered on at the same time.  I'm just waiting for the unit.  It's going to be at least another 2-4 weeks before I get them.

@ Citronick - I had the same exact issue.  My network spans out multiple directions from my office to living room switch, to kids play room switch, to garage and then split again in the garage etc.  The more switches I added, the more likely 1 of them wont' play nice with each other.  They are all dumb switches so you can't hard set the port at 10/100 or gigabit.  My guess is 1 of them are dumb and don't auto negotiate well and creating a lot of unnecessary traffic.  Surprisingly I found my 16 and 24 ports caused more issues then my cheap 4 and 8 port switch. 

My problematic switches.

Netis - 24 ports
Linksys - 24 ports. 

This is when I upgraded to the following switches

NetGear ProSafe

The Cisco (OLD) I got for free also don't play well with some switches as it does need a Crossover cable to work with some switches.  I just use the NetGear to act as the crossover for it. I would recommend not using any Linksys / Dlink / Netis or non Business class switch as they work in the beginning but will have interaction issues or congestion issues later.  Reduce the number of switch and invest in Business class one, they seem to have bigger backplane and better traffic control even though both would be dumb not managed switch.  I like the NetGear Prosafe, no issues so far and it seems to play well with all the switches.

Thank you Yun, your project is awesome!

[yun9999]

@Bhughatti - Incredible find!!!  Wow, they're like $700 on Amazon but used for $40 Plus $7 shipping.  I just ordered 1.  Thanks!

https://www.amazon.com/gp/offer-listing/B0001LFZV2/ref=sr_1_2_olp?ie=UTF8&qid=1488170824&sr=8-2&keywords=Cisco+WS-C3560-48PS&condition=used

[bughatti]

@Bhughatti - Incredible find!!!  Wow, they're like $700 on Amazon but used for $40 Plus $7 shipping.  I just ordered 1.  Thanks!

https://www.amazon.com/gp/offer-listing/B0001LFZV2/ref=sr_1_2_olp?ie=UTF8&qid=1488170824&sr=8-2&keywords=Cisco+WS-C3560-48PS&condition=used

The one you just linked is the 10/100 version, still a good buy but it is not 1gb.  Think the one I linked was the 1gb.  Since the miners dont send alot of data the 10/100 will probably work fine though.  Glad you picked one up

[yun9999]

@kryptonite - Newegg.com, from other members, Ebay, Amazon, Jet.com.  You can always have friends or family order for you.  Amazon and Newegg QTY limits resets after 48hrs, rinse, repeat.

@Bhugatti - Yeah the one in my garage is 10/100.  It does the job well and I only need a spare.  But for anyone else, I think it's better to pay a little extra and get the 1gb version, hoping the newer one doesn't require crossover connections like my 10/100 version did.  These switch are SOLID and for $100, getting a used NetGear ProSafe or the Cisco is the best money you can spend.  Skip those little Personal switch, it's going to give you headache in the longer run and the cost saving is not much.