Data Center Mining Garage and Man Mining Cave ( PART 2 !!!!!!!!)

[CryptoWatcher420]

@ Citronick - Haven't had the chance put this Motherboard into much action.  It's a pain in the A$$.  You have to use another supported CPU to flash the board, and each time you add or remove GPUs, it turns the system on and off multiple times like it's about to break.  Support for BIOSTAR is one of the worst I've seen.  Contacted the Cali office and the guy doesn't even know how to reset CMOS, he told to send it in for RMA.   This board is just trying me nuts.  I got Windows to work with 5 not 6 cards.  Maybe I need to try another board, just no time since I have to flash it.  Simple OS doesn't like more than 1 GPU on it.  I think I might return 10 of them and just keep 5 for play.  Wasted way too much time on it.  

@ Marvell - Cool, so there is HOPE!  However if it can't also work with Simple, it's kind of a weak MB.

@ Whyme - Prove it.  Splitters don't always work and it's not as easy as you think.  Windows is definitely not the answer either.

@ CryptoWatcher420 - I did check out that link but it say max of 4 GPU per Cluster or is Max?

@ bughatti - Don't blame him but I would love to buy that solution to put in my Play build.  Got the 4KW I can easily power more than 16 GPUs.

its 4 gpus per cluster and theres 4 clusters per pci-e host card, 16 gpu per x1 slot, think a use for that 4k psu for just one motherboard with many many many gpu's lolz

[1714038300]

1714038300

[JaniHun123]

@yun9999 can you make a photo of the 42u rigs inside?
Also what kind of "chinese cases" are that? I want to build more rigs but i dont know how i can place them in a 42u cabinet.

Thanks

[citronick]

@ Citronick - Haven't had the chance put this Motherboard into much action.  It's a pain in the A$$.  You have to use another supported CPU to flash the board, and each time you add or remove GPUs, it turns the system on and off multiple times like it's about to break.  Support for BIOSTAR is one of the worst I've seen.  Contacted the Cali office and the guy doesn't even know how to reset CMOS, he told to send it in for RMA.   This board is just trying me nuts.  I got Windows to work with 5 not 6 cards.  Maybe I need to try another board, just no time since I have to flash it.  Simple OS doesn't like more than 1 GPU on it.  I think I might return 10 of them and just keep 5 for play.  Wasted way too much time on it.  

@ Marvell - Cool, so there is HOPE!  However if it can't also work with Simple, it's kind of a weak MB.

@ Whyme - Prove it.  Splitters don't always work and it's not as easy as you think.  Windows is definitely not the answer either.

@ CryptoWatcher420 - I did check out that link but it say max of 4 GPU per Cluster or is Max?

@ bughatti - Don't blame him but I would love to buy that solution to put in my Play build.  Got the 4KW I can easily power more than 16 GPUs.

I will leave this mobo alone then because I too lazy to spend a sauna afternoon in the warehouse.... furthermore SRR is kind off funky on the mobo - unlike the H81s.
Its running fine for my 4-riggers - may do some consolidation of the Nanos to 6 and 7s riggers, since using Optiminer v1.7 for ZEC saves me at least 100w per rig.

@WhyMe.... I have tried this experiment last year, not that straight forward to just arrange an array of these splitters and expect double the amount of GPUs. Windows maximum 7 GPU and 1 x splitter only. Anything more than that -- will need Win10-Ent hacks, special drivers and Linux. Would love to see how you do it though - I am assuming that you have actually done it

[dudutti]

My last build, I redone all my wooden rig

[yun9999]

@ duduti - Nice work.  Very clean.  Once you see my Pico setup, I think you will like it.  It makes cabling so much more simple and clean.  Just waiting for all the necessary parts.  Finksy got the 150 Picos in already.  I just need the custom cablings.

@ Citronick - Perhaps next month.  I have 4 more SRR coming.  I might use 1 for the custom design stuff including the Racing  MB.   Right now focusing on Custom Prototypes.  

@ Crypto - I don't think it's that straight forward bud.  I've used splitters before.  I highly doubt you can just connect a few of these into your MB and be able to recognize all

[dudutti]

@ duduti - Nice work.  Very clean.  Once you see my Pico setup, I think you will like it.  It makes cabling so much more simple and clean.  Just waiting for all the necessary parts.  Finksy got the 150 Picos in already.  I just need the custom cablings.

Yeah, i can't wait for your pico setup. This is really a good improvement

[yun9999]

@yun9999 can you make a photo of the 42u rigs inside?
Also what kind of "chinese cases" are that? I want to build more rigs but i dont know how i can place them in a 42u cabinet.

Thanks

There's no name for these Chinese Rigs.  They just call it Open Air Rig.  I was helping them sell these rigs for a while to get the goodies for the Community at cheap prices of $99-129 shipped depending on QTY  vs $179-249 retail price.  However the Vendor got greedy and started jacking up the prices and also made it harder for me to direct ship to people as they required a min of 11 vs 4.  It also added too much work as people would ask me so many technical and sales questions as if I was a reseller.  So I stop.  Was not worth the hassle for me but I did hook up over 60 different people prior to stopping.

It's a sexy design but you have to build and keep the fans on even though they are very weak, they do Act as support beams for the cheap plastic.  I won't name any names but I'm sure he's reading this................When you don't listen.................you end up with bendy Chinese RIGS!!!  Hahahaha.    So if you do decide to buy from Ebay, take my advice, keep the fans installed or even replace with stronger fans, but use it.  


Here's a picture of the rear of my Rack.  This is with the 4 ft long Dell PDU.  If it's APC, it's mounted horizontally and I put it on the top portion of the rack where I have 3.75U+ of empty space.

[Marvell1]

do your racks come with all the rails? whats your source for the racks

[mettalmag]

@ m1n1ngP4d4w4n -  Thanks...........just getting warm up bud.  Got more crazier / sexier prototypes coming out of my lab in a few weeks

Damn man, i would dream to have enough means to realize all the craziness going through my head, at least i can vicariously realise a portion of that through you ahah  

yun - this air flow strategy -- does it work?

This is respecting the optimised air flow scheme you can find in any good datacenters, the principles of cold / hot aisles, in the cold aisle in front of the racks, you bring AC air from the ground, then this air get sucked up from the front of the rack, pass through the servers, and get rejected in the back in the hot aisle, where it's sucked up from the ceiling ventilation system. The tarp at the top of the rack serve as a separation between both aisles for optimised air flow and balanced negative pressure

here is a schematic explaining this system.



Yun does it with normal air circulation instead of AC air, but the principle is the same.

This is some nice airflow setup, can you be more specific about pulling the AC air from the ground?

[deadsix]

This is some nice airflow setup, can you be more specific about pulling the AC air from the ground?

This is a typical DC airflow diagram bro, most if not all data centers are designed like this ...

[mettalmag]

This is some nice airflow setup, can you be more specific about pulling the AC air from the ground?

This is a typical DC airflow diagram bro, most if not all data centers are designed like this ...

I mean what are the most cost effective ways of data center air cooling systems?

[m1n1ngP4d4w4n]

This is some nice airflow setup, can you be more specific about pulling the AC air from the ground?

This is a typical DC airflow diagram bro, most if not all data centers are designed like this ...

I mean what are the most cost effective ways of data center air cooling systems?

The air is pulsed from the ground through grilled plates that replace the normal floor plates and pulled by negative pressure, made from the racks intakes and the hot aisle separation/ceiling air-pull, this is the most efficient design as air cooling goes, and the sad part is that you have plenty of datacenters that aren't respecting it fully, even though it has been invented for nearly 25+ years now. You also have some system variant, but the principles are the same.

[mettalmag]

This is some nice airflow setup, can you be more specific about pulling the AC air from the ground?

This is a typical DC airflow diagram bro, most if not all data centers are designed like this ...

I mean what are the most cost effective ways of data center air cooling systems?

The air is pulsed from the ground through grilled plates that replace the normal floor plates and pulled by negative pressure, made from the racks intakes and the hot aisle separation/ceiling air-pull, this is the most efficient design as air cooling goes, and the sad part is that you have plenty of datacenters that aren't respecting it fully, even though it has been invented for nearly 25+ years now. You also have some system variant, but the principles are the same.

Now I understand what you were talking about, the picture says it all, but were they take the cool air from? I heard that they do it from deep underground, I'm not sure is this true or not

[yun9999]

@ Marvell1 - I got it from liquidation place.  You just have to search local listings like Craigslist, you'll be surprise what you can find.   Some are Gems at cheap prices and may come with rails.  Rails are not cheap x 9 or 10.  They can cost more than the racks!  

@ m1n1ngP4d4w4n - You may confuse him more.  Mining is not the same as Data Center!  But I'll be very happy if the day comes where it pays well enough to be used with AC cooling.

@ Mettalmag - You will never find a profitable miner who is using AC to cool his gears unless he's operating under 2 cents per kw.  But even then the cost for the kind of AC you would need would cost you a fortune. Just to give you an idea 10 rigs at 1200W each on the average.  = 12KW.  The general rule of cooling heat is 3.4 btu per watt.  It's a very general rule and depends on your layout and how sealed the room is, etc.  Noticed key word is sealed.  So just for  10 rigs you need a 4 TON AC!  That's something you cool your entire 2K SF house with.  No miner is going to be putting many 4 TON ACs to keep their rig cool and be profitable.  

Data center is different as they operates as a COST, not base on profits from the mining or gear.  AC is also designed as a CLOSE system, it won't work with a Open Air flow design where you're venting out the heat.  Hence why it can't keep up if you have a few windows open in the office on hot days.  The key to battling heat for Mining is not with AC, it's by how to get the heat out quickly enough so it doesn't accumulate so you can be as close to Ambient temp as possible.  Our GPUs can run under 105F Ambients without issues.  Keep in mind it's not exactly that cool inside a computer case.   The only thing you can add to a good open air system regardless of server cases or regular racks is a Swamp cooler but if your flow and design is good enough, that shouldn't be required.

[mettalmag]

@ Marvell1 - I got it from liquidation place.  You just have to search local listings like Craigslist, you'll be surprise what you can find.   Some are Gems at cheap prices and may come with rails.  Rails are not cheap x 9 or 10.  They can cost more than the racks! 

@ m1n1ngP4d4w4n - You may confuse him more.  Mining is not the same as Data Center!  But I'll be very happy if the day comes where it pays well enough to be used with AC cooling.

@ Mettalmag - You will never find a profitable miner who is using AC to cool his gears unless he's operating under 2 cents per kw.  But even then the cost for the kind of AC you would need would cost you a fortune. Just to give you an idea 10 rigs at 1200W each on the average.  = 12KW.  The general rule of cooling heat is 3.4 btu per watt.  It's a very general rule and depends on your layout and how sealed the room is, etc.  Noticed key word is sealed.  So just for  10 rigs you need a 4 TON AC!  That's something you cool your entire 2K SF house with.  No miner is going to be putting many 4 TON ACs to keep their rig cool and be profitable.  

Data center is different as they operates as a COST, not base on profits from the mining or gear.  AC is also designed as a CLOSE system, it won't work with a Open Air flow design where you're venting out the heat.  The key to battling heat for Mining is not with AC, it's by how to get the heat out quickly enough so it doesn't accumulate so you can be as close to Ambient temp as possible.  Our GPUs can run under 105F Ambients without issues.  Keep in mind it's not exactly that cool inside a computer case.   The only thing you can add to a good open air system regardless of server cases or regular racks is a Swamp cooler but if your flow and design is good enough, that shouldn't be required.

ATM we are using only air flow system and GPUs are feeling quite cool, but I'm not sure what will happen in summer when outside temp reaches 30+C.

[yun9999]

Only 30C?  I live in Texas where it can get to 40C.  The key is your exhaust.  You need to be able to pull the heat out well especially since your layout is so massive and you're using Open air frames which is spreading the heat everywhere.  This just means you have a higher total CFM area to exhaust out vs funneling where you're only covering a small portion.  Exhaust via ceiling or up top is efficient for Open air, similar to Genesis mining.  

[mettalmag]

Only 30C?  I live in Texas where it can get to 40C.  The key is your exhaust.  You need to be able to pull the heat out well especially since your layout is so massive and you're using Open air frames which is spreading the heat everywhere.  This just means you have a higher total CFM area to exhaust out vs funneling where you're only covering a small portion.  Exhaust via ceiling or up top is efficient for Open air, similar to Genesis mining.  

I want to cover the ceiling with exhaust tubes, because upper miners get higher temps because of the set up style itself, but the thing I want to give up on this and build another place like data center, that's why I'm asking for cost effective solution for getting cool air, the best part about this is the proce of kw per hour which in my case is roughly 3 Cents.

[yslyung]

I will show step by step on that.  Here is short idea.

OLD WAY
Baby ATX PSU (Crap efficiency and reliability)  powers risers and MB.  Bulky, cheap but not cheap enough and if you go really cheap (Below $45-50), it's more headache then it's worth.  More wires and a big ATX PSU behind your rig blocking flow and your acccessibility to the GPUs.  

PICO OLD Way
Expensive and the built in CPU connector is too short to reach our mining Rig CPU connection.  You need to use bulky AC adapter which makes it more expensive and more connections and things to plug in and hide. $40-60 each!  Too much money

*********PICO NEW WAY***************
Powered completely by Server PSU via 6 pin direct to soldered PICO connection or via 2.5MM Barrel connector.  It would also split to power the CPU connections.  The PICO PSU comes with at least 1 molex and 1 SATA connection so you can power SSD, etc.  98% of your power will come from the SERVER PSU.  Even if you got a crappy Pico, if it dies.  It's a cheap $20-25 replacement.  The PICO PSU is tiny and the size of the motherboard connector head.  You won't see it.  The server PSU cable can go up to 48" easily.  Using PCI powered risers and splitters, you reduce all the SATA, MOLEX, MB cable clutters.  Also no ATX PSU in the rear of the rig.  So better airflow.  I will have lots of examples later but look at my previous examples on the Ebay PICO I bought.

just a prototype i made with some free time to work on. pcie cable is a 16 awg cable. cpu power is original 18 awg, long enough for most mobo even on the msi gaming 5's. sata cables are a little smaller. sorry for the late reply, tied up with building rigs & was feeling a bit unwell. chk pm.



hope you get the ones from finsky soon and test them out. so far on my side they have been working well even with the stock cables.

speaking of more gpu's per mobo is good & can lower operating costs + saving up some space but if 1 or more gpu decides to fail or acts up, the entire system will be down. imo best is still a 6 gpu setup per rig & per psu per rig. more gpu per rig sometimes causes more issues instead of a better solution.

issues that i've experienced are such as rig design (need a bigger rig), weight, heat, more clutter of cables just to name a few but of course it does depends on each individuals setup & preferences.

i like the 4kw setup, i still have the very early version of breakout boards from j4bberwock for the ibm 200 bb's about 3 years ago. they still work very well.



they are reliable psu's but what if (it happened to me before during asic days, using load sharing) either 1 of the pair fails, then affected rigs will also be affected & won't have enough power. maybe there's a solution for that on gpu mining ?

that's 1 of the reason i decided to use a dedicated PSU for each rig, the pico idea (been using it for a little over 6 months or so including some tests + r&d) for a more efficient in terms of psu efficiency, cost & reliability.

still trying to improve by making more efficient rigs in terms of power, cost, reliability & minimal downtime.

[shield132]

This is some nice airflow setup, can you be more specific about pulling the AC air from the ground?

This is a typical DC airflow diagram bro, most if not all data centers are designed like this ...

I mean what are the most cost effective ways of data center air cooling systems?

The air is pulsed from the ground through grilled plates that replace the normal floor plates and pulled by negative pressure, made from the racks intakes and the hot aisle separation/ceiling air-pull, this is the most efficient design as air cooling goes, and the sad part is that you have plenty of datacenters that aren't respecting it fully, even though it has been invented for nearly 25+ years now. You also have some system variant, but the principles are the same.

Wow, that's very great, I haven't seen such mining datacenter ever because mostly everyone use airflow system. I guess it won't be your only cooling system and don't you think that that will be smally waste of money?
As I know Bitfury is using air flow system as I know because mettalmag has mining center here, near bitfury.

[yun9999]

@metalmag = For Mining, if you need to use AC, you're not doing it right.  Especially if your region upper temperature is that cool.  Again I'm in the 40C in the summer and others are even higher in Arizona and we can do it fine.  Even if you have free Electricity the cost of buying, installing, maintenance, etc is very high.  Let's do some quick math.

Assuming Free electricity.

10 Rigs = 4 TON AC @ $4-7K each depending brand and SEER levels installed.
100 Rigs = 4 x 4 Ton AC  $40-70K.

1 x 4 Ton uses at peak up to 60AMP.  So you're wasting 600AMP just for AC alone!  Now you have to factor in that you will need around 600-800AMP more for your 100 Rigs so the total is up to 1400AMP.   Even Commercial space the transformer may not be powerful enough to handle 1400AMP as you're sharing with other tenants.  You can request for more power if it's available but they will charge you for it.  Now this isn't even taking the consideration of the 80% rule which means you really need at least 1700AMP.  This is taking you into Industrial spaces which requires very large warehouses.   So you're talking about something you can achieve for less than $40K which includes 600AMP of power to drive your rigs into something well north over over $200K and require special zoning to accommodate.  If you design your environment right, you SHOULD NOT need AC.  Mining is a Mathematical game.  Even at 3cents / kw, you won't even be able to compete against someone with 10 cents KW even you have to use AC.  I would invest that money into more RIGs and a better layout / exhaust strategy. If you have that kind of budget, it shouldn't be hard.  To win in this game you must keep your cost low and invest the rest into HASH.

@yslylung - Yes Finksy just got his 150 Picos in.  Your price is very competitive for a custom solution.  Finksy's cost is actually still lower so there's a small margin for reseller on top but his goal is to keep the cost as low as possible for the community.  He's sending me 20 units to test for my different Prototypes designs.  I have 10 of the HP 1200W including the V6 breakout boards plus more 4KW to play.  There's never a 1 size fit all solutions and always pros and cons with every options.  The HP solution is best if you can get them in bulk for $20 or less each as there's breakout boards $25-30, Pico $30 and cabling cost $18-25.  That brings you to about $105 for the normal consumer.  People will always argue that you can buy a ATX for almost the same price with better future resales, yadda yadda.  Maybe you can during hot sales but the cabling and configuration will not be as sexy.  The issue with this solution is that many people in America use 120V which this PSU can  only deliver 900W vs the world on 240 which this PSU can deliver 1200W. I don't have this issue so I'm not concern.    Also the HP 1200W have only 1 TINY fan, I'm not sure how well it can handle the heat but man does that fan make a LOUD pitch noise!!  This thing is definitely not something I want to run inside my house.

For large miners the 4KW is important as they don't want to rearrange rigs after it's installed in the RACK.  They also want be future proof and plug things in and never have to really think about power usage later one.  Heck most people didn't even know that ZEC and ETH first version and current version consumes WAY MORE power.  So if they're sizing based on original estimates, they're screwed.  However as you said, if you have a PSU go down, you're not just affecting 1 rig but multiple.  That's the downside, the upside is that each breakout board and PSU is so cheap, there's no reason why you shouldn't have at least 1 spare bundle if you're a large miner.  Swapping is super fast and simple.  Efficiency for PSU is also better when you're using less than 70% of load and that will be harder to do on the HP.   I like bother solutions and at the end of the day, paired with PCI Risers and Picos, I'm really liking the Server PSU setup so so much better.  No more clutters.

I have design prototypes that will take advantages of both the PROs on each solution.  At the end of the day, if you're a large farmer, you will have a variety.  There's no solution that can meet every need.   But I do think that once people see more of the cool server PSU and PICO designs now that we PCI Powered risers, they will LOVE TO GET their hands on these PICOs.