Data Center Mining Garage and Man Mining Cave ( PART 2 !!!!!!!!)

[groovy1962]

@yun999 -- I'm curious about how you use the 4k PSU bundles with open rigs.  It seems one PSU would power 4 or more rigs (at 6 cards each).  So when you have to take one rig down, what do you do?  Disconnect the 9 power cables going to the cards and risers?  Or are you rigs set up so you do maintenance in place?  I have to move mine to replace a card.  Right now the ATX PSU goes with it, so to do maintenance all I have to do is disconnect the 120v AC and the network cable... and its a unit.

I am liking the 4k PSU bundle, just not sure how to deal with it.  Or maybe I should build a massive open rig with multiple motherboards and GPUs and treat it all as one unit. LOL!.

Nobody answered my question.

With these 4k PSU setups, its really nice to support so many GPUS, but it looks like you would have to disconnect a large number of cables to take a rig out of service.  I'm thinking about building a custom setup with 2 Mother boards, 16GPUs, and one FINKSY 2880 PSU all in one custom case.  This way the connection to that case is just power and network.

[1713978589]

1713978589

[1713978589]

1713978589

[1713978589]

1713978589

[deadsix]

Nobody answered my question.
With these 4k PSU setups, its really nice to support so many GPUS, but it looks like you would have to disconnect a large number of cables to take a rig out of service.  I'm thinking about building a custom setup with 2 Mother boards, 16GPUs, and one FINKSY 2880 PSU all in one custom case.  This way the connection to that case is just power and network.

I did answer your question, did you miss it? Pasting it here just for reference.

@yun999 -- I'm curious about how you use the 4k PSU bundles with open rigs.  It seems one PSU would power 4 or more rigs (at 6 cards each).  So when you have to take one rig down, what do you do?  Disconnect the 9 power cables going to the cards and risers?  Or are you rigs set up so you do maintenance in place?  I have to move mine to replace a card.  Right now the ATX PSU goes with it, so to do maintenance all I have to do is disconnect the 120v AC and the network cable... and its a unit.

I am liking the 4k PSU bundle, just not sure how to deal with it.  Or maybe I should build a massive open rig with multiple motherboards and GPUs and treat it all as one unit. LOL!.

While the 4k PSU delivers most of the power for the system (namely, all the cards) each system still needs a mini PSU (basically just a low capacity regular PSU like 450W) which is what you use to turn on/off each system.
The alternative, and what everyone is switching to, is a nanoPSU which powers off of the 4K PSU and has a small on/off switch.

Anyways, no harm in building a custom setup, don't forget to share pics here. Best of Luck.

[acostaja]

I was working on a similar solution as stefanop (we are far away to have it ready) our setup is a PCIx1 child board that plugged to the motherboard and linked with eSata Express cables to an board using moreless the same chip in 1-3x switches ( PI7C9X2G404SL ), with 3 PCIe x1 ports with open ends so the GPU are plugged directly to the PCIe Swith board 3 GPUs (to keep the board practical for production, each board is about 9" and provides about 1 inch per gpu clearance for cooling, is powered by the eSata Express cable draws no power from the motherboard, and we hope it could be stable enoug. my plan is to plug 6 of these on Biostar Motherboards (assuming we solve the bios issue), each 3 ports board inlcuding cable has a BOM cost about 45$ (not including manufacturing) so out target numbers are close to that from stefano, of course I follow stefano development singe he has solved the gruesome dirty development we just started, in case their boards are good enough I'll not mess with my design and buy/commission from stefano.

I'd like to know about stefano approach to solve the link issue (custom flat ribbon cable, eSata Express, U.2 even multiple USB-C are valid as long has enough lines to full link the PCIe port (18-20 lines total, but it could be less w/o redundant lines, but not so few as with USB3 10 lines).

[Spotswood]

Setup is built around a 8 GPUs per daughterboard(the actual board I'm building). Two of my boards are connected to the mainboard (~$150). My current estimates are around 150 per board..so 300+100 = $450...I guess my estimate price is a bit too low for you then  huh The current board I'm testing can physically connect a total of 18 GPUs since it has two extra PCIE slots...youll will just need risers for that last two (there is a possibility a future BIOS update I could make it to post with 3 or even 4 of my boards for a total of 32 GPUs, but I'm just working on 16 for now...plus I would need to work closely with miner devs for anything more than 16...I have slowly hinted them improvements which they have already implemented for multi GPU setups in the past year).
Also the board has status LEDs per GPU lane...if a GPU goes down you'll know right away which one it is, and yes if one GPU crashes the board will still work. This is no different than any other mutigpu setup, other than it having just more GPU lanes. Your doing something wrong if one GPU crash brings down your whole system.
Keep in mind that this price is mostly dependent on the quantities I can order for parts on the first batch. Obviously more boards = less price per board. I am also pretty much charging the cost of components + a small profit margin for me. The design + dev + testing + prototyping etc is all done by me so thats why I can price it so low.

Hey Man, could you share some pics, even a sketchup that would tell us about the size/layout of the daughter boards and the motherboards? I'm asking because it would give us a head start on building cases/frames for these boards.

I also asked for a simple mechanical drawing of the board.  Just the overall dimensions and mounting hole information would be a good start.   

[yslyung]

sorry guys, been really busy & some unscheduled urgent appointments showed up. i'm around though. those who pm'ed me, i did reply & will deal with shipping by this coming friday. apologies.

OP i solved the HighPitch  solution for you hahaha ... psu is cooler, less load + better efficiency too.



single 1200w platinum server psu is not enough for your power hungry multiple gpu setup? well the capt got a dual solution! *ordered a small batch of customized dual c13 cables*



running your miner in a hot environment? hot psu? fans are screeching? no problem cooling solution is here too...**i'm running & testing 2 sets now without the extra fan, it's cool & quiet



some stock arrived, SOLD OUT ! 100 risers, gonna order another batch, this time round 50 shorter usb cables & another 50 regular 60cm ones. rigs will me much cleaner !



after some drilling then continue with lots of screwing

more work to be done, enjoy folks.

as for multiple gpus setup more than the regular 7, i won't really recommend to most people due to :

1) you will need NOT only custom BIOS but probably some mods in WIN too but for more than 8 it has to be linux for sure

2) for the 16 or even 32 gpu setup, it's amazing development but still no pix & it's a meh for me tbh.
YES there are some indicator on the boards if a GPU hangs & the mining continues, good job, -wd 0 can do that as of now too but nevertheless the operator still have to RESTART the mining software in order to get all gpus up & running = longer startup time for DAG generation = more gpus will be down = more downtime. what we actually want to know is more towards which gpu is down & which is the one that needs to be adjusted ie in WattTool or in any other software or parameters.

3) Extensions are good for up to 2 gpu or probably even 3 on a single extension BUT either BIOS or WIN limit will hit ya (WIN will be something like NOT enough resources)

4) then it could be fleet of psu's that an operator will need to run & or to figure out the cables & wiring mess, sorry folks & yes i am NOT a fan of messy cables.

5) the boards will be quite BIG in size to fit all gpus = a new design for a rig/rack to fit all of them. say to fit all 16 gpus side by side with a decent amount of spacing between them for adequate cooling the width has to be a minimum 130cm or more.

6) i'd really like the no risers idea & the capability to run many gpus at 1 time but as the old saying goes, do not keep all your eggs in 1 basket.

KISS is sometimes the BEST solution to most problems.

OP i did NOT buy all the mobos HAHAHA.

**again, i DO NOT intend to be the "bad guy" here with regards to the 16 gpu setup but it's just my personal opinions i hope it's being viewed & taken as a constructive criticism**

  

[yun9999]

@ yslyung - YOU DA MAN.  Now P.M me on where to buy or how much for those standing breakout boards.  Since my lay flat one definitely won't work with fan flow.  I still have 10 HP PSU I want to use.   The breakout board will work with the HP 1500W correct?  Is there specific version of HP 1500 revs only?  Great job as usual.  

[jstefanop]

I'd like to know about stefano approach to solve the link issue (custom flat ribbon cable, eSata Express, U.2 even multiple USB-C are valid as long has enough lines to full link the PCIe port (18-20 lines total, but it could be less w/o redundant lines, but not so few as with USB3 10 lines).

Custom designed cable, plus the chipset I am using allows for custom PCIe lane tuning, which is the reason why I am able to have this rock solid. Gotta open them eyes (anyone thats done high speed signal engineering will know what I mean).

For people asking about dimensions I will have all that info in my announcement thread soon. But its designed to fit in a rackmount demensions.

[PeaMine]

Are there any plug and play type cases and risers?
You provide the computer and cards?
I have a mix of 3 16x and 3 1x pcie, wanting to put 6 cards raised in an open or closed case.

[Pennywis3]

Here is some info on doing multi GPU (8+) setups.  It has some great insight into what all is needed.  gstoner on the link is a Senior Director at AMD and is full of knowledge about this type of set up.

https://community.amd.com/thread/197524

this thread was started by jstefanop himself

Nice link Keith, looks like AMD has a working MB with 16 S9150 made by Cirascale. But damn, those GPUs seem too close.

Hi Hawkfish 

I see that your USB riser on amazon are sold out. Will you be geting any more? I hope that shiping to Europe is not a problem.
I heard you guys talk about risers, that have a power converstion from 12V to 5V so that it prevents burning cables? Are yours with the conversion?
How can we spot risers that have that, could you point them out on Amazon for me please.
I will be building a few rigs in the future and would realy like some advice about risers, so i can get some quality ones, as i don't know how to spot the good ones
Example:
https://www.amazon.de/PCI-E-Powered-3-0-Verl%C3%A4ngerung-Kabel-GPU-Extender/dp/B01N3UVJHM/ref=sr_1_6?ie=UTF8&qid=1490771302&sr=8-6&keywords=USB+riser

https://www.amazon.de/Demiawaking-Express-Extender-Adapterkabel-Stromkabel/dp/B01N6JZ3O4/ref=sr_1_16?ie=UTF8&qid=1490771180&sr=8-16&keywords=USB+riser

Tnx in advance.

[Marvell1]

Here is some info on doing multi GPU (8+) setups.  It has some great insight into what all is needed.  gstoner on the link is a Senior Director at AMD and is full of knowledge about this type of set up.

https://community.amd.com/thread/197524

this thread was started by jstefanop himself

Nice link Keith, looks like AMD has a working MB with 16 S9150 made by Cirascale. But damn, those GPUs seem too close.

Hi Hawkfish 

I see that your USB riser on amazon are sold out. Will you be geting any more? I hope that shiping to Europe is not a problem.
I heard you guys talk about risers, that have a power converstion from 12V to 5V so that it prevents burning cables? Are yours with the conversion?
How can we spot risers that have that, could you point them out on Amazon for me please.
I will be building a few rigs in the future and would realy like some advice about risers, so i can get some quality ones, as i don't know how to spot the good ones
Example:
https://www.amazon.de/PCI-E-Powered-3-0-Verl%C3%A4ngerung-Kabel-GPU-Extender/dp/B01N3UVJHM/ref=sr_1_6?ie=UTF8&qid=1490771302&sr=8-6&keywords=USB+riser

https://www.amazon.de/Demiawaking-Express-Extender-Adapterkabel-Stromkabel/dp/B01N6JZ3O4/ref=sr_1_16?ie=UTF8&qid=1490771180&sr=8-16&keywords=USB+riser

Tnx in advance.

can i get a link for the picos you sell as well as your amazon risers too hawkfish ?

[deadsix]

I Believe Finksey was going to upgrade his 4k and 2880 PSU bundles with picoPSU's soon.

[hawkfish007]

Here is some info on doing multi GPU (8+) setups.  It has some great insight into what all is needed.  gstoner on the link is a Senior Director at AMD and is full of knowledge about this type of set up.

https://community.amd.com/thread/197524

this thread was started by jstefanop himself

Nice link Keith, looks like AMD has a working MB with 16 S9150 made by Cirascale. But damn, those GPUs seem too close.

Hi Hawkfish 

I see that your USB riser on amazon are sold out. Will you be geting any more? I hope that shiping to Europe is not a problem.
I heard you guys talk about risers, that have a power converstion from 12V to 5V so that it prevents burning cables? Are yours with the conversion?
How can we spot risers that have that, could you point them out on Amazon for me please.
I will be building a few rigs in the future and would realy like some advice about risers, so i can get some quality ones, as i don't know how to spot the good ones
Example:
https://www.amazon.de/PCI-E-Powered-3-0-Verl%C3%A4ngerung-Kabel-GPU-Extender/dp/B01N3UVJHM/ref=sr_1_6?ie=UTF8&qid=1490771302&sr=8-6&keywords=USB+riser

https://www.amazon.de/Demiawaking-Express-Extender-Adapterkabel-Stromkabel/dp/B01N6JZ3O4/ref=sr_1_16?ie=UTF8&qid=1490771180&sr=8-16&keywords=USB+riser

Tnx in advance.

Hi,

Sorry about that, 6-pin risers should be available anytime now. They were delivered to Amazon yesterday and I am waiting on them to make them available. Molex to Sata should be available next Monday on Amazon, I will be shipping a bunch to Amazon tomorrow. However, both of these risers are available on eBay (search my seller ID hawkfish007). I am also stocking 6-pin to  2x 6+2 pin splitters and 133 CFM 120x38 cm double ball bearing fans.

Thanks.

[matmator]

ok last teaser for you guys I promise  

!!!! You can explain how you do to have as much gpu?

You have a picture of your installation because 15 gpu I am curious to see how you do.

[CryptoWatcher420]

ok last teaser for you guys I promise  

!!!! You can explain how you do to have as much gpu?

You have a picture of your installation because 15 gpu I am curious to see how you do.

theres nothing to explain as of yet, thats a special setup

[philipma1957]

A motherboard config that is riser free and have a GPU Fault LED / sensor?  Plus it's competitively priced against normal config?  Hmm, now I'm interested again.  Haaha

Okay   @ yun9999  waiting for   the srr when do they come in.

@ yun9999

I pulled trigger on this


http://www.ebay.com/itm/141720603862?


looking forward to using your gear with it.

[yun9999]

@ Phillip - Seems you're attracted to high pitch PSU noises.  Haha.  YOu will need a breakout board from YSYLUNG to fix that issue.  Don't get the ebay one! 

[CryptoWatcher420]

@ Phillip - Seems you're attracted to high pitch PSU noises.  Haha.  YOu will need a breakout board from YSYLUNG to fix that issue.  Don't get the ebay one! 

30 dollars for the breakout board is just a tad over what I would pay for one, then you gotta add shipping from where he is located, if he managed to bring costs down to 20 dollars plus shipping, I would buy from him

@yun9999 will he be shipping any to you for US shipping?

[yun9999]

I can offer the free warehousing  and perhaps ship it to 1 person who will handle the group order or something but I don't have the cycles to be mailing things out to each individuals.  I don't even have time to finish building all my rigs here which is an opportunity lost of $15-20 / rig.   It may make more sense for him to partner up with Hawkfish or someone similar for USA distribution as Hawkfish have many sales channel, Amazon, Ebay and direct.  Yeah at $30 / breakout board, the total cost after PSU, Pico, and cables and the value is diminished a bit compared to ATX with the better resell value.  I already have a bunch of these HP PSU and Picos so for me, doesn't really matter.  I just want to use them but I don't want any loud noise.  Those Antminer days are long gone for me.

[miner417]

Hey guys, after reading through the entire thread, and been monitoring it for a while, you've made me want to take the jump into a real first rig. I currently run a 3GPU rig in my main computer, just using a whole regular computer setup to power/run it, with an added box fan to cool since it's in a semi closed (1 side panel off) case. I want to make the jump into a full 6 GPU rig, and was wondering if anyone had any suggestions on what to buy. I know GPU's well, so that's not an issue. I searched for all the boards that are in this initial post, and none of them are available for any decent prices anymore. So I'm stuck right at the gate with what motherboard to go with. It would be nice to get some information on a mobo and risers to use, plus a power supply. I see this talk about using a server supply and a pico, but honestly I'm lost with that. My thought would just be a 1200W EVGA platinum for PSU.

So I guess my question is, what mobo + risers do you suggest for a first full rig, and is a 1200W EVGA platinum a good PSU to go with it? I plan to use 6x RX470's for it, with undervolting. Currently I have 2 RX470's both running at about 110W each.

I would also like to build my own open air case for it, does anyone have any suggestions for designs or where to look to go find some?

I love the seeing everyone's builds, and hope I can get one to show off here as well. Thanks for any help guys.

Edit: This is my current board in my regular PC, from my understanding, this thing can run 5 cards with the expresses and risers?
https://www.newegg.com/Product/Product.aspx?Item=N82E16813128841&nm_mc=AFC-C8Junction&cm_mmc=AFC-C8Junction-PCPartPicker,%20LLC-_-na-_-na-_-na&cm_sp=&AID=10446076&PID=3938566&SID=

1 more Edit: Using that same thought process, this board could run 5 cards with risers?
https://www.newegg.com/Product/Product.aspx?Item=N82E16813157682&nm_mc=AFC-C8Junction&cm_mmc=AFC-C8Junction-PCPartPicker,%20LLC-_-na-_-na-_-na&cm_sp=&AID=10446076&PID=3938566&SID=

[philipma1957]

@ Phillip - Seems you're attracted to high pitch PSU noises.  Haha.  YOu will need a breakout board from YSYLUNG to fix that issue.  Don't get the ebay one!  

you have confused me  quite a bit
I thought you were shipping a package of gear to me.

no worries if you changed your mind.




@ yslyung
  

let me know when you get the srr's as I am going to need a setup for 8 pc's



edit I hate the spring  I always mirror image shit.


@ yun9999   let me know when you   have the   srr for me

@ yslyung    let me know when you ship the  gear to me-------   ie breakout board wires and pico