19 GPU in one Motherboard

[creepas]

Hello,

i just wanted to say, that we managed to get running with no big problems 19 GPUs in one Motherboard.

MOBO: Asus B250 Mining Expert

GPUs: 13x ASUS STRIX GTX 1060 6 GB + 6x ASUS MINING P106

MAIN PSU: 2400W
Secondary PSU (for powering pcie extenders and mobo) 750W
We have also 3th PSU in video, but in reality you need only main and secondary PSU.
16GB RAM

NO HDD just one 32GB FAST USB. SYSTEM [OS] nvOC easy-to-use Linux Nvidia Mining v0019

In windows we were able to get all cards do device manager, but mining was not running

https://drive.google.com/drive/u/1/folders/0B4y5D_zzH1oeLWpIcGtQOXI1MUU

will post tomorow more videos and detailed analysis of mobo with manual how to do that.

Best regards

cReepas

(Invictus Mining Team)

[1714121088]

1714121088

[1714121088]

1714121088

[1714121088]

1714121088

[fanatic26]

One thing you might want to be careful with. If you are powering 19 risers with that one PSU those cards only have to draw 35 watts each before you are really stressing the PSU. Thats 650w+ in just risers, plus the mobo itself.

[bathrobehero]

Sounds great until you have to troubleshoot frequent crashes due to a failure.

You can and will have cards that will behave badly and each riser has 3 parts (the riser itself, the cable and the small PCB going into the PCIE slot) so that's already 76 points of failure making it a PITA to troubleshoot.

And multiple PSUs can also make things unstable/difficult and you need a very good and expensive CPU for some miners and tons of memory and (preferably SSD) for I'm guessing 64GB pagefile so more GPU density isn't that much cheaper.

If it works though, it's nice but I much more prefer simple, 4-6 card rigs with a single PSU with a bunch of wallets I solomine to or stake distributed over them.

[fanatic26]

64GB pagefile

the pagefile is a windows thing....and you cant run more than 8 of one brand of card in windows so literally NOONE will be tryin to run this board on windows with 19 cards

[bathrobehero]

64GB pagefile

the pagefile is a windows thing....and you cant run more than 8 of one brand of card in windows so literally NOONE will be tryin to run this board on windows with 19 cards

Good to know. I thought it was a universal OS requirement to have as much RAM+pagefile as much is required by all the GPU memory that's assigned to be used.

Though thinking about it, it shouldn't really happen under windows either (meaning it's a silly bug) because that space is never used, just needs to be there for a second.

Thinking about it even further, initializing cards one after the other should propably get rid of the issue though I'm not sure, just spitting balls.

[MA3A]

1060 6gb  version eats about 50-60W via PCIE slot, when fully loaded ; having 19 cards @50-60W each brings to about 1100-1200W... risk of witnessing fireworks is quite high, if cards will be loaded beyond 50%.   I must admit i did not see video, but specs a bit off... its cool though if you got it to work

[Undefined31415]

Most Linux distros allow you to set a custom swap partition size, but those do have some fundamental differences from the page file on Windows.

[mikespax]

when/where can I buy this mobo?

[Metroid]

when/where can I buy this mobo?

That is a good question. I guess if the op has then it means soon will be on sale.

[creepas]

when/where can I buy this mobo?

That is a good question. I guess if the op has then it means soon will be on sale.

We can buy officially next week in stores.

cReepas

[darkshad]

very cool.

I would be interested in seeing if 19x 1080tis will work in Linux using something like nvOC or SimpleMining distro.

please keep me posted on when/if you have a chance to do something like this and when the MB are for sale.  Would at least like to try one out and see how frustrating it is to try to get 19x cards to work/stable.

[creepas]

very cool.

I would be interested in seeing if 19x 1080tis will work in Linux using something like nvOC or SimpleMining distro.

please keep me posted on when/if you have a chance to do something like this and when the MB are for sale.  Would at least like to try one out and see how frustrating it is to try to get 19x cards to work/stable.

Its actually easy in linux. In Windows it seems its not possible.

19x 1080Ti is hard to get.

I will post later manual how to make it run.

cReepas

(Invictus Mining Team)

[dstm]

Might you do a quick test and check how my miner performs on this system? Whether there are any issues. The miner is designed specifically for linux, there is no windows version yet. I've designed the miner such that it should be able to drive 100+ GPUs.

[creepas]

Might you do a quick test and check how my miner performs on this system? Whether there are any issues. The miner is designed specifically for linux, there is no windows version yet. I've designed the miner such that it should be able to drive 100+ GPUs.

Sure just give us the link etc.

[dstm]

Might you do a quick test and check how my miner performs on this system? Whether there are any issues. The miner is designed specifically for linux, there is no windows version yet. I've designed the miner such that it should be able to drive 100+ GPUs.

Sure just give us the link etc.

It's a miner for ZCash

https://bitcointalk.org/index.php?topic=2021765.0

[Metroid]

I wonder how power works on this motherboard, if you mix 2 different power sources will fry the motherboard?

[creepas]

Hello,

im adding video: https://www.youtube.com/watch?v=DjFP1BfN5Xs&t=232s

Best regards

cReepas

Invictus Mining Team

[Metroid]

Hello,

im adding video: https://www.youtube.com/watch?v=DjFP1BfN5Xs&t=232s

Good work, too many things to test it out with this motherboard. I hope it goes on sale soon. Asus said AMD will create a driver that supports all 19 cards by the end of this year. Let's hope they deliver it.

[arielbit]

what we need is not a motherboard that have lots and lots of pcie...

what we need is an interface that can manage many cards where we can plug and play cards without turning off the system..where cards that have a problem would hang or stop separately without affecting the whole system so other cards will keep on mining.

with that kind of interface we can use almost any board out there..that interface should have a software management and hardware switches(like for disabling power) for the plug and play feature.

[creepas]

what we need is not a motherboard that have lots and lots of pcie...

what we need is an interface that can manage many cards where we can plug and play cards without turning off the system..where cards that have a problem would hang or stop separately without affecting the whole system so other cards will keep on mining.

with that kind of interface we can use almost any board out there..that interface should have a software management and hardware switches(like for disabling power) for the plug and play feature.

This mobo has usefull feature that shows what is wrong with PCIE!

Its far from plug and play but we actually dont have that many issues with stability of systems and if something goes wrong with pcie we see it.

cReepas

[mikespax]

If I tried to use 4GB RAM and a G3900 Celeron CPU with a 19 GPU rig like this, I would probably have a bad time right? I've got 9 GPUs on these specs right now, and wondering how long I'll be able to go on like this.

[Metroid]

If I tried to use 4GB RAM and a G3900 Celeron CPU with a 19 GPU rig like this, I would probably have a bad time right? I've got 9 GPUs on these specs right now, and wondering how long I'll be able to go on like this.

You would have to check the cpu usage, as long as 19 gpu's are working and cpu usage is less than 80% then is okay.

This mobo has usefull feature that shows what is wrong with PCIE!

Its far from plug and play but we actually dont have that many issues with stability of systems and if something goes wrong with pcie we see it.

cReepas

He is talking about hotplug gpu's.

[mikespax]

Currently, my 9x GPU rig has CPU1: 13.7% and CPU2: 90%

Also using 2.3 GB of 3.8 GB of ram.

So I should probably invest in some upgrades for more GPUs yeah?

[arielbit]

what we need is not a motherboard that have lots and lots of pcie...

what we need is an interface that can manage many cards where we can plug and play cards without turning off the system..where cards that have a problem would hang or stop separately without affecting the whole system so other cards will keep on mining.

with that kind of interface we can use almost any board out there..that interface should have a software management and hardware switches(like for disabling power) for the plug and play feature.

This mobo has usefull feature that shows what is wrong with PCIE!

Its far from plug and play but we actually dont have that many issues with stability of systems and if something goes wrong with pcie we see it.

cReepas

my old 775 motherboard circa 2006-2007..has led that shows connection state if connected..it doesn't show error color light but i can identify 1 out of 4 gpu is not mining by simply touching the gpus, the one that is not hot is not mining.

[mikespax]

As soon as I plug in the 14th GPU, I can't get this thing to boot. Any ideas?

[Vann]

As soon as I plug in the 14th GPU, I can't get this thing to boot. Any ideas?

What cards and OS?

[mikespax]

1080ti nvOS G3930 4GB ram

using an SD card on a card reader, two 2400w server PSUs and one 1600w ATX as the main PSU.

Seems no matter what I do, 14th card results in completely not booting at all, no bios or anything. Right back up when I remove one slot. Got another 4GB ram on the way already. Computer is stupid slow when I try to do anything on 13 cards, but seems to be mining stable so far.

[siforek]

As soon as I plug in the 14th GPU, I can't get this thing to boot. Any ideas?

Like no post? Not even to bios? I was going to say make sure you're powering everything. Depending on how you've got things connected you might have the 14th GPU on it's own in the 3rd PSU slots. If so are they getting power from 3rd PSU?

[mikespax]

2nd and 3rd server PSUs are powering all GPUs

Does not boot to bios. Screen does not register a signal.

Doesn't matter which slots are plugged in and which are empty, the 14th one just kills it.

Is this getting to look like a bad motherboard or one that I may have already damaged from improper initial configurations?

[Vann]

2nd and 3rd server PSUs are powering all GPUs

Does not boot to bios. Screen does not register a signal.

Doesn't matter which slots are plugged in and which are empty, the 14th one just kills it.

Is this getting to look like a bad motherboard or one that I may have already damaged from improper initial configurations?

If 13 GPU's are working, I doubt it's a bad motherboard. More likely it's a Nvidia driver limitation.

[jyakulis]

If I tried to use 4GB RAM and a G3900 Celeron CPU with a 19 GPU rig like this, I would probably have a bad time right? I've got 9 GPUs on these specs right now, and wondering how long I'll be able to go on like this.

You would have to check the cpu usage, as long as 19 gpu's are working and cpu usage is less than 80% then is okay.

This mobo has usefull feature that shows what is wrong with PCIE!

Its far from plug and play but we actually dont have that many issues with stability of systems and if something goes wrong with pcie we see it.

cReepas

He is talking about hotplug gpu's.

Yes, I'm curious as well. What kind of CPU are you using for this build?

[huynhdat1989]

Hello,

i just wanted to say, that we managed to get running with no big problems 19 GPUs in one Motherboard.

MOBO: Asus B250 Mining Expert

GPUs: 13x ASUS STRIX GTX 1060 6 GB + 6x ASUS MINING P106

MAIN PSU: 2400W
Secondary PSU (for powering pcie extenders and mobo) 750W
We have also 3th PSU in video, but in reality you need only main and secondary PSU.
16GB RAM

NO HDD just one 32GB FAST USB. SYSTEM [OS] nvOC easy-to-use Linux Nvidia Mining v0019

In windows we were able to get all cards do device manager, but mining was not running

https://drive.google.com/drive/u/1/folders/0B4y5D_zzH1oeLWpIcGtQOXI1MUU

will post tomorow more videos and detailed analysis of mobo with manual how to do that.

Best regards

cReepas

(Invictus Mining Team)

It sounds great but useless because any gpu or any component has problem, you need to restart all the system.
- GPU hang for long time running.
- Failed Risers
- Unstable Power Supply.

It could use for your hobby. But I would avoid any motherboard over than 7 gpus, you might check all large farms, they only use 4 to 7 gpus for every rig.

[Vann]

2nd and 3rd server PSUs are powering all GPUs

Does not boot to bios. Screen does not register a signal.

Doesn't matter which slots are plugged in and which are empty, the 14th one just kills it.

Is this getting to look like a bad motherboard or one that I may have already damaged from improper initial configurations?

Also the B250 chipset only supports up to 12 dedicated PCI-E lanes natively. The rest of the PCI-E slots use a PCI-E expansion chip. Nvidia cards are notorious for having PCI-E compatibility problems when using expansion chips, which is why they don't work with those x1 to 3 x1 PCI-E expansion cards.



 

[Metroid]

I wonder how Asus b250 mining expert compares to biostar b250 btc pro. I still think the Asus b250 mining expert is way better than the biostar in everything.

[Metroid]

Seems no matter what I do, 14th card results in completely not booting at all

You must be doing something wrong or you must not have done something right to make it to work properly, its one or the other, The asus b250 mining expert should have no problem booting all 16 cards, 8 amd and 8 nvidia on windows.

[mikespax]

Just got off the phone with OP. There's some sort of hardware limitation on 13 GPUs. I guess P106s work because they have no outputs, and supposedly AMD cards may also work because they have different drivers.

So yeah I need either P106 cards or AMD cards. I'm not yet sure how to run both AMD and NVDA on nvOS though. But perhaps I will put in some orders for Vega 64s to mix with my 1080tis.

[yatta]

I do have the same issue.
13 rx 560 boots fine, when i add 14 it does not boot. No screen signal, no keyboard signal, nothing.

[Metroid]

I do have the same issue.
13 rx 560 boots fine, when i add 14 it does not boot. No screen signal, no keyboard signal, nothing.

Linux?

[Vann]

From this graphic, when using more than 8 cards, Asus is recommending using a maximum of 8 Nvidia and 11 AMD cards due to I/O space and driver compatibility on Windows.



BBT tweeted the Powercolor Red Dragon RX 570 cards he ordered for a 19 card build with this board came in, so he should do a livestream setup in the next couple of days.

[mikespax]

I ordered a 19 gpu, 3 psu frame from this guy on eBay btw.

http://www.ebay.es/itm/322780828479

[mikespax]

Mine: https://www.youtube.com/watch?v=9_mv1pd3BIA

[darkshad]

Saw your youtube video.  Quite impressive.

Know where I could find a 19 GPU frame to be shipped to the U.S.?

I've looked at ebay.com, but so far, the max they have is a frame for 14 GPUs.

I ordered a 19 gpu, 3 psu frame from this guy on eBay btw.

http://www.ebay.es/itm/322780828479

[mikespax]

Didn't exist when I ordered. I just found someone on eBay who made custom frames and asked him to do one for me.

Metal is definitely great to help with the heat dissipation. My frame is super hot to the touch with the blower cards.

Does anyone really think a bios update will enable over 13 non mining GPUs? Seems pretty impossible when you consider it doesn't even boot to bios on 14 1080tis.

[Spotswood]

Saw your youtube video.  Quite impressive.

Know where I could find a 19 GPU frame to be shipped to the U.S.?

I've looked at ebay.com, but so far, the max they have is a frame for 14 GPUs.

I ordered a 19 gpu, 3 psu frame from this guy on eBay btw.

http://www.ebay.es/itm/322780828479

PM sent.   

[davemanet]

Hello,

i just wanted to say, that we managed to get running with no big problems 19 GPUs in one Motherboard.

MOBO: Asus B250 Mining Expert

GPUs: 13x ASUS STRIX GTX 1060 6 GB + 6x ASUS MINING P106

MAIN PSU: 2400W
Secondary PSU (for powering pcie extenders and mobo) 750W
We have also 3th PSU in video, but in reality you need only main and secondary PSU.
16GB RAM

NO HDD just one 32GB FAST USB. SYSTEM [OS] nvOC easy-to-use Linux Nvidia Mining v0019

In windows we were able to get all cards do device manager, but mining was not running

https://drive.google.com/drive/u/1/folders/0B4y5D_zzH1oeLWpIcGtQOXI1MUU

will post tomorow more videos and detailed analysis of mobo with manual how to do that.

Best regards

cReepas

(Invictus Mining Team)

Hi so you have one PSU for all VGA power and one for all risers?  I thought that you have to have riser and VGA power coming from same PSU?

[Lampaster]

I heard that Windows doesn't support such hardware. But now not about it. In order to talk about the feasibility of using such boards, it is necessary to calculate the estimate. I'm not sure that the usage of this motherboard economically justified. It seems to me that the use for mining of old equipment for 4 GPU on Board will be cheaper. If you fail something or will crash the computer all 19, the GPU will stand. If you use the motherboard for 4 GPU will only stop a small part of the GPU.

[umine]

Mine: https://www.youtube.com/watch?v=9_mv1pd3BIA

Looks really great! What is the weight of you rig? I think it's about 60-80 kilogramms

[mikespax]

Almost killed myself putting it on that shelf by myself. Yeah it's heavy and the frame isn't exactly the most sturdy thing in the world under all that weight. At least the server PSUs are relatively light compared to the EVGA 1600w PSU.

[Helioshel]

Omg. What system do you use to mine? I thought how difficult was to set up everything

[worldofcrypto]

This motherboard is great!

Now, the issue is: How am I able to use 19 P106 in EthOS with a 60% power limit? Is it possible?

Also, I need to get those 19 card racks!

[mikespax]

This motherboard is great!

Now, the issue is: How am I able to use 19 P106 in EthOS with a 60% power limit? Is it possible?

Also, I need to get those 19 card racks!

nvOC on Linux

[jadefalke]

nice work Guys, pretty Impressive to see such an high amount of Cards attached to ONE Motherboard

[HardFireMiner]

I have some setups mining right now, I am also new to mining, only 6 months mining with GPU's so don't have a lot of experience.

What I discovered alone is that it is much better to have more rigs than less, a lot less of points of failure and a lot less time to find the broken part in case something doesn't work. I have 3,4,5,6 GPU's rigs and I like the most the 4 GPU's Rigs, maybe I am silly here but in case a 4 GPU rig doesn't mine, it's only 4 GPU.

Debugging a 19 GPU failure must be a pain in the ass.

[mikespax]

Myeah. I like saving space and have a small apt so I'll be a one rig man. Unless I want to build a second 19 GPU rig perhaps.

[radiera]

how much vga run when use windows 10 ? its so need linux for get 19card... any one has try in windows ?

[snaidervp]

15 min to boot?
this is unstable AF.

[namct]

Does it run stable on win10?

Show me the tutorial video,plss

[Strebba90]

is possible run 19 vega?
linux or windows?

[mikespax]

Is anyone else using this board with 19 cards yet? I'm finding that every even numbered card is like 4mh slower than the others.

[risinqsun]

Not possible with windows os 9Gpu +
SimpleMining around  12-16  Gpu I'm not sure !
EthOS Guesses 12Gpu support

For now  impossible 19 gpu

[shibob]

I've already tried with all GPUs Asus mining P106-100 which are available for me now:

1. Update window 10 to ver.1709: still regconize only 8 GPUs.
I'm not sure how some Chinese did for 16 GPUs P106-100 in win10, but my friend visited and confirmed that's real.
 
2. Simplemining OS: regconize maximum 16 GPUs.

My AMD cards are in another place so could not go further test by mixing AMD+Nvidia.
Btw, Bits Be Trippin had some tests on this, it's really interesting to take a look:
https://www.youtube.com/user/BitsBeTrippin/videos

[weka]

i have 8GB RAM single channel DDR4 and 18 of NVIDIA GTX 1080TI

somehow my ASUS Mining Expert board won't let me boot on the 14th GPU, do i have to add more RAM? or is it just really can't support 18 of GTX 1080 TI?

[jeswin]

Very good indeed...

[codestaxx]

i have 8GB RAM single channel DDR4 and 18 of NVIDIA GTX 1080TI

somehow my ASUS Mining Expert board won't let me boot on the 14th GPU, do i have to add more RAM? or is it just really can't support 18 of GTX 1080 TI?

I'm running into the same thing with my B250.  8 AM RX570s and 8 NVIDIA 1060s on 8GB of RAM.  I can verify all cards show up fine in Device Manager.  I can't get the computer to post with more than 13 GPUS.  I have to unplug the AMDs.  It will finally post but Claymore hangs when I run it. 

Funny thing Claymore only recommends the AMD cards.  I know I have the Claymore dual AMD/NVIDIA miner too.  Device Manager shows all connected cards as working properly.  I wonder if there's a bios update....hmmm.

I'm thinking more ram too? 

[Peshny]

Prepare sleighs in summer, and the farm in the winter
Criticize who may have collected similar modular pattern !
2 point for number 4
Staggered arrangement, the bottom card upside down for better air movement.
all 19 cards 106-100
http://forumimage.ru/uploads/20171128/151184810259955067.jpg
http://forumimage.ru/uploads/20171128/151184810259955067.jpg
http://forumimage.ru/uploads/20171128/151184811151114141.jpg
http://forumimage.ru/uploads/20171128/15118488165196111.jpg
http://forumimage.ru/uploads/20171128/151184812419887711.jpg

[shibob]

Prepare sleighs in summer, and the farm in the winter
Criticize who may have collected similar modular pattern !
2 point for number 4
Staggered arrangement, the bottom card upside down for better air movement.
all 19 cards 106-100

Your designs look very cool!
But I'm curious that in the outtake side, is it better and simpler if you design totally same as intake side?

[Peshny]

This is just one rig from lots of future.
When they will stand in the ranks of the space will be divided into the cold zone and hot.
In the cold zone of the room will be pumped purified air of the street, and from the output pipe of the hot stream will be connected to a single duct and disposed beyond the premises.
Each duct will stand duct fan capacity of 920 cubic meters per hour
According to the calculations at the temperature of the air flow +35 at the entrance and the allocation of the 2.3 kW heat output will be the difference in 7 degrees.
The performance of the duct fan will adjust the Arduino.

http://forumimage.ru/uploads/20171129/15119331034390342.jpg
http://forumimage.ru/uploads/20171129/151193311233383344.jpg

 

[risinqsun]

Prepare sleighs in summer, and the farm in the winter
Criticize who may have collected similar modular pattern !
2 point for number 4
Staggered arrangement, the bottom card upside down for better air movement.
all 19 cards 106-100

Wow amazing graphics ! Nice Bro
When Can I buy

[Peshny]

It's still a prototype!
He is still not in operation.
After commissioning it is necessary to test and eliminate defects.
Exposed to discuss and eliminate errors at the design stage
Not for sale, it is done for yourself.
Although it is possible for the subject to think closer to spring

[bel123]

any news when this mb will work for 19 cards without using min 6 mining cards?

[burnsauce]

64GB pagefile

the pagefile is a windows thing....and you cant run more than 8 of one brand of card in windows so literally NOONE will be tryin to run this board on windows with 19 cards

A pagefile is a general OS thing (actually a general Von Neumann "computer" thing).  In Linux, the OS pagefile is called 'swap'.

I think you might be talking about two different things. A OS pagefile is used to store memory pages that aren't in use to leave room in main memory to execute more programs. A miner may or may not require it's own, septate, pagefile for dealing with a graphics card that doesn't have enough physical memory to run what you are trying to run on it.  In that case, the 'extra' stuff that doesn't fit in working memory will have to be stored in a 'page file'.

[kriskarthik]

Is it possible to run 16 GTX1070 in B250? In the MOBO booklet it is mentioned as only 13 NVidea can be done max. And rest with AMD. is that it or anyone successfully did All NVidea cards in B250?

[Bigdrago]

Thanks for this informative thread.

How much memory would u recommend for this? 2x8gb?
What cpu would u recommend? Celeron G3930?

[obit8]

I'm using this mobo with 13 nvidia 1060 3gb, i have other 5 cards and i don't know how to use it!!!!

Asus told that with the bios update the gpu limit would be removed but it doesn't work anyway!!!

[IndigoSun]

HI im newin this board, but one thing is taking my time and my patience out , may cuz i am newbie. May? Lol

So my question is about all the time when the windows restart its got stocked in the motherboard bios set up again . Please how to take out this ?How to could set up the bios to do not restart in the set up screen please  cuz its gimming a lot of pain cake

Thanks