Onda B250-D12P-D1 cannot boot more then two Tesla P100 -16GB Cards - 100$ Reward

[jadefalke]

Hi Guys,

i have here multiple Tesla P100 16 GB and try to get them working in a Mining Case with a Onda B250-D12P-D1 Motherboard.

Whole Setup:
2550Watt Power Supply
celeron cpu
8 GB DDR3 RAM
M2 Drive (64 gig)
8X Tesla P100 16 GB PCIE

If i add 2 Tesla P100 Cards it is working as expected, as soon as i add a 3rd card i have a black Screen and Rig does not boot up and hive is marking it as offline.

Bios has:

4 GB encoding active

CSM off

For the first who provides me a Solution to get the Rig fully populated with 8 Tesla P100 Cards working i will send out 100$ in USDT as a reward!

[1715352933]

1715352933

[1715352933]

1715352933

[1715352933]

1715352933

[1715352933]

1715352933

[1971ECPT]

Hi, try to use bigger SSD (256GB) and increase virtual memory at least as sum of entire quantity of  memory of all video cards used. By example if you have 10 cards of 16 GB video memory each, virtual memory must be at least 160GB. Also, update motherboad bios to the latest version available on producer website. Try to leave CSM on, too, with less UEFI and more legacy.Good luck!

[JayDDee]

Available PCIe lanes could also be an issue considering the CPU Is Celeron. Try with a SATA SSD instead of m2NVME and check if any other peripherals
share lanes with the PCIe slots. Also make sure all the motherboard's extra power connectors are powered.

I get the feeling that Celeron & Tesla are a bad combination, like putting cheap tires on a Ferrari.

[jadefalke]

Available PCIe lanes could also be an issue considering the CPU Is Celeron. Try with a SATA SSD instead of m2NVME and check if any other peripherals
share lanes with the PCIe slots. Also make sure all the motherboard's extra power connectors are powered.

I get the feeling that Celeron & Tesla are a bad combination, like putting cheap tires on a Ferrari.

good input, will try that out

[jadefalke]

update, replaced the M2. with SSD, no change, also CSM on or of does not change anything. also UEFI vs. Legacy seem to be not a factor. With 3 cards it won't post at all, just a black screen
t

[JayDDee]

update, replaced the M2. with SSD, no change, also CSM on or of does not change anything. also UEFI vs. Legacy seem to be not a factor. With 3 cards it won't post at all, just a black screen
t

You need to determine if it's a display issue or boot issue, Any difference in beep tones if your MB has a speaker? Does it respond to a ping?
I assume you've tried different cards in different slots to rule that out. Do your Teslas have display output? Check those if you have no ouput from iGPU.
Don't forget about VM. Try a different OS.

[jadefalke]

update, replaced the M2. with SSD, no change, also CSM on or of does not change anything. also UEFI vs. Legacy seem to be not a factor. With 3 cards it won't post at all, just a black screen
t

You need to determine if it's a display issue or boot issue, Any difference in beep tones if your MB has a speaker? Does it respond to a ping?
I assume you've tried different cards in different slots to rule that out. Do your Teslas have display output? Check those if you have no ouput from iGPU.
Don't forget about VM. Try a different OS.

Hey,

Tesla don't have a Video Output, and it is Hive installed, so even if no video output would be there, it would boot into hive, (which is not happening). I tried also Windows, but again it is not going to boot at all. It is all quite, does not beep and Monitor does not turn on at any time.

[1971ECPT]

Maybe the last option is bios update. Try it, just with integrated graphic video. Also, try to use better CPU, maybe Celeron is too weak.

[JayDDee]

It's not the mobo or the GPUs if any 2 cards work in any 2 slots and it's not the OS if both Win & HiveOS fail with > 2 GPUs.
I don't suspect BIOS unless it's been buggered with.
It's an incremental problem pointing to a resource exhaustion. That means not enough power or not enough lanes. VM is ruled out if it doesn't POST.
Celerons apparently work in minimalist large rigs with GTX but maybe Teslas are a different story.
I suggest try a different PSU and try installing >2 gaming GPUs if you have some to test with. Reflashing the BIOS or trying a bigger CPU would be
last thing to try but would be a long shot.

[jadefalke]

Bigger CPU is coming in today, but 4-Core is the max. for that board anyway. Bios update is not so easy, as i could not find any newer bios at all.

[jadefalke]

I gave up and moved to a IBM Server System looks like everything with a B250 Chipset and PCI-Lane splitting will not work for the P100, sad ;(

[FP91G]

I wish I had read this earlier. In such cases, you usually need to change the motherboard and processor. Many Tesla P100 video cards do not work on all older platforms. And for old video cards, your Onda B250-D12P-D1 Motherboard will still be relevant even with a old Celeron.

[batsonxl]

1.Try with risers:NO
2.Change motherboard

[666mrga999]

Not every celeron cpu is good for multi gpu mining. I remember from eth mining days when I bought wrong celeron for my 19xpcie motherboard. Celeron G3900 works with multiple (8+) gpus at x1 pcie.
Do not use first pcie slot, IMPORTANT it is x16 and it uses up most pcie lanes.

TLDR: DO NOT PUT GPU IN FIRST SLOT, LEAVE EMPTY

[JayDDee]

Not every celeron cpu is good for multi gpu mining. I remember from eth mining days when I bought wrong celeron for my 19xpcie motherboard. Celeron G3900 works with multiple (8+) gpus at x1 pcie.
Do not use first pcie slot, IMPORTANT it is x16 and it uses up most pcie lanes.

TLDR: DO NOT PUT GPU IN FIRST SLOT, LEAVE EMPTY

This makes sense, though using a Celeron for a large (> $5k) GPU rig is too minimalist IMO. Spend an extra $100 for a i3 or i5.
In this case there is still the issue of it being Tesla GPUs. They have much more VRAM to support and may have minimum lane requirements.
I'm not speaking from actual experience. I expect a motherboard designed for mining would work out of the box in most cases. The unknown
variable seems to be using Tesla GPUs in a large rig. It's also possible Nvidia deliberately borked Teslas to require higher end platforms. After
LHR I wouldn't put it past them.

[FP91G]

Not every celeron cpu is good for multi gpu mining. I remember from eth mining days when I bought wrong celeron for my 19xpcie motherboard. Celeron G3900 works with multiple (8+) gpus at x1 pcie.
Do not use first pcie slot, IMPORTANT it is x16 and it uses up most pcie lanes.

TLDR: DO NOT PUT GPU IN FIRST SLOT, LEAVE EMPTY

If you are using an x1 raiser, then there should be no problems with 8 video cards on a regular motherboard. And you should also not use NVMe drives.
If there are more video cards, then it is better to think about buying another processor in advance. But for me, the ideal number of video cards on a motherboard is 6, so that consumption is no more than 1 kilowatt per hour.

[jadefalke]

Just to keep you all in the loop, the Celeron theory was tested i bought i3-4 Core CPU, same result. i suspect the B250 Chipset and how the PCI Lanes are address is part of the Problem. Expensive Lesson for us, as we have to buy now Server ontop of the cases we got initially.
(we event tried two other different Mining mainboards, but they had also the B250 Chipsets and did not work out.)
We even were going as far as getting an Octominer Case, but that did not work either. Only working solition so far was a professional Server System.

[666mrga999]

Did you leave first PCIE slot empty, like I told you, this link has solved similar problem to yours with this MB

https://rsts11.com/2021/03/14/wrestling-with-an-onda-b250-d8p-d4-mining-motherboard/

[JayDDee]

Different Tesla,  same problem.

https://linustechtips.com/topic/1327883-folding-at-home-gpu-mining-rig-18-tesla-k20ms-a-little-help-needed/

[batsonxl]

Just to keep you all in the loop, the Celeron theory was tested i bought i3-4 Core CPU, same result. i suspect the B250 Chipset and how the PCI Lanes are address is part of the Problem. Expensive Lesson for us, as we have to buy now Server ontop of the cases we got initially.
(we event tried two other different Mining mainboards, but they had also the B250 Chipsets and did not work out.)
We even were going as far as getting an Octominer Case, but that did not work either. Only working solition so far was a professional Server System.

Is it works with riser?
Is it windows?or? Driver?

[jadefalke]

Just to keep you all in the loop, the Celeron theory was tested i bought i3-4 Core CPU, same result. i suspect the B250 Chipset and how the PCI Lanes are address is part of the Problem. Expensive Lesson for us, as we have to buy now Server ontop of the cases we got initially.
(we event tried two other different Mining mainboards, but they had also the B250 Chipsets and did not work out.)
We even were going as far as getting an Octominer Case, but that did not work either. Only working solition so far was a professional Server System.

Is it works with riser?
Is it windows?or? Driver?

it is on bios level, and no riser do not work either. we stopped the efforts on this one, we bought professional Server now.

[jadefalke]

Different Tesla,  same problem.

https://linustechtips.com/topic/1327883-folding-at-home-gpu-mining-rig-18-tesla-k20ms-a-little-help-needed/

yeah, sadly they did not find a solution either ;(

[JayDDee]

Different Tesla,  same problem.

https://linustechtips.com/topic/1327883-folding-at-home-gpu-mining-rig-18-tesla-k20ms-a-little-help-needed/

yeah, sadly they did not find a solution either ;(

It's kinda ironic, the slot spacing is better suited for Teslas than Gaming GPUs.

[jadefalke]

Small Update i got a lead from reddit :
Does the BIOS have resizeable bar support? I heard that P100/P40 requires this for multiple cards to work well.

i could not locate resizable Bar Support in the American Megatrends Bios ;(

somehow i cannot add pictures here, otherwise i would have share the bios options.

[DaveF]

Small Update i got a lead from reddit :
Does the BIOS have resizeable bar support? I heard that P100/P40 requires this for multiple cards to work well.

i could not locate resizable Bar Support in the American Megatrends Bios ;(

somehow i cannot add pictures here, otherwise i would have share the bios options.

1) No it does not. Or at least it did not the last time I used one of these boards.

2) Is it *really* worth all the time you have been putting into this? Would you be better off getting something that is known to support it. YES there is a cost, but how much are you loosing each day not running all the cards?

3) Go to https://www.talkimg.com/ and you can upload an image there and then post it here. If you want an account react out to @joker_josue https://bitcointalk.org/index.php?action=profile;u=97582

-Dave

[jadefalke]

Small Update i got a lead from reddit :
Does the BIOS have resizeable bar support? I heard that P100/P40 requires this for multiple cards to work well.

i could not locate resizable Bar Support in the American Megatrends Bios ;(

somehow i cannot add pictures here, otherwise i would have share the bios options.

1) No it does not. Or at least it did not the last time I used one of these boards.

2) Is it *really* worth all the time you have been putting into this? Would you be better off getting something that is known to support it. YES there is a cost, but how much are you loosing each day not running all the cards?

3) Go to https://www.talkimg.com/ and you can upload an image there and then post it here. If you want an account react out to @joker_josue https://bitcointalk.org/index.php?action=profile;u=97582

-Dave

Thanks Dave for the heads up, appreciated. currently i have a truckload of theses ondo cases sitting and catching dust. We currently deploy that batch of P100 with GPU Server that can take 8 of them at a time and work flawlessly. When i get the ondo cases working, i could deploy a second batch P100 pretty rapidly. So yes, it is worth the time, cause otherwise the ondo cases are dead in the water.

[JayDDee]

Small Update i got a lead from reddit :
Does the BIOS have resizeable bar support? I heard that P100/P40 requires this for multiple cards to work well.

i could not locate resizable Bar Support in the American Megatrends Bios ;(

somehow i cannot add pictures here, otherwise i would have share the bios options.

I think you have something here, but probably not leading to a solution. It could be the B250 chipset. I have a Asus B350 with rebar but it's not documented in the
manual so possibly was introduced with B450 and back ported to B350 with a BIOS update. It's unlikely possible on the B250 but if so would require the update from Onda.

[jadefalke]

Small Update i got a lead from reddit :
Does the BIOS have resizeable bar support? I heard that P100/P40 requires this for multiple cards to work well.

i could not locate resizable Bar Support in the American Megatrends Bios ;(

somehow i cannot add pictures here, otherwise i would have share the bios options.

I think you have something here, but probably not leading to a solution. It could be the B250 chipset. I have a Asus B350 with rebar but it's not documented in the
manual so possibly was introduced with B450 and back ported to B350 with a BIOS update. It's unlikely possible on the B250 but if so would require the update from Onda.

The Problem is, Onda is not very supportive at all, as most of these chinese "cheap,cheap,cheap"  mining companys...

[jadefalke]

Small Update: Rebar Support is not an option it needs Generation 10 CPU and that Board support only Generation 6+7, so that is a dead end.