X6500 Custom FPGA Miner

[TheSeven]

Some mining rig photos: I went through with my plans for a mineral oil cooled setup. :-)

What you can't see here: The USB and power cable is rising up to the top, as I placed a nail there above the whole setup to run the cables along that. This is an attempt to prevent oil from trickling through the cables. So far the oil stays where it should, hopefully that will last.

All in all it seems to work fairly well. I would like your input though on what is an acceptable temperature to run the FPGA at. First I tried running without a fan and the oil would slowly start heating up and after about 30 minutes the thermometer showed 46 deg C (you can see in the picture that the thermometer is measuring in the middle of the heat sink). At that point I stopped and added a fan. It's not really helping all that much, but it is keeping the setup at about 43 deg C right now.

On the other hand I have yet to see a single invalid share. The system has been mining for over a day at 200 MHz (per FPGA) with 0 invalids. So would that indicate that I can risk a slightly higher temperature? I would really like to run this completely passive.

I was also thinking that this might be a good excuse to play around with 3d printing a little bit and maybe build something that would increase the surface area for the oil to cool down and maybe even have some kind of circular setup driven by convection. Although if I do that, which seems like a fun project, it would probably be just me guessing how to do that rather than being based on actual physics. =)

Looking at the thermal data of the FPGA, I'd say that a heatsink temperature of 50°C and a board temperature of 60°C should be perfectly fine. 10°C more might cause invalids but shouldn't really hurt it. So as long as the oil stays around 50°C it should be fine.

I'm not sure if the heatsink (and thermal adhesive) improves or worsens the situation here though... If you manage to establish oil convection across the FPGA's package, you might be better off without a heatsink at all, if the oil at the FPGA's surface stays at 50°C that would be ideal.

[1714159644]

1714159644

[1714159644]

1714159644

[shackleford]

Some mining rig photos: I went through with my plans for a mineral oil cooled setup. :-)







What you can't see here: The USB and power cable is rising up to the top, as I placed a nail there above the whole setup to run the cables along that. This is an attempt to prevent oil from trickling through the cables. So far the oil stays where it should, hopefully that will last.

All in all it seems to work fairly well. I would like your input though on what is an acceptable temperature to run the FPGA at. First I tried running without a fan and the oil would slowly start heating up and after about 30 minutes the thermometer showed 46 deg C (you can see in the picture that the thermometer is measuring in the middle of the heat sink). At that point I stopped and added a fan. It's not really helping all that much, but it is keeping the setup at about 43 deg C right now.

On the other hand I have yet to see a single invalid share. The system has been mining for over a day at 200 MHz (per FPGA) with 0 invalids. So would that indicate that I can risk a slightly higher temperature? I would really like to run this completely passive.

I was also thinking that this might be a good excuse to play around with 3d printing a little bit and maybe build something that would increase the surface area for the oil to cool down and maybe even have some kind of circular setup driven by convection. Although if I do that, which seems like a fun project, it would probably be just me guessing how to do that rather than being based on actual physics. =)

You beat me to it, I just received my tech grade mineral oil yesterday and the rest of my parts should arrive by the end of the week. I think I have a pretty good design, building it completely out of acrylic glued like a fish tank. I plan to include the cooling into the lid. One lid will be two heat sinks connected through the lid. The other plan is the nuclear option with a peltier cooler sandwiched between them.  The enclosure itself is designed for convention so the cooling in the lid may be completely unneeded.. we will see.

[BR0KK]

Awww you know that you will end up with oil on you desks? And u lost your warranty btw.

It's called cappilar effect .... i did that once with my computer and the oil crawled up the mouse cable and after a short while my mouse began to bleed oil ...

For the Cooling you could try some parts from Watercooling projects. A pump that moves the oil through a radiator and back into the tank; or stick a small Fan in there that moves the oil thru the x6500 cooler.

[O_Shovah]

Hello everybody,

As promised i release some images and data of my FPGA family at its daily work.

[...]

[From left to right: ARM board "Gumstix" (the first i developed on), Icarus(rev0),120mmfan ,x6500(rev2),120mmfan,Pandaboard ES(revB2),Bitforce single(1.0)]

[...]

Do you use the Pandaboard to control the FPGAs?
How does the Pandaboard cope with the workload? What do you think, how many FPGAs could you manage with it?

The panda board acts a a full size desktop with a full size ubuntu distribution.
It is running the mpbm miner and cotrols the boards via one or multiple USB hubs.
( and is using 8,5 W with panda board plus hub mouse etc)

It is also capaple of fully running a bitcoin-qt client( i recompiled one for ARM boards) and also act as a p2pool server at marginal cpu usage levels.
But p2pool is not recommended for the current mpbm version as this will cause high stale levels due to timing constraints.
While it is used as a board running the miner connected to a pool its mostly ideling. The cpu is really bored.

The board most on the left is my first ARM board  the "Gumstix"  it also ran the miner without major computation problems whe connected to a pool.(It consumes 4 W on the wall for a full ubuntu)
But it encounters performance limitations when you try to run a p2pool server on it. 


I am currently working to provide a full working image  for the panda board and the gumstix as well.
I also started creating a server to distribute .deb packages with fully ported ARM aplications.

Furthermore i want to publish similar things for the rasberry pi ( wich would be perfect as a host pc for mining in an pool)
as soon as i get my hands on one.

Time flys you know, so i cant give a certain date for publishing now.


I also took thermal imaging of the bitforce box today an redid the power measurement.
But more of that tomorrow.

[TheSeven]

It is also capaple of fully running a bitcoin-qt client( i recompiled one for ARM boards) and also act as a p2pool server at marginal cpu usage levels.
But p2pool is not recommended for the current mpbm version as this will cause high stale levels due to timing constraints.
While it is used as a board running the miner connected to a pool its mostly ideling. The cpu is really bored.

P2Pool should work fine with current master / testing tree. However it looks like p2pool is running into performance issues on the pandaboard as well, causing ~6% DOA for me. If I host it somewhere else it's running fine.

[O_Shovah]

It is also capaple of fully running a bitcoin-qt client( i recompiled one for ARM boards) and also act as a p2pool server at marginal cpu usage levels.
But p2pool is not recommended for the current mpbm version as this will cause high stale levels due to timing constraints.
While it is used as a board running the miner connected to a pool its mostly ideling. The cpu is really bored.

P2Pool should work fine with current master / testing tree. However it looks like p2pool is running into performance issues on the pandaboard as well, causing ~6% DOA for me. If I host it somewhere else it's running fine.

Might be somehow related to the I/O perfromance of your SD card. Mine is very fast and it just unns fine. ( well i also did some modifications to ubuntu to optimise).
As far as TheSeven told me mpbm has some latency issues wich are the price of the modular approach.

[TheSeven]

It is also capaple of fully running a bitcoin-qt client( i recompiled one for ARM boards) and also act as a p2pool server at marginal cpu usage levels.
But p2pool is not recommended for the current mpbm version as this will cause high stale levels due to timing constraints.
While it is used as a board running the miner connected to a pool its mostly ideling. The cpu is really bored.

P2Pool should work fine with current master / testing tree. However it looks like p2pool is running into performance issues on the pandaboard as well, causing ~6% DOA for me. If I host it somewhere else it's running fine.

Might be somehow related to the I/O perfromance of your SD card. Mine is very fast and it just unns fine. ( well i also did some modifications to ubuntu to optimise).
As far as TheSeven told me mpbm has some latency issues wich are the price of the modular approach.

Yes, it might well be related to SD card performance, that's what I was suspecting as well. (Getting a very high response latency variance close to found shares.)
MPBM surely isn't the miner with the lowest latencies, but since I've fixed a new block detection bug and decoupled the cancellation of jobs into another thread, it seems to work well enough. I'm getting around 1.7% DOA stales with a remote P2Pool instance, running MPBM on a pandaboard.

[twmz]

Some mining rig photos: I went through with my plans for a mineral oil cooled setup. :-)

What you can't see here: The USB and power cable is rising up to the top, as I placed a nail there above the whole setup to run the cables along that. This is an attempt to prevent oil from trickling through the cables. So far the oil stays where it should, hopefully that will last.

All in all it seems to work fairly well. I would like your input though on what is an acceptable temperature to run the FPGA at. First I tried running without a fan and the oil would slowly start heating up and after about 30 minutes the thermometer showed 46 deg C (you can see in the picture that the thermometer is measuring in the middle of the heat sink). At that point I stopped and added a fan. It's not really helping all that much, but it is keeping the setup at about 43 deg C right now.

On the other hand I have yet to see a single invalid share. The system has been mining for over a day at 200 MHz (per FPGA) with 0 invalids. So would that indicate that I can risk a slightly higher temperature? I would really like to run this completely passive.

I was also thinking that this might be a good excuse to play around with 3d printing a little bit and maybe build something that would increase the surface area for the oil to cool down and maybe even have some kind of circular setup driven by convection. Although if I do that, which seems like a fun project, it would probably be just me guessing how to do that rather than being based on actual physics. =)

You beat me to it, I just received my tech grade mineral oil yesterday and the rest of my parts should arrive by the end of the week. I think I have a pretty good design, building it completely out of acrylic glued like a fish tank. I plan to include the cooling into the lid. One lid will be two heat sinks connected through the lid. The other plan is the nuclear option with a peltier cooler sandwiched between them.  The enclosure itself is designed for convention so the cooling in the lid may be completely unneeded.. we will see.

These guys did a lot of experimenting across several revisions to a fully submerged aquarium gaming PC.  Might be some tips in their articles about how they did cooling, etc:

http://www.pugetsystems.com/aquarium-computer.php

http://www.pugetsystems.com/mineral-oil-pc.php

http://www.pugetsystems.com/submerged.php

[shackleford]

These guys did a lot of experimenting across several revisions to a fully submerged aquarium gaming PC.  Might be some tips in their articles about how they did cooling, etc:

http://www.pugetsystems.com/aquarium-computer.php

http://www.pugetsystems.com/mineral-oil-pc.php

http://www.pugetsystems.com/submerged.php

Yup, I got the oil from their partnered vendor. Mineral oil has the general opinion that it is not worth it in the PC arena and water cooling is better. I think mainly because people would like to use it to cool high end gaming rigs. For FPGA's we have significantly less wattage to worry about.

[antirack]

I am a bit surprised that there are no finished solutions for this problem. I can't imagine that there are no companies (scientific research?) that wouldn't need to submerge some equipment in mineral oil for cooling once in a while.

A (small) tank with some sort of rail/frame to attach equipment/PCBs and then established oil flow with external heat exchanger would be a neat product. But my five minutes of research didn't end up on any web site/manufacturer with a product like this. The closest thing I found was cryogenic equipment...

I can imagine it would only be interesting if it would work for a very high density, say minimum of 125 X6500 or Icarus boards (since this is the hashing power that the rig "box" is supposed to deliver, if it ever shows up).

[kakobrekla]

I am a bit surprised that there are no finished solutions for this problem. I can't imagine that there are no companies (scientific research?) that wouldn't need to submerge some equipment in mineral oil for cooling once in a while.

A (small) tank with some sort of rail/frame to attach equipment/PCBs and then established oil flow with external heat exchanger would be a neat product. But my five minutes of research didn't end up on any web site/manufacturer with a product like this. The closest thing I found was cryogenic equipment...

I can imagine it would only be interesting if it would work for a very high density, say minimum of 125 X6500 or Icarus boards (since this is the hashing power that the rig "box" is supposed to deliver, if it ever shows up).

Inert fluids are better than mineral oil. Like 3M Novec...

Iceotope Liquid Cooled Cabinets (with Novec)
http://www.youtube.com/watch?v=d2TS_qQ3UNQ

3M™ Novec™ 1230 Fire Protection Fluid Electronics
http://www.youtube.com/watch?v=1E7Nr72rxB8

You can put a piece of paper in this and it will be dry when you pull it out.

[antirack]

Thanks for the links kakobrekla, I was more talking about a solution that a mere mortal like me or a small business can buy off the shelf (or through some specialized company). A case and some equipment just for cooling small stuff or small computers like we use them for mining... so far all I could find was home-made stuff.

Basically something like this, but built by a company so people/organizations in need could just place an order:
http://www.g4tv.com/articles/37474/build-a-submersion-cooling-case/

[lame.duck]

I am a bit surprised that there are no finished solutions for this problem. I can't imagine that there are no companies (scientific research?) that wouldn't need to submerge some equipment in mineral oil for cooling once in a while.

A (small) tank with some sort of rail/frame to attach equipment/PCBs and then established oil flow with external heat exchanger would be a neat product. But my five minutes of research didn't end up on any web site/manufacturer with a product like this. The closest thing I found was cryogenic equipment...

I can imagine it would only be interesting if it would work for a very high density, say minimum of 125 X6500 or Icarus boards (since this is the hashing power that the rig "box" is supposed to deliver, if it ever shows up).

Why somebody should invent a suboptimal solution while there is already the better solution watercooling? Water can carry a lot of energy per volume and the temperature range  is perfect. Oil cooling would only make sense if the temperature would be  higher or there were high voltages involved.

And there are companies that produce  Water cooling equipment, and yes, you have to adapt it to your special needs, its like the water pipe system in a house that has to be assembled by a plumber as avery house is more or less different.

[DeepBit]

I can imagine it would only be interesting if it would work for a very high density, say minimum of 125 X6500 or Icarus boards (since this is the hashing power that the rig "box" is supposed to deliver, if it ever shows up).

Why somebody should invent a suboptimal solution while there is already the better solution watercooling? Water can carry a lot of energy per volume and the temperature range  is perfect. Oil cooling would only make sense if the temperature would be  higher or there were high voltages involved.

Water cooling is WAY too expensive.

[lame.duck]

I can imagine it would only be interesting if it would work for a very high density, say minimum of 125 X6500 or Icarus boards (since this is the hashing power that the rig "box" is supposed to deliver, if it ever shows up).

Why somebody should invent a suboptimal solution while there is already the better solution watercooling? Water can carry a lot of energy per volume and the temperature range  is perfect. Oil cooling would only make sense if the temperature would be  higher or there were high voltages involved.

Water cooling is WAY too expensive.

I doubt if cryogenic or oil would be less expensive, since you need customized heat exchanger and  pipes to.
I bet with proper heat sinks, dense packing and a clean air flow through the heatsinks air cooling should be  sufficient.

 

[DeepBit]

Why somebody should invent a suboptimal solution while there is already the better solution watercooling? Water can carry a lot of energy per volume and the temperature range  is perfect. Oil cooling would only make sense if the temperature would be  higher or there were high voltages involved.

Water cooling is WAY too expensive.

I doubt if cryogenic or oil would be less expensive, since you need customized heat exchanger and  pipes to.
I bet with proper heat sinks, dense packing and a clean air flow through the heatsinks air cooling should be  sufficient.

For water-cooling you'll need to machine special heat exchangers for EACH chip (or each two in some cases). May be some chipset waterblocks are compatible, but those aren't really cheap too.
For simple oil immersion you just need a bucket and the oil :)

Chipset waterblock is about $40 to $50 for low quantity orders, so it's at least $80 for each board. Notice that I'm not talking about pipes, pumps, water-to-air exchangers, fans and other accessories. FPGA miner costs about $420 to $570, so just two waterblocks will cost you around 16% of your miner and that's too much. Mining gear should be as cheap as possible.

2. Spartan package is not so good for heat transfer, so you can't cool it efficiently from just the top. PCB backside will definitely need some airflow.

[Icoin]

Hi everyone

I realized that there is a cooling problem for the fpga, thats why i developed a watercooling element specialy for the 6500 boards. The element is developed in a way that it should fit to all generations developed by now. Till today i could only use the measurements of the 1. gen but a 3. gen Board is on its way to me.


This element should be avalable i guess some weeks after the 3. gen arrives




This is easy integratable in a existing CPU/GPU watercooling solution

[li_gangyi]

This element should be avalable i guess some weeks after the 3. gen arrives

Hi Icoin, wow that looks fantastic. I suppose it'll come with spring loaded screws or something to mount onto the board. I just wished the Spartan-6 had a metal lid instead of it being encapsulated in epoxy like at the moment, that seriously hampers our cooling efforts. Ah well...

[hashking]

Does the x6500 work on gpumax?

[Garr255]

Yes, with a proper PC it appears it should run great.