X6500 Custom FPGA Miner

[fizzisist]

fizzisist: Do the fgpa's have a temp sensor in them we can read back? Could you make a command line option for a "pausing" temperature threshold while the fpga is above a certain temp? 

Failing that, could you make a command line option reject rate pausing threshold? ie 3 parameters, possibly with the following defaults: if the reject/invalid rate within the last 30s goes above 10%, indicating something is most certainly wrong, pause for 120s to let things cool a bit. 

This throttling will no doubt save many fpga's from disaster as more people start pushing them harder, and fans start failing over time etc.

Spartan 6s do not include internal temp sensors, unfortunately. The only feedback you really get is from the number of invalids, and I'm not 100% confident that using that is completely bullet proof. It also has the problem that it can take a while to get reasonable feedback on this. At 200 MHz you're getting 1 nonce roughly every 20 seconds it would take 200 seconds to get an invalid (based on 10% invalids). Still, this is a good idea to use as a backup measure. 3 or 4 minutes at an unsafe temperature is a lot better than 3 or 4 hours.

The other problem is that we haven't worked out a really good way to pause and resume the FPGA. I pushed some new code to Github last week that experiments with this, but it's not ready for the real thing yet. Pausing seems to work perfectly, and it brings power usage down to basically zero, but waking the FPGA back up seems to not work right once every 10 times or so. When this happens, the FPGA reports only invalid nonces and the only way to recover it is to reprogram.

Anyway, this is a work in progress. I think we'll get something good figured out soon. Until then, keep an eye on your FPGAs!

[1713278180]

1713278180

[1713278180]

1713278180

[1713278180]

1713278180

[fizzisist]

1.)I'm currently shopping for Thermal Adhesive; before my adhesive arrives would it be safe to run my unit with the 180mhz bitstream and active cooling (I plan on having a 120mm fan actively cooling the card)?

2.) For people who own the unit with the stock heat sinks; do you find the stock sinks to be enough? Do you have any other ones you might recommend?

I plan on playing around with various cooling schemes for my x6500 (which I will share in later post with pics!) and if the mining goes well I may get a second unit!

Yep, lots of people have used 180 MHz with the board right out of the box and it seems to work well. Of course, you will need the fan. It's a good idea to channel the air from that big fan a little bit so that it's moving at maximum velocity as it passes over the FPGAs. A few strategically placed pieces of cardboard might help a lot. You'll have to play around to see what works well for you!

I would love to see how your cooling experiments go!

Thanks!

[ZodiacDragon84]

Would there be a way to mount a temp sensor under the chips?

[bitcowok]

Heres something for Linux users I worked out a while back but haven't had a chance to share yet.

If it works out well for me I'd like to incorporate it in the user guide on the website, if that's alright with you.

Yep np.

The best solution of course to all of this is upstream your changes to pyusb (if you haven't already) so that eventually the distro's version will be sufficient.

Does the licence on libftd2xx allow redistribution?  it might be easier on people to distribute that (the "working" 1.0.4 version) and the modified pyusb within your git tree.  Then you could provide an example shell script to launch using LD_PRELOAD and PYTHONPATH, with a note in the readme/instructions on fpgamining.com that this method of launching is available.

Re pausing: does simply stopping mining reduce power and temperature? or is it simply once the fpga is programmed its always using its max power?  The latter would explain why your pausing function is important.

I've been looking for a (cheap) usb external temperature probe system which supports multiple sensors on the same USB device. I'd like to connect the sensors to the heatsinks, and string together scripts to stop mining (or even better use your "pause" function) if temp goes above a threshold.

Thanks for the work!

[ZodiacDragon84]

one of these per board, (2x thermocouple per controller)
http://www.amazon.com/Ultra-ULT40123-MD3-Media-Dashboard/dp/B000ZN2KEQ/ref=sr_1_8?ie=UTF8&qid=1328064723&sr=8-8

[bitcowok]

It only displays the temp on an LCD.. i need to read it into the PC so script can react to fan failures.  Nice try though...

[nbtcminer]

@ Fizzisist:

Thanks for the reply! I definitely will post stuff when I get my unit (its in a sorting center atm lol). I did have another questions for you regarding putting thermal epoxy:

1.) How long did you let the epoxy cure for on the unit you ran the 200mhz bitstreams?
2.) Did you use any weights like li_gangyi? If so is there any particular method or weight you recommend using?

I was thinking of using 2 hard drives (approx 1kg) as weights and putting them on top of the heat stinks during the curing period (was going to try 12 hours). Thanks in advance for any input!


Cheers,
nbtcminer

O_shovah: you sure you use the thermal imager just for PCBs? xD hehehe

But yeah it'd be interesting to see.

Fizzi: told ya thermal epoxy makes a world of difference. I've been successful taking mine off by popping it off with a wide plastic spatula between the FPGA and the heatsink when it's warm. This way you don't put stress on the balls. You might crush the FPGA itself with too much force though. So it's still not a 100% solution especially if the epoxy is strong.

I was using this

http://www.arcticsilver.com/arctic_silver_thermal_adhesive.htm

I mixed up a small amount on a disposable spoon. Applied it to the center of my heatsink and put a 1KG weight on top of the heatsink overnight.

[fizzisist]

1.) How long did you let the epoxy cure for on the unit you ran the 200mhz bitstreams?
2.) Did you use any weights like li_gangyi? If so is there any particular method or weight you recommend using?

I was thinking of using 2 hard drives (approx 1kg) as weights and putting them on top of the heat stinks during the curing period (was going to try 12 hours). Thanks in advance for any input!

Yeah, I'm sure Li knows better than I do how to work with thermal epoxy. By the way, in case this isn't clear, li_gangyi is one of the fpgamining.com team, primarily in charge of the hardware design.

I simply followed these directions: http://www.arcticsilver.com/pdf/appinstruct/asta/ins_asepxy.pdf

I used about 6 paperback books for weight, but I flipped the board over so it was resting on the heatsinks and the books were on the back of the board. I removed the standoffs first, too. One important thing for me was to wait until the epoxy has just started to cure enough that the heatsinks didn't slide all over the place when I tried to put the weight on. Don't wait too long though, or it will cure without the pressure and not make good thermal contact!

[fizzisist]

Heres something for Linux users I worked out a while back but haven't had a chance to share yet.

If it works out well for me I'd like to incorporate it in the user guide on the website, if that's alright with you.

Yep np.

The best solution of course to all of this is upstream your changes to pyusb (if you haven't already) so that eventually the distro's version will be sufficient.

Does the licence on libftd2xx allow redistribution?  it might be easier on people to distribute that (the "working" 1.0.4 version) and the modified pyusb within your git tree.  Then you could provide an example shell script to launch using LD_PRELOAD and PYTHONPATH, with a note in the readme/instructions on fpgamining.com that this method of launching is available.

Re pausing: does simply stopping mining reduce power and temperature? or is it simply once the fpga is programmed its always using its max power?  The latter would explain why your pausing function is important.

I've been looking for a (cheap) usb external temperature probe system which supports multiple sensors on the same USB device. I'd like to connect the sensors to the heatsinks, and string together scripts to stop mining (or even better use your "pause" function) if temp goes above a threshold.

Thanks for the work!

I think there's some confusion about PyUSB, because of the terrible naming they chose. There are at least 3 projects with that name. What we're using is really just a python port of the FTDI D2XX drivers. A much better name would have been PyD2XX...

I'll think about the redistribution idea...

That is correct, when the FPGA is programmed it hashes continuously, whether or not it's working on nice fresh data or old stale stuff. The FPGA itself is extremely dumb in that way.

This is still a work in progress, but here's a shot of a temp sensor I'm testing out:



That little Frankenstein experiment is a temp sensor placed on the PCB right next to the FPGA and read out by the FT232R chip. The temperature is then available directly by the mining software.

So far, it seems to work well. We're planning to add a sensor like this for each FPGA on the next revision.

[fizzisist]

I've been looking for a (cheap) usb external temperature probe system which supports multiple sensors on the same USB device. I'd like to connect the sensors to the heatsinks, and string together scripts to stop mining (or even better use your "pause" function) if temp goes above a threshold.

FPGA.contract shareholder ArsenShnurkov suggested using the Maxim DS9490R with the DS18B20 for an easy USB temp sensor. Worth looking into I think, if you don't want to try wiring up a sensor like I did...

[bitcowok]

Awesome work re testing for next revision..  and thanks for the USB temp sensor info, will definitely be looking into it for current rev boards.

[2112]

Spartan 6s do not include internal temp sensors, unfortunately. The only feedback you really get is from the number of invalids, and I'm not 100% confident that using that is completely bullet proof. It also has the problem that it can take a while to get reasonable feedback on this. At 200 MHz you're getting 1 nonce roughly every 20 seconds it would take 200 seconds to get an invalid (based on 10% invalids). Still, this is a good idea to use as a backup measure. 3 or 4 minutes at an unsafe temperature is a lot better than 3 or 4 hours.

You could interleave normal work with a pre-computed work that will immediately find a valid golden nonce. So the maximum delay for discovering the hashing failure would be about two periods of search. This is usually called "watchdog task" or "continuous self-test".

[nbtcminer]

@Fizzisist:
Actually I didn't know! Now I may bother him with some of my questions BTW I've look at those instructions and it calls for non-corrosive silicone. Did you end up apply it the same way they've listed?  I'm actually thinking of trying an alternative method of apply the thermal adhesive which involves putting thermal grease on the outpart of the core (like a square) and then putting the adhesive on in the middle part. What do you think about that?

Here's the guide where I got the idea from:

http://www.d-silence.com/feature.php?id=259

P.S Just got my package in the mail! I'm very excited!!! Pics to come soon!

1.) How long did you let the epoxy cure for on the unit you ran the 200mhz bitstreams?
2.) Did you use any weights like li_gangyi? If so is there any particular method or weight you recommend using?

I was thinking of using 2 hard drives (approx 1kg) as weights and putting them on top of the heat stinks during the curing period (was going to try 12 hours). Thanks in advance for any input!

Yeah, I'm sure Li knows better than I do how to work with thermal epoxy. By the way, in case this isn't clear, li_gangyi is one of the fpgamining.com team, primarily in charge of the hardware design.

I simply followed these directions: http://www.arcticsilver.com/pdf/appinstruct/asta/ins_asepxy.pdf

I used about 6 paperback books for weight, but I flipped the board over so it was resting on the heatsinks and the books were on the back of the board. I removed the standoffs first, too. One important thing for me was to wait until the epoxy has just started to cure enough that the heatsinks didn't slide all over the place when I tried to put the weight on. Don't wait too long though, or it will cure without the pressure and not make good thermal contact!

[fizzisist]

@Fizzisist:
Actually I didn't know! Now I may bother him with some of my questions BTW I've look at those instructions and it calls for non-corrosive silicone. Did you end up apply it the same way they've listed?  I'm actually thinking of trying an alternative method of apply the thermal adhesive which involves putting thermal grease on the outpart of the core (like a square) and then putting the adhesive on in the middle part. What do you think about that?

Here's the guide where I got the idea from:

http://www.d-silence.com/feature.php?id=259

P.S Just got my package in the mail! I'm very excited!!! Pics to come soon!

There's really no need to insulate anything with silicone on this chip. These FPGAs have a raised square in the middle:



You should just apply the epoxy to that square, keeping it well away from the actual edges of the FPGA or the nearby components.

That sounds like a really interesting idea to use a combination of grease and epoxy. Please let us know how it goes for you!

[ZodiacDragon84]

Is there a way to drill a few holes in the PCB without nuking any traces to put in a chipset cooler, or something of the like? would this be a consideration for a possible future design if it is not in the current iteration?

[bitcowok]

Hi.. A couple more pictures and some stats.

I intend to make perspex "tunnel" covers for these, but cardboard it is for now.

http://jabawok.net/gallery/d/29221-2/20120204_001_001.jpg

http://jabawok.net/gallery/d/29224-2/20120204_002.jpg

Thats a reading of the top heatsink @180mhz with 7V fans

I'm still running all fans at 7V. Heres some power stats, taken from the wall point (ie including power supply losses and fans):
Non-programmed FPGAs (idle?): 17.3W
166mhz: 49.7 to 50.1 W
180mhz: 52.7 to 53.4 W
200mhz: 55.5 to 57.2 W

So 56W at 800MH/s with ~1% rejects/invalids. Remember this is 2 x x6500 boards, power supply and 5 fans total power consumption. I'm very happy with the above results!

I'm still running totally stock heatsinks/ thermal adhesive. I was hoping the "tunnel" setup as pictured above would equalize the temperatures between the top and bottom heatsinks, but it hasn't appeared to have made much difference.

The only other function of the "tunnel" is protection against fan failure. If one or even two of the fans fail, the heatsinks on both boards should still get some airflow.

And as you could probably guess form the power consumption figures above, the temp measurements are very similar between bitstreams. These are taken after the tunnel was fitted:

180mhz: ~38 lower heatsink,  ~41 upper heatsink   (30 ambient 5pm)
180mhz: ~35 lower heatsink,  ~37 upper heatsink   (24 ambient 1am)
200mhz: ~36 lower heatsink,  ~38 upper heatsink   (24 ambient 12am)
yeah... it obviously makes a huge difference what time of the day you measure the temperatures...  yes thats two measurements at 180 taken at different ambient temperatures.

*** I have noticed since I started writing this post that it appears sometimes the bitstream doesnt seem to program properly. The first time I did a run with the 166/180/200 bitstreams, I noticed my hash rate was stuck around 320 per board and power measurement was 50W regardless. After mucking around with a few bitstreams and programming back to 200mhz, I noticed power was now at 56W and hash rate at 400MH/s... further investigation required, but it appears that sometimes the bitstreams dont program properly. :/ Fizzisist: this is what was causing the strange figures i quoted on IRC. Could it be something to do with the preprocessing stage? sometimes it takes a (pre?) preprocessed file othertimes it runs a routine to preprocess. I take it its not safe to run the same script twice simultaniously to program 2 boards?

[bitcowok]

Ok I started to get some invalids at 200mhz, around 5% on each board.

Raising the fan voltage to 12V took the system to 60W, so i disconnected the top two 120mm fans which are not really needed yet, leaving the 3 "tunnel" fans operational, which lowered system power draw to 58W.

heatsink measurements with 12V Fans and 200mhz bitstream after 15mins:
33.6 lower, 35.9 upper.  (24 ambient)

invalids improved slightly, but looks like i'm going to have to order that aftermarket thermal adhesive after all.

back to 180mhz for now!

Suggestions for next hardware revision: 
- arctic silver (or whatever) as standard for "with heatsinks" model.
- keep same board footprint and mount hole positions! (you can probably work out from the above pictures why this is important to me )

[nbtcminer]

@bitcowok & Fizzisist:

I've been a little busy with work at the moment but I did manage to go through with my Thermal grease + epoxy project. Results are very positive with active cooling (a 120mm fan). So far getting around 375-400mh/s with it all going:

376.87 MH/s | 0: 3276/9/0 0.3%/0.0% | 1: 3260/9/9 0.3%/0.3% | 19h55m |

(This is using a getwork of -i 15)

BTW even with the epoxy I'm finding that active cooling + ambient room tempurature plays a huge role in mh/s stability (I have this thing running in my room where I do have heating at the moment).

I'm very likely going to let it cool down for a few hours to test out see if it helps with the curing process (hot for a few hours / cold for a few).

I'll do a few more tests then post some more results later! (I'm also toying around with air flow over the unit (i.e tunnel vs box).

Ok I started to get some invalids at 200mhz, around 5% on each board.

Raising the fan voltage to 12V took the system to 60W, so i disconnected the top two 120mm fans which are not really needed yet, leaving the 3 "tunnel" fans operational, which lowered system power draw to 58W.

heatsink measurements with 12V Fans and 200mhz bitstream after 15mins:
33.6 lower, 35.9 upper.  (24 ambient)

invalids improved slightly, but looks like i'm going to have to order that aftermarket thermal adhesive after all.

back to 180mhz for now!

Suggestions for next hardware revision: 
- arctic silver (or whatever) as standard for "with heatsinks" model.
- keep same board footprint and mount hole positions! (you can probably work out from the above pictures why this is important to me )

[nbtcminer]

Here are some pictures and an update to my cooling test:


Bottom of the stock heatsink (before cleaning* Do not attach your heatinks back with the bits of tape still on the bottom):


With the thermal paste around the core in a square shape:


With thermal adhesive in the middle:


I've got a 120mm fan blowing room tempurature air on it and things are looking not bad. Need another 12 hour to know what things are like 24 hours later. Here is what I'm seeing now:

417.06 MH/s | 0: 158/0/0 0.0%/0.0% | 1: 174/0/2 0.0%/1.1% | 57m22s |

[TheHarbinger]

That is an excessive amount of thermal compound.