X6500 Custom FPGA Miner

[ZodiacDragon84]

So how hard you guys think we can push these things? I would love to put them BFL folks in a competitive place if their hardware exists. With proper cooling, and well regulated power, whats the theoretical limit? I plan on buying one of these and water cooling it when my tax return comes this week, I will offer a bounty if someone can squeeze 500 M/hash out of these. (250 MHz each chip I believe)

You can see more in-depth conversation about trying to squeeze additional performance over at the ztex thread -- my read on it is that 200MHz may be close to the max you'll get from the LX150. There is a little noise about upping the voltage on the chips to try to get a tiny bit more performance (~4%) but it's a good way to blow the chip if you screw up.

Hmmm, wonder how much tweaking in the code could increase efficiency. I figure you guys are already all over that though 

[njloof]

Hmmm, wonder how much tweaking in the code could increase efficiency. I figure you guys are already all over that though 

They seem to be doing a pretty good job of optimizing the use of the chip to maximize its throughput given the existing algorithm. ArtForz likes dropping hints about improvements to the structure of the code that they haven't found yet. He was ahead of the FPGA curve by about a year.

[ZodiacDragon84]

art sounds like an FPGA ASIC god...lol

[freshzive]

anyone found an enclosure that will work for these things yet?

[bitcowok]

Hi, I've been lurking for ages and only just worked out the 5 post minimum rule... ("wheres the frigging reply button?" )

Heres a shot of the enclosure i have just built, which will eventually have 30 x6500's in it.

http://jabawok.net/gallery/d/29202-2/20120129_001.jpg

the rest of the images are here if anyone is interested:
http://jabawok.net/gallery/v/random/201201-x6500-30x-enclosure/

thanks to fizzisist for the double stacked fan tower idea. Only difference with my layout is the stack is on its side so the heat will be rising naturally (and sucked out the top with the big fans)  Air inlet is through the bottom  of the case and lower side vents.

Temp measurements at 166mhz are 39 degrees for the lower heatsinks and 41 degrees for the upper heatsinks. (measured with an IR temp sensor from hobbyking)

All fans are running at 7V at the moment. I plan to put plastic or cardboard sides and top covers to tunnel the air properly, which will hopefully see less temperature difference between the heatsinks.

I'll probably get some power and more temp measurements etc done soon at various hash rates and post back. (I've just changed to 180mhz, 413/6/1 and 433/2/0.. looks ok!)

[fizzisist]

Hi, I've been lurking for ages and only just worked out the 5 post minimum rule... ("wheres the frigging reply button?" )

Heres a shot of the enclosure i have just built, which will eventually have 30 x6500's in it.

That is awesome! I'm thoroughly impressed. I've been working on something similar, but not nearly as nicely done or well planned. I'm hoping to fit a few towers inside a case that's also housing a PSU and mini-ITX mobo. I got as far as removing the hard drive cage and drilling a few holes for the X6500s.

How are you mounting the fans? I was having trouble there, because I thought I would use angle bracket but the bracket is too tall that it blocks some of the air flow.

I love how carefully you routed the cables. When you have 5 board stacks, I highly recommend these types of splitters: http://cablesaurus.com/8-slot-4-pin-molex-splitter, but make sure you switch to 12V at the input of that. If you leave all the boards on 5V, you're drawing potentially 4*5 A through the first connector.

[rjk]

According to the thermal image that was posted earlier, the hot air from one chip was blowing over the other chip, causing the second chip to be warmer than the first. Would it be possible to turn the heatsinks 90 degree sideways, so that the airflow goes over both chips in parallel, instead of in series? It seems that that would ensure the longest lifespan.

Then, you could stack them up in a high-rise configuration and just keep going up up up!

[torusJKL]

[...]
How are you mounting the fans? I was having trouble there, because I thought I would use angle bracket but the bracket is too tall that it blocks some of the air flow.
[...]

He straps them down using some kind of cable through holes near the fans.
See this picture.

According to the thermal image that was posted earlier, the hot air from one chip was blowing over the other chip, causing the second chip to be warmer than the first. Would it be possible to turn the heatsinks 90 degree sideways, so that the airflow goes over both chips in parallel, instead of in series? It seems that that would ensure the longest lifespan.

Then, you could stack them up in a high-rise configuration and just keep going up up up!

The problem with this is that you have the molex connection on one side and the usb on the other.

[thirdlight]

the miner is so stable since the last big changes

All my boards are running the 200MHz bitstream, and have been up for 4 days now.

All cores giving ~0.8% rejects, except 2: AH00WOW1 core 1 is 10%, AH00WOW4 core 0 is 3.5%, so my lowest performing chip is returning 180MH/s.

Fantastic product!

[freshzive]

394.73 MH/s | 0: 8017/163/153 2.0%/1.8% | 1: 8091/122/52 1.5%/0.6% | 2d2h43m | AH00WOWI

Stales/rejects are slightly higher (~1-2%) with 200mhz, but hashrate is steady around 390Mhz now. Awesome!!

I had one FPGA that was behaving poorly (~10% rejects). Popping the heatsink off and reseating it solved the problem....you may want to try that. Heat seems to be directly correlated with the amount of stales/rejects.

[nbtcminer]

Just wanted to drop in and introduce myself; I'm new to the FPGA scene and have recently purchased an X6500 (Dominator)! I'll be getting my unit either late this week or early next week and wanted some basic advice on the unit:

1.)I'm currently shopping for Thermal Adhesive; before my adhesive arrives would it be safe to run my unit with the 180mhz bitstream and active cooling (I plan on having a 120mm fan actively cooling the card)?

2.) For people who own the unit with the stock heat sinks; do you find the stock sinks to be enough? Do you have any other ones you might recommend?

I plan on playing around with various cooling schemes for my x6500 (which I will share in later post with pics!) and if the mining goes well I may get a second unit!


Cheers,
nbtcminer

[shad]

[...]
How are you mounting the fans? I was having trouble there, because I thought I would use angle bracket but the bracket is too tall that it blocks some of the air flow.
[...]

He straps them down using some kind of cable through holes near the fans.
See this picture.

i think those are
    wire tie
    cable fixer
    lace
    cable tie
don't know the right translation
i also use them for my seconed fan

and some stats
418.06 MH/s 0: 12488/127/1 1.0%/0.0% 1: 12506/131/14 1.0%/0.1% 3d3h27m

395MH/s over all 3 days

[bitcowok]

How are you mounting the fans? I was having trouble there, because I thought I would use angle bracket but the bracket is too tall that it blocks some of the air flow.

Yep just cable ties through holes drilled in the motherboard mounting plate.

When you have 5 board stacks, I highly recommend these types of splitters: http://cablesaurus.com/8-slot-4-pin-molex-splitter, but make sure you switch to 12V at the input of that. If you leave all the boards on 5V, you're drawing potentially 4*5 A through the first connector.

Yep I bought one of those, which I'll use when I get more boards. The plan was to switch 12/5V pins on 5 of the 8 connectors (labeling them clearly!) making 5 x 12V and 3 x 5V to correctly load the power supply.

I'm also planning on perspex sides and tops for the fan tunnel, and an idea i had yesterday was some sort of fast strobing LEDs on each fan so i can visually check everything is turning. Failing that, just standard leds. And for visual checks, I'll cut a big square out of the side panel of the case for perspex.

Temp measurements with fans @7V, x6500's @ 180mhz, and no side tunnel walls on the fans: 41C lower and 43C top,  ~1% rejects.

fizzisist: Do the fgpa's have a temp sensor in them we can read back? Could you make a command line option for a "pausing" temperature threshold while the fpga is above a certain temp? 

Failing that, could you make a command line option reject rate pausing threshold? ie 3 parameters, possibly with the following defaults: if the reject/invalid rate within the last 30s goes above 10%, indicating something is most certainly wrong, pause for 120s to let things cool a bit. 

This throttling will no doubt save many fpga's from disaster as more people start pushing them harder, and fans start failing over time etc.

[A1BITCOINPOOL]

What cables are needed for an fpga.  Do they come with it or do I have to buy them seperate.  Thanks.

[freshzive]

What cables are needed for an fpga.  Do they come with it or do I have to buy them seperate.  Thanks.

It doesn't come with cables. You need mini-USB to USB cable (http://www.amazon.com/AmazonBasics--Male-Mini-B-Cable-Meters/dp/B001TH7GUK/ref=sr_1_2?ie=UTF8&qid=1327971683&sr=8-2). And either a power supply with a free molex connector or a barrel power adapter http://cablesaurus.com/ac-adapter-12v-20a-100-240v-50-60hz .

[A1BITCOINPOOL]

Are they accepting bitcoins as payment.

[bitcowok]

Heres something for Linux users I worked out a while back but haven't had a chance to share yet.

As most people running Linux know, writing to system folders manually (outside of your distro's package management system), is a seriously bad idea.

That is (from http://fpgamining.com/software at time of writing), this:

Code:

Install the FTDI D2XX drivers

and this:

Code:

sudo python setup.py install

bad, bad, bad! 

Heres how to run fizzisist's pyusb and the FTDID2XX driver stack without polluting your system folders with files your package manager doesn't know about.

I have directories and some scripts set up as follows, under my fpgaminer user's home directory:
./bin
./bin/list_x6500s.sh
./bin/program_x6500_0.sh
./bin/program_x6500_1.sh
./bin/mine_x6500_0.sh
./bin/mine_x6500_1.sh
./x6500/fizzisist-x6500-miner-007fcc8      (contains unzipped zipball of latest git code)
./x6500/bitstreams                     (all the *.bit files)
./x6500/libftd2xx1.0.4                 (contains untarred ftdi d2xx driver)
./x6500/PyUSB-1.6-fizzisist-linux   (contains fizzisist's modified pyusb, untarred)


My fpgamining user is a member of my distro's "usb" group which gives it permissions to write directly to the correct usb devices in /dev, without needing to use sudo at all.

Heres a cat of one of my program_x6500 scripts:

Code:

#!/bin/bash
FPGAROOT="/home/fpgamining/x6500"

cd "$FPGAROOT/fizzisist-x6500-miner-007fcc8"

LD_PRELOAD="$FPGAROOT/libftd2xx1.0.4/build/i386/libftd2xx.so.1.0.4:$FPGAROOT/PyUSB-1.6-fizzisist-linux/build/lib.linux-i686-2.6/d2xx/_d2xx.so" PYTHONPATH="$FPGAROOT/PyUSB-1.6-fizzisist-linux/build/lib.linux-i686-2.6/" python program.py -d 0 ../bitstreams/ztexmerge_180mhz.bit

Note the trick of setting "LD_PRELOAD" environment variable, which lets you manually specify to preload libraries which are not in the system library search path, and the PYTHONPATH, which does the same thing for non-system python files.

voila: I only needed sudo to useradd the fpgamining user.  Everything runs non-root, and without breaking my distro-installed pyusb etc, because it's all nicely self contained in one directory under /home.

for completeness sake: heres one of my mining scripts:

Code:

#!/bin/bash

#for permissions to the usb device, this user needs to be part of the usb group
FPGAROOT="/home/fpgamining/x6500"

cd "$FPGAROOT/fizzisist-x6500-miner-007fcc8"

LD_PRELOAD="$FPGAROOT/libftd2xx1.0.4/build/i386/libftd2xx.so.1.0.4:$FPGAROOT/PyUSB-1.6-fizzisist-linux/build/lib.linux-i686-2.6/d2xx/_d2xx.so" PYTHONPATH="$FPGAROOT/PyUSB-1.6-fizzisist-linux/build/lib.linux-i686-2.6/" python mine.py -d 0 -u api.bitcoin.cz:8332 -w myworker:sensored_secret

I'm running gentoo linux, if anyone was interested, but the above should apply for almost any linux distro.

[bitcowok]

Are they accepting bitcoins as payment.

Yes they accept bitcoins, check out cablesaurus website mate, they have the usb cable and everything else you could need.

[freshzive]

Except that they seem to be sold out of x6500s currently

[fizzisist]

Heres something for Linux users I worked out a while back but haven't had a chance to share yet.

Beautiful! I knew I was doing things in a bad way, but didn't know a good way to do it. I'm going to follow your recommendations when I set up a new machine this week. If it works out well for me I'd like to incorporate it in the user guide on the website, if that's alright with you.

Thanks for sharing!