Use Beagleboard/Pandaboard to run FGPA mining rig

[ArtForz]

Im talking total power draw measured using a kill-a-watt.  You figure in the ineffeciency of the PSU, HDD, and the mobo, you will see around 50 watts.

PSU? What else are you going to run your FPGAs off of? A few dozen <75% efficient chinese wall warts?
HDD? USB stick or CF card in PATA adapter. Even a 7.2k 2.5" is <1W spinning.
Mobo? I give you that, 3.5W vs. 20W peak ...
So we're talking about 430 vs 450W for 10Gh with S6s, 23.3 vs. 22.2 Mh/J ... not exactly a massive deal.
For something with efficiency similar to BFLs vaporboards, it's even less of a difference, 2-3% max.

Now, what something like a pi does have going for it is power (see above, not *that* big of an issue), cost and size. = A lot cheaper, and way easier to tuck into a corner somewhere.

Benefits of a generic x86 ... can install any random OS, way easier to set up a full dev toolchain without having to fiiddle with crosscompiling, > 256MB RAM. Also potentially a lot faster and has GigE (somewhat pointless here...).

So yeah, if I wanted to build a few dozen 10Gh rigs something like the pi as a controller would make sense... for a one-off... too much work, just slap a x86 in.

[1713574634]

1713574634

[1713574634]

1713574634

[1713574634]

1713574634

[BkkCoins]

My GA-970-D3 with Athlon draws < 25W. My Jetway mini-ITX about 20W.

Using a Raspberry-Pi is more for the novelty of it and just finding a use for one when they finally are available.

To really save power... here's my current project: a PIC18F86J65 (cost $3), a RJ45 with MAGS ($1), a 25MHz crystal ($.50) and a few passive parts. The only real power draw here is driving the ethernet signals. And a simple PIC miner program that relays work to the FPGA cluster of up to 128 devices.

PCBs done, just waiting to send to fab. Overhead cost per FPGA < $21 qty 1, <$16 qty 25.

@ArtForz - I'd be interested in comments on PCB/Design if you were willing to look. I can post PNGs here.

[rjk]

My GA-970-D3 with Athlon draws < 25W. My Jetway mini-ITX about 20W.

Using a Raspberry-Pi is more for the novelty of it and just finding a use for one when they finally are available.

To really save power... here's my current project: a PIC18F86J65 (cost $3), a RJ45 with MAGS ($1), a 25MHz crystal ($.50) and a few passive parts. The only real power draw here is driving the ethernet signals. And a simple PIC miner program that relays work to the FPGA cluster of up to 128 devices.

PCBs done, just waiting for fab. Overhead cost per FPGA < $21 qty 1, <$16 qty 25.

@ArtForz - I'd be interested in comments on PCB/Design if you were willing to look. I can post PNGs here.

I would love to see your design on this - have you considered making it PoE powered?

[runeks]

Hi

I'm looking into creating a rig using FGPAs.
To control them I would like to use a Beagleboard-xM or Pandaboard ES.

The Beagleboard-xM has a 1GHz ARM Cortex-A8.
The PandaBoard ES has a Dual-core ARM Cortex-A9 MPCore with Symmetric Multiprocessing (SMP) at 1.2 GHz each.

Has somebody used these boards?
How many FGPAs do you think it will able to manage?

Thanks.

I've heard reports of the Pandaboard being unstable. I've heard better things of the Efika MX Smarttop. One user, I remember, reported an uptime of 30 days, which would have been more if he didn't have to reboot the machine for a kernel upgrade. This seems to me to be of chief importance in a mining rig configuration. The Smarttop only sports an 800 MHz Cortex-A8 though. For pooled mining that should be more than sufficient to drive several FPGAs, as far as I know. If you're talking p2pool, where you have to do transaction verification, initial block chain download might be slow (though we can just move the block chian over from another computer), but at the current transaction rate I think it should be able to keep up with incoming transactions. It has an onboard cryptographic HW processor too, but I'm not sure if this does ECC-DSA.
It comes with Ubuntu 10.10 installed by default, but they have a Debian hardfloat image up on their support/community site (http://www.powerdeveloper.org/).

[rjk]

I've heard reports of the Pandaboard being unstable. I've heard better things of the Efika MX Smarttop. One user, I remember, reported an uptime of 30 days, which would have been more if he didn't have to reboot the machine for a kernel upgrade. This seems to me to be of chief importance in a mining rig configuration. The Smarttop only sports an 800 MHz Cortex-A8 though. For pooled mining that should be more than sufficient to drive several FPGAs, as far as I know. If you're talking p2pool, where you have to do transaction verification, initial block chain download might be slow (though we can just move the block chian over from another computer), but at the current transaction rate I think it should be able to keep up with incoming transactions. It has an onboard cryptographic HW processor too, but I'm not sure if this does ECC-DSA.
It comes with Ubuntu 10.10 installed by default, but they have a Debian hardfloat image up on their support/community site (http://www.powerdeveloper.org/).

If you are going this in-depth and installing a whole OS, I prefer VIA's Pico-ITX formfactor, in particular the ARTiGO A1150 (Dual core VIA Eden X2, x86_64 compatible, max 4 GB RAM). I've got the single core version (The A1100), and it is damn sweet - just be sure to install the video drivers, otherwise the thing is slower than molasses due to video being processed on the CPU instead of the chipset.

It supports expansion boards, and includes I/O headers for tons of things including serial ports, plenty of USB ports, standalone LCD displays, and tons more. Kind of expensive, but compact and reliable.

[runeks]

^ Installing the OS is as simple as downloading an image and booting it. The Smarttop is $129 (€129 in Europe with free shipping). But I admit to it being somewhat of an experiment. It's possible that you will hit some sort of bottleneck in the system if you're attaching 10 FPGAs to it doing 5 GH/s in total. I know memory bandwidth in particular is pretty poor. But if I were to build a mining rig consisting of only a single or a few FPGAs, I'd definitely go for this. It uses 5-10W, is cheap as hell and stable. Also, I think ARM is cool .

[rjk]

^ Installing the OS is as simple as downloading an image and booting it. The Smarttop is $129 (€129 in Europe with free shipping). But I admit to it being somewhat of an experiment. It's possible that you will hit some sort of bottleneck in the system if you're attaching 10 FPGAs to it doing 5 GH/s in total. I know memory bandwidth in particular is pretty poor. But if I were to build a mining rig consisting of only a single or a few FPGAs, I'd definitely go for this. It uses 5-10W, is cheap as hell and stable. Also, I think ARM is cool .

Well, BkkCoins' post up at the top of this page makes x86 (and ARM) look like power hogs by comparison. I would like to know how well his design can scale, and how much power it uses.

[runeks]

^ I must admit that I'm satisfied with the 5W that the Smarttop consumes. Personally, I'd be worried about the upgradability of custom hardware. What if I want to do p2pool mining on BkkCoins' hardware, where I have to verify transactions and store the block chain?
Also, Smarttops are ready to ship now, with working software. It seems BkkCoins hasn't even had hardware built yet. But if OP is not in a hurry, and he wants to save a couple of watts, it might be worth waiting for, cause we won't get lower power than a custom design like that.

[BkkCoins]

My GA-970-D3 with Athlon draws < 25W. My Jetway mini-ITX about 20W.

Using a Raspberry-Pi is more for the novelty of it and just finding a use for one when they finally are available.

To really save power... here's my current project: a PIC18F86J65 (cost $3), a RJ45 with MAGS ($1), a 25MHz crystal ($.50) and a few passive parts. The only real power draw here is driving the ethernet signals. And a simple PIC miner program that relays work to the FPGA cluster of up to 128 devices.

PCBs done, just waiting for fab. Overhead cost per FPGA < $21 qty 1, <$16 qty 25.

@ArtForz - I'd be interested in comments on PCB/Design if you were willing to look. I can post PNGs here.

I would love to see your design on this - have you considered making it PoE powered?

I did consider POE but it was too much hassle. And since there is power from the FPGA board I chose the economy approach. The first FPGA board provides 3.3V power to the Ethernet PIC via it's I2C connector. Simple.

I did the design in Eagle on Linux. I don't mind sharing the design files but since they use a custom library I'm not sure it's the easiest for a quick view. So I've exported a rather large PNG of the schematic, and then converted a couple Gerber views to PNG for easy viewing.

Some Notes:

This is based on an Icarus compatible FPGA design. I tried pretty hard to synthesize and place my own version of the Verilog so I could optimize everything into the FPGA (no on board MPU) but kept on having failures (maybe due to some Ubuntu issues or whatnot). Finally I decided to just adapt to the serial input on the Icarus design. The Ztex uses the CSBGA and I can't see myself working with 0.65 pitch layout on a 2 sided board. That's even more crazy than just attempting a 2 sided board in the first place.

Power is from a good quality PSU. It uses 12V, 5V, and 3.3V input. 5V is used as bias for FAN2108 DC converter, 12V -> 1.2V, and 3.3V is used directly for 1 FPGA bank and dropped to 2.5V for VCCAUX and other banks (as required by Icarus design).

Each (5cm x 5cm) FPGA board can act as master for up to 3 slaves. They tile together like square dominoes, up to 127 boards (limited by I2C address, seems like enough to me). There is a PIC16LF1503 on board that queues work/results and acts as multiplexer for CFG init and work data. It co-ordinates data from the Ethernet PIC and passes it off to slaves. It talks via bitbanged serial to FPGA. This chip was chosen as it has a CLC (configurable logic block) inside allowing routing high speed CFG data via itself without external parts. It talks to master and slaves using I2C.

There is fan control, temperature monitoring on each board. It uses the PWM of the PIC and internal temp sensor. Not the most accurate but likely good enough for this.

The Ethernet PIC is master of the whole cluster. It gets work, feeds it to the first FPGA slave, and requests work back. It has a 64Mb Flash chip on board to store FPGA CFG. I'm building these parts as modular units so that I can use them for other purposes.

So, here is PNG version of schematic and PNG version of PCB layouts with screenprint overlay.
This is just the FPGA board now. I can post the EthernetPIC/Flash board soon.

Miner Schematic
Top Layer
Bottom Layer

I'll draw up a quick overview diagram so it makes more sense how these connect together in a cluster. I'm going to send the Gerbers off for making 10 boards once I'm more confident this is a final layout. As you can see this is my 4th revision (miner4)!

[rjk]

BkkCoins, epically awesome. So does the program that gets flashed to the PIC also contain stuff like what pool to mine on, and IP address, etc? Or am I misunderstanding how it will be used?

[BkkCoins]

BkkCoins, epically awesome. So does the program that gets flashed to the PIC also contain stuff like what pool to mine on, and IP address, etc? Or am I misunderstanding how it will be used?

The program for the Ethernet PIC (MasterPIC below) would have a web interface like your router. You can login and edit the pool values (no reason it can't have a few for failover). Also, later should be able to view some stats/graphics, though haven't thought much about that yet. It would have an upload button for updating firmware for both itself and each FPGA, and PIC too. Both these model PICs support self-writing to their EEPROM allowing online updates. Only the initial programming needs to be done via programmer to install bootloader. That's why little space is given to ISP connectors.

Each FPGA-PIC has simple code to inspect I2C data, queue work data for FPGA, or relay on to slaves. During power up it does a process of device enumeration to set addresses for whatever devices are connected. Then it disconnects itself and passes thru CFG data until "Done" on each FPGA tells it to reconnect as I2C relay. This code has to fit in 2K so it's assembler and minimal. Since Spartan 6s are happy with hotplugging I don't see a reason why devices couldn't be added/removed while running, with minimal disruption, as long as my addressing code adapts. It would have to rescan every few seconds maybe.

Here is a PNG of the overall idea of how you would connect a cluster of these units. Note that each FPGA board has a Molex MicroFit connector (small!) for power. The I2C connectors don't provide power to slaves (each has it's own cable to PSU). But the master connector (top one, red) has power on it to provide power to the EthernetPIC.

[JusticeForYou]

watching

[BkkCoins]

Here's a few more images to check out.

Schematic of MasterPIC.
Schematic of Flash support for MasterPIC.
PCB Top Layer MasterPIC.
PCB Bottom Layer MasterPIC.
(I may still replace Xtal with Oscillator pkg on Net2Blox. I like the small size but it costs about 50 cents more.)

I've created them as little I2C/SPI modules allowing them to be used for other things. I tentatively call these modules "2Blox" as I may market them later.

Net2Blox and Flash2Blox. The Flash2Blox allows using either W25Q64 64Mbit flash devices or alternately a microSD card. I haven't decided my choice yet but designed board to work with either. If the microSD can't output at 16Mbps then I'd choose the chip instead. I don't want it to take too long to load FPGA CFG. At 16Mbps it's approx 2 seconds to do the whole cluster.

The Flash2Blox has a gated oscillator so the Net2Blox (MasterPIC) can set it up, enable the clock, and let the CFG fly out to the slaves from the Flash - very fast.

These little 2Blox boards stack on top of each other. So the Flash sits just on top of the Net one. And the Net one plugs into the first FPGA slave.

This is very much a work-in-progress.

[BkkCoins]

Here's a couple diagrams to stimulate the imagination...
(the blue number is the chain level, ie. data passes thru this number of nodes to worker)

Cluster of 8 - takes 6" x 10"



Cluster of 37 - takes 14" x 14" (and probably a big fan over top!)

[rupy]

Im talking total power draw measured using a kill-a-watt.  You figure in the ineffeciency of the PSU, HDD, and the mobo, you will see around 50 watts.

No, the processor draws 13W for the one I have and 10W for the next generation. If you compare that to the ztex FPGA 8W it's really ok.

The peripherals will draw on any other card too.

And the linux you get on any non x86 architecture has NO software support.

The choice is easy!

[runeks]

Im talking total power draw measured using a kill-a-watt.  You figure in the ineffeciency of the PSU, HDD, and the mobo, you will see around 50 watts.

And the linux you get on any non x86 architecture has NO software support.

That's wrong. If you buy a Smarttop from Genesi there is a software support community at http://www.powerdeveloper.org/. Of course, if you're not interested in playing around with a new architecture, which will have issues that are not present on firmly established archs like x86, it's probably not for you.

And come on dude, do you really know all vendors of Linux ARM hardware well enough to say that none of them offer software support? It's a bold statement to make at least, and pretty much impossible to verify.

[Red Emerald]

I'm going to have to get an FPGA and see if I can plug it into my chumby.

[torusJKL]

I use a mini-itx atom D510MO as desktop and run my fpga from that, economical, silent and better for the environment and with proper linux!

How many FPGAs do you run?
How much is theCPU load?

[rupy]

Just try to use your ARM linux as a workstation with java, eclipse, chrome WITH flash support and spotify, skype, wine etc. If you wan't to do anything else in your life you won't be able to.

[rupy]

I use a mini-itx atom D510MO as desktop and run my fpga from that, economical, silent and better for the environment and with proper linux!

How many FPGAs do you run?
How much is theCPU load?

Now I run 5 via USB, but its only 1 java process that controls the cluster. CPU is 0% for that process... It would be close to 0% for the 128 I could run with the limitations of USB.