Minimalist Spartan6-LX150 board

[DeathAndTaxes]

Nobody seriously looking to buy FPGA is looking to buy a single board and leave it at that.  Just like nobody runs a hashing farm today with a single GPU.  I mean a single board running @ 200MH will generate roughly $0.60 per day.  Hardly something to get excited over. It will take tens of thousands of those FPGA to displace GPU as the dominant technology.  That won't be done by fifty thousand people buying a single board it will be done by couple hundred people by hundreds of chips and that is easier done with more scalable solutions.

Regardless if you have 1 FPGA or a dozen of them you still need a host, you still need a power supply, you still need to monitor it (time value).  It is a lot easier to justify one's time if generating $60 a day than $0.60 per day.  Like I said I understand WHY FPGA is at the stage it is today.  You have to start somewhere and Serial/USB single chip FPGA boards are an easier place to start.  However just because we start here doesn't mean we will end up here.

[1713579648]

1713579648

[1713579648]

1713579648

[1713579648]

1713579648

[1713579648]

1713579648

[1713579648]

1713579648

[1713579648]

1713579648

[DeathAndTaxes]

Something like that?

http://jchblue.blogspot.com/2009/08/pico-computing-fpga-cluster.html

16 Xilinx Spartan XC3S5000 FPGAs

I think something more like this ...


Kinda hard to tell from the photo but the FPGA are on daughter boards.  There are 6 expansion slots on each side (2 are populated).  Now granted they are high end Virtex FPGA, onboard memory, and PCIe 16x connector so not economical for mining.  Still the same kind of concept could be done for lower end chips. A single large PCIe board with room for up to 12 daughter cards.  You can buy 1 board and 1 to 12 FPGA depending on your budget.

Look I hope the FGPA authors/designers don't take this as bashing.  What they have accomplished is amazing. It has really brought the pricepoint for FPGA from "pie in the sky" to expensive but viable.  I am just saying long term scalability will matter.  A board like this would allow someone to go from 1 to 48 FPGA in a single server (200 MH/s to 9600 MH/s).  Everything other than the card will be standard data center stuff.

[Dexter770221]

Nice looking monster Price is propably somwhere on stratosphere And performance rather poor. That single Spartan 3S5000 maybe can reach 50MH/s (*16=800MH/s).

[DeathAndTaxes]

I researched this and I think it is quite nice solution but not that cheap unfortunately. Eg will the costs be insane ? Power distribution in backplane ? Bandwidth in backplane etc. ? Thanks !

Yes cost will be high.  Backplanes tend to run $300 to $1000+.  The single board computer (mother board equivalent) runs another $200.  The way GPU economics work there really is no advantage to putting more than 6 or so GPU per board.  The main advantage of a backplane would be ability to put it all in a chassis but the thermal load of 10+ GPU makes that totally impossible so there really is no point.

[fizzisist]

Kinda hard to tell from the photo but the FPGA are on daughter boards.  There are 6 expansion slots on each side (2 are populated).  Now granted they are high end Virtex FPGA, onboard memory, and PCIe 16x connector so not economical for mining.  Still the same kind of concept could be done for lower end chips. A single large PCIe board with room for up to 12 daughter cards.  You can buy 1 board and 1 to 12 FPGA depending on your budget.

I actually looked into that board. They sell a similar one with Spartan 6 LX150s on it actually. The daughter boards have 2 FPGAs each. The price per daughter board? Around $3000!

Here's the pic:


The backplane card itself is about $1,500. That means the fully loaded card costs about $20,000. That's 12 FPGAs, or 1.5-2 GH/s or so. All told, $10 / MH/s! If you only bought one daughterboard, it would be $4500 for only about 300 MH/s!!

Finding this out was the final straw that drove me to start working on an FPGA board specifically for bitcoin mining. It just seemed insane that they could get away with charging so much. I knew that it could be done for less. Unfortunately, part of the cost savings was eliminating the PCIe interface.

For us, USB is a natural decision, but I'm not opposed to other designs. If you really want to build a PCIe based system like that, and think others would be interested in it, I'd be happy to work with you to design it together. Let's talk!

[aTg]

For a PCI expansion chassis with 13 connections over 2mil dollars ... we have a problem, we would have to design ourselves all hardware including PCI.

Another solution I can think of is to design an FPGA circuit that is powered by USB, one PC card for pci usb expansion could feed many circuits with multiple FPGAs



This is a quick sketch

[bulanula]

For a PCI expansion chassis with 13 connections over 2mil dollars ... we have a problem, we would have to design ourselves all hardware including PCI.

Another solution I can think of is to design an FPGA circuit that is powered by USB, one PC card for pci usb expansion could feed many circuits with multiple FPGAs



This is a quick sketch

Don't quite understand what you are saying. Why not just use USB FPGAs and USB hubs etc. I though bitcoin mining is supposed to be "ghetto".

[ElectricMucus]

Something like that?

http://jchblue.blogspot.com/2009/08/pico-computing-fpga-cluster.html

16 Xilinx Spartan XC3S5000 FPGAs

These with  16 Spartan-6 LX150s would have to come close to 3 GH/s, a miners wet dream if they were affordable. But most likely they have a huge premium on the chips themselves.

[ElectricMucus]

Why aren't there any pcie cards with only a few fpgas & some power converters on it?

That would be the most cost effective solution as modern fpgas have native pcie endnodes and pcie even has a jtag interface built in. All we need is drivers, a 2 layer pcb and a panel sheet to mount it.

Why use PCIE interfaces? USB1.1 is much better.

How better?
With pcie you could spare:
USB chip, jtag connector, power connector, programming cable
With USB you spare:
implementing the pcie endnote according to free xilinx instructions, a level shifter ic and a few CLBs

[DeathAndTaxes]

implementing the pcie endnote according to free xilinx instructions, a level shifter ic and a few CLBs

Yes, on the *T model chips that have an on board pci-e interface and which cost something like 15% more.

Then you're stuck attaching the damn things to a motherboard... and confined to the computer cases which are available or can be constructed.

Far better to have a USB (or ethernet, or RS422, or anything _not_ pcie) interface — then you can build an enclosure which meets your density and cooling needs optimally— e.g. nice linear airflow because your system is a single plane, and you can service hundreds of miner chips off a pair of standard rackmount servers.

I doubt you are going to build an enclosure including all data transfer, cooling, and power cheaper than bulk purchased 2U & 4U rackmount chassis.

Density is a non-issue if using PCIe cards.  You can radiate about 150W from an expansion slot using chassis mounted fans and passive heat sink.  That is now NVidia cools Tesla cards.  FPGA are so power efficient that 150W is something like 3GH/s.  

So there is no need to build an enclosure to "meet your needs".  The enclosure that meets your needs is a standard 4U server.  Put 16 FGPA on a single PCIe card and you got 3.2GH/s per expansion slot.  4 cards per server works out to "only" 12.8GH per 4U server.  Standard data center rack could hold 128GH.  If for some reason you needed more capacity they make Single Board Systems with 12+ PCIe expansion slots in standard rackmount chassis.  So you got standardized and commoditized expansion, cooling and power distribution options.

Granted USB is easier to get working but eventually cost effective PCIe cards will come along.  PCIe isn't that expensive.  



If a $14 NIC can implement PCIe eventually someone will figure out how to cost effectively add PCIe support to $2000 worth of FPGAs.

[DeathAndTaxes]

Have you priced out any of those backplanes?

Have you priced out the non-PCIe boards.  Hell the dual Spartan-6 150 module (just the FPGA and a couple MB of RAM) is $4000.  By that logic nobody can make a mining board for less than $2K. 

If a NIC can implement a PCIe interface for the same price as a USB interface then obviously there isn't anything magical that makes FPGA add $1500 to the cost on the board.  The prices a high because the margins are insane in semi-custom FPGA land.  The margins are equally high on that companies USB, compact flash, and serial modules too.  That doesn't tell us anything about the actual cost to implement. 

[rph]

PCI-e is just a pain in the ass for mining. You need gold finger controlled impedance PCBs, a
flash PROM to configure the FPGA before the PC BIOS scans the bus, much more complex
interface logic in the FPGA, more complex SW, and possibly a $$$$ high speed scope if
something goes wrong, just to provide insane amounts of bandwidth you don't even need.

Plus, putting 16 $150 BGAs onto a single giant PCB is a bad idea. 1 out of every 20-30 boards will have
a critical defect under 1 of the FPGAs, costing you either $2400 to scrap the board, or a bunch of time and
money to try to remove/reball/repair it. There's a good manufacturability reason to use only 1-2 FPGAs per PCB -
if something goes very wrong you're out $300, not $2000.

-rph

[ElectricMucus]

What the hell, with a high speed CPLD you can probably bitbang pcie, last time I checked those cost around $5.
Some of those can interface with almost and logic level & impendace.

The thing what concerns me is: Why use a PC anyway? We could possibly implement the mining software on a cheap ARM, pair it with a ethernet MAC ic and be done with it.

[DeathAndTaxes]

What the hell, with a high speed CPLD you can probably bitbang pcie, last time I checked those cost around $5.
Some of those can interface with almost and logic level & impendace.

The thing what concerns me is: Why use a PC anyway? We could possibly implement the mining software on a cheap ARM, pair it with a ethernet MAC ic and be done with it.

Scalability.  If you wanted to build a 20G/s cluster are you going to have 100 cases, 100 boards, 100 ARM processors, 100 ethernet ports, 100 wallwart power-supplies?

[ElectricMucus]

Just try to jam as many fpgas on the board as possible, I know the whole fault rate issue as described by rph... Just make one faulty board cost less to repair than a pc and you're done.

However: 1HE 48port switch, 47 1HE units * 64 Spartan-6 LX150 * 180MH = 541 GH/Rack
Ok costs would probably kill it the fpgas alone would cost almost half a million dollar  

[ngzhang]

PCI-e is just a pain in the ass for mining. You need gold finger controlled impedance PCBs, a
flash PROM to configure the FPGA before the PC BIOS scans the bus, much more complex
interface logic in the FPGA, more complex SW, and possibly a $$$$ high speed scope if
something goes wrong, just to provide insane amounts of bandwidth you don't even need.

Plus, putting 16 $150 BGAs onto a single giant PCB is a bad idea. 1 out of every 20-30 boards will have
a critical defect under 1 of the FPGAs, costing you either $2400 to scrap the board, or a bunch of time and
money to try to remove/reball/repair it. There's a good manufacturability reason to use only 1-2 FPGAs per PCB -
if something goes very wrong you're out $300, not $2000.

-rph

totally  agree with you.

What the hell, with a high speed CPLD you can probably bitbang pcie, last time I checked those cost around $5.
Some of those can interface with almost and logic level & impendace.

The thing what concerns me is: Why use a PC anyway? We could possibly implement the mining software on a cheap ARM, pair it with a ethernet MAC ic and be done with it.

on my design, the upper computer only need send and receive mining datas, so a hacked router is enough.

What the hell, with a high speed CPLD you can probably bitbang pcie, last time I checked those cost around $5.
Some of those can interface with almost and logic level & impendace.

The thing what concerns me is: Why use a PC anyway? We could possibly implement the mining software on a cheap ARM, pair it with a ethernet MAC ic and be done with it.

Scalability.  If you wanted to build a 20G/s cluster are you going to have 100 cases, 100 boards, 100 ARM processors, 100 ethernet ports, 100 wallwart power-supplies?

no case, 100 boards, 1 arm processor, 1 Ethernet ports, 2~3KW  12V power-supplies is ok.

[bulanula]

WOW FPGA really is picking some steam. I think ngzhang took it to the next level with his really neat professional board. Now only if the cost was reasonable

[aTg]

PCI-e is just a pain in the ass for mining. You need gold finger controlled impedance PCBs, a
flash PROM to configure the FPGA before the PC BIOS scans the bus, much more complex
interface logic in the FPGA, more complex SW, and possibly a $$$$ high speed scope if
something goes wrong, just to provide insane amounts of bandwidth you don't even need.

Plus, putting 16 $150 BGAs onto a single giant PCB is a bad idea. 1 out of every 20-30 boards will have
a critical defect under 1 of the FPGAs, costing you either $2400 to scrap the board, or a bunch of time and
money to try to remove/reball/repair it. There's a good manufacturability reason to use only 1-2 FPGAs per PCB -
if something goes very wrong you're out $300, not $2000.

-rph

That oblem is solved using a socket on the motherboard or something like what you have invented in your design.

[Albert]

I must say, months ago when I first stumbled upon the bitcoin, I lost hope in ever utilizing it as a online payment alternative because of GPU+Power costs, and the fact that the value dropped greatly within 2 months (Back when I was concerned with profiting from it, alone).

I'd be able to afford these up-and-coming FPGA boards, and would be willing to tag along with a few pools. I am devoted again thanks to these recent advancements.

What'll they think up next? 

[DeepBit]

I must say, months ago when I first stumbled upon the bitcoin, I lost hope in ever utilizing it as a online payment alternative because of GPU+Power costs, and the fact that the value dropped greatly within 2 months (Back when I was concerned with profiting from it, alone).

May be you misunderstood how the bitcoin works ?
You don't need to mine bitcoins for using it as online payment system. You just earn or buy coins when you need them.