Modular FPGA Miner Hardware Design Development

[max3t]

Exactly, the ARM will have to boot the FPGAs, unless they have a configuration flash on the board itself.

Okay. Then i request an autoupdater to a server where "bootcode" for every daughterboard is stored, so that i just have to insert my wallet address (you know.. by pushing little buttons on the backplane. it would have a little display for this, combined with btc counter!) and keep paying the electricity bills.

Thanks again.

[1714033802]

1714033802

[1714033802]

1714033802

[1714033802]

1714033802

[1714033802]

1714033802

[1714033802]

1714033802

[O_Shovah]

So maybe we can decide on the first few core points. Please tell me if you have any objections and if yes what you would use instead

- Every daughterboard has a power supply Molex 8981 and in addition one connection for a standart Laptop power supply for high power daughterboards.
  The power for lower currents is supplied through the connector Lanes of the daughterboards.

- Each board has one USB mini B port

- The connection between the Mainboard an the daughterboards is provided through DIMM sockets  ( low price, standart part ~3€) [ Number of Lanes to be determined]  

[TheSeven]

Exactly, the ARM will have to boot the FPGAs, unless they have a configuration flash on the board itself.

Okay. Then i request an autoupdater to a server where "bootcode" for every daughterboard is stored, so that i just have to insert my wallet address (you know.. by pushing little buttons on the backplane. it would have a little display for this, combined with btc counter!) and keep paying the electricity bills.

Thanks again.

I'd seriously perfer a router-like web interface for that... It needs a network connection anyway, and buttons aren't exactly comfortable to use.

[TheSeven]

So maybe we can decide on the first few core points. Please tell me if you have any objections and if yes what you would use instead

- Every daughterboard has a power supply Molex 8981 and in addition one connection for a standart Laptop power supply for high power daughterboards.
  The power for lower currents is supplied through the connector Lanes of the daughterboards.

- Each board has one USB mini B port

- The connection between the Mainboard an the daughterboards is provided through DIMM sockets  ( low price, standart part ~3€) [ Number of Lanes to be determined]  

Having a barrel connector in addition to the molex one definitely can't hurt. I'd favor 2.1mm ID, 5.5mm OD - Tip Positive, as most other devices have, not one of those more recent HP or Lenovo ones.

DIMM sockets certainly have their advantages (like locking clips), but they limit the width of the FPGA boards. For a rack-mount setup it might be desirable to have low profile (2-3U), but long FPGA cards with heatsinks in between and a couple fans on both ends providing airflow through those. PCI-style connectors seem to be suited better for that.

[Olaf.Mandel]

[...]
My suggestion would be the following:
- One of those FTDIs on every FPGA/ASIC board, with access to the JTAG scan chain, the IRQ signal, and possibly an the I2C bus
- A USB mini B connector on each board, mounted in a way that doesn't allow you to plug a cable when the board is sitting on a backplane
- Each board has a JTAG scan chain, an I2C bus and the USB interface to the FTDI connected to the backplane connector. This increases flexibility and you get it almost for free.
- Future backplane implementations may either use USB or JTAG/I2C.
- FPGA designs may choose to do everything via JTAG, or go for I2C for post-boot communication.
- I'd still be strongly in favor of a small I2C EEPROM on each card. The cost for this is neglegible.
[...]

So... you are suggesting to have the daughter boards be also operable stand alone? While that is probably possible, what is then the point of having the backplane at all? Is this the idea of adding a complete computer (ARM or other) to the backplane?

If yes, then maybe O_Shovah as the original poster should clarify what he intended the purpose of the backplane to be: just cost saving or making a stand-along hashing appliance. If it is the latter, than the host-to-JTAG interface may not be common between stand-alone and backplane based.

[Olaf.Mandel]

[...]

• The FT2232 operates bus powered (works without having +12V connected).

Since you can't power the whole FPGA-array through one USB-port and we will need an external power-supply anyway, I'd suggest to also power the FT2232 from the power supply. Why? There are potential host-devices that can't even handle the minimum 100mA. For example an Android phone with a modified kernel to support host-mode ... would also be a very power-efficient host.
[...]

Hi,

the idea behind host powered was simply to save one power supply (for 5V in the case of a FT2232D). How do these small devices provide USB host at all if they don't have 100mA? I thought the standard required at least that (and the chip would need less).

[Olaf.Mandel]

[...]
Having a barrel connector in addition to the molex one definitely can't hurt. I'd favor 2.1mm ID, 5.5mm OD - Tip Positive, as most other devices have, not one of those more recent HP or Lenovo ones.

DIMM sockets certainly have their advantages (like locking clips), but they limit the width of the FPGA boards. For a rack-mount setup it might be desirable to have low profile (2-3U), but long FPGA cards with heatsinks in between and a couple fans on both ends providing airflow through those. PCI-style connectors seem to be suited better for that.

I agree that the barrel connector cannot hurt. But what is a "standard" laptop voltage? The range seems to span from 17V to 24V (and I probably missed some here). Should the power supply support all?

Concerning DIMMs: is putting any other connectors on a board that is slotted into a DIMM socket safe? I am thinking of the stresses excerted on the socket due to the cables (or the heatsinks, if there are any).

[lame.duck]

I doubt if such  'no power' USB-Hosts are a really cost optimized solution, and  we need a Power-supply for the FPGA anyway, so there should be enough for a small arm system.

Switching between the jtag-lines of the FTDI and the jtag-lines from the is just a question of an simple TTL multiplexer,  you would require  some driving chips anyway, at least for clock and reset and you have to watch out for the propagation delay on the jtag chain. The XDS510 syste mfrom Texas instruments there is a clock feedback  pin on the connctor for this ...
Maybe there is a microcontroller with usb device funcionality that ist comparable to the FTDI price wise, that could bitbang the stream into the FPGA, and store some config data that could reported to the Host.

[lame.duck]

[...]
Having a barrel connector in addition to the molex one definitely can't hurt. I'd favor 2.1mm ID, 5.5mm OD - Tip Positive, as most other devices have, not one of those more recent HP or Lenovo ones.

DIMM sockets certainly have their advantages (like locking clips), but they limit the width of the FPGA boards. For a rack-mount setup it might be desirable to have low profile (2-3U), but long FPGA cards with heatsinks in between and a couple fans on both ends providing airflow through those. PCI-style connectors seem to be suited better for that.

I agree that the barrel connector cannot hurt. But what is a "standard" laptop voltage? The range seems to span from 17V to 24V (and I probably missed some here). Should the power supply support all?

Concerning DIMMs: is putting any other connectors on a board that is slotted into a DIMM socket safe? I am thinking of the stresses excerted on the socket due to the cables (or the heatsinks, if there are any).

Wide input converters are costly, provide some lanes with the standard ATX voltages so the DIMM designer can choose of his requirements. DIMMs for SDRAM are mostly THT, this could be  good for stability. Small heatsinks are not a problem, there are already Dimms with that, the question is how much heat the can dissipate.

From theory you could provide some areas with a defined module height so you could mount one or two metal bars to secure the DIMMs and if proper designed some bigger heatsinks.

[tschaboo]

Hi,

the idea behind host powered was simply to save one power supply (for 5V in the case of a FT2232D).

I thought that for such small currents you will get a voltage regulator for something like 30 cents?

How do these small devices provide USB host at all if they don't have 100mA? I thought the standard required at least that (and the chip would need less).

Well, they don't conform to the standard. The manufacturer puts in a chip that could support USB slave, OTG and host but they officially support only slave. By changing the driver in the linux kernel you can change the operating mode of the chip to host. At xda-developers some people did that. But they have to use a powered USB-Hub inbetween the devices because of the missing power supply from the host.

Of course that won't be a common use case. But if I have an old smartphone that is worth < 20 EUR it might be the most cost effective solution.

[Olaf.Mandel]

[...]
Switching between the jtag-lines of the FTDI and the jtag-lines from the [bus / backplane?] is just a question of an simple TTL multiplexer,  you would require  some driving chips anyway, at least for clock and reset and you have to watch out for the propagation delay on the jtag chain. The XDS510 syste mfrom Texas instruments there is a clock feedback  pin on the connctor for this ...
Maybe there is a microcontroller with usb device funcionality that ist comparable to the FTDI price wise, that could bitbang the stream into the FPGA, and store some config data that could reported to the Host.

You may not even need the multiplexers: if the FT2232 is not needed in when such a hybrid card is plugged into the motherboard, then connect its RESET# pin to the bus: all outputs of the FT2232 are tri-stated when that pin is pulled low and you can connect to the JTAG without the FTDI chip interfering.

I am not set on an FTDI product: anything that works is good! But with a microcontroller, I would ask that one is chosen that can not be bricked by wrong programming: always leave the option of reprogramming through the USB. And what about software support (you can get FT2232D JTAG support in many different programs).

Any feature requests there are for a microcontroller over an FTDI? I heard of these two, now:

• Store additional configuration data (i.e. obviate the external I2C EEPROM for board identification)
• Cheap(er) price

Anything else come to mind?

[O_Shovah]

 Seems i've started a quite lively project

Having a barrel connector in addition to the molex one definitely can't hurt. I'd favor 2.1mm ID, 5.5mm OD - Tip Positive, as most other devices have, not one of those more recent HP or Lenovo ones.

So i take it as agreed that we take this both power supply connections.The voltage supplied for the barrel connecter will still be to decide. I will insert a table in my first post with facts i asume to have been decided on.  

DIMM sockets certainly have their advantages (like locking clips), but they limit the width of the FPGA boards. For a rack-mount setup it might be desirable to have low profile (2-3U), but long FPGA cards with heatsinks in between and a couple fans on both ends providing airflow through those. PCI-style connectors seem to be suited better for that.

Basically i have a rebuild of the hardware arrangment of the copacobana setup in mind http://www.copacobana.org/photos/photo_1.jpg

As you may see they use DIMM like slots to fix the cards (wich i personally prefere) and they use both  Spartan 6 LX150 and Virtex 4-230 without any heatsink at all.
On the other hand they claim the Spartan 6 to reach 200 Mhash/s. Therefore i asume the size of the heatsink we might wish for wouldn't be an obstacle for stability.

A Fan assembley for the cards will be needed anyway. But i don't see the point in a PCI slot. Im think they got even less structural stability as they lack the fixations laps.

So... you are suggesting to have the daughter boards be also operable stand alone? While that is probably possible, what is then the point of having the backplane at all? Is this the idea of adding a complete computer (ARM or other) to the backplane?
If yes, then maybe O_Shovah as the original poster should clarify what he intended the purpose of the backplane to be: just cost saving or making a stand-along hashing appliance. If it is the latter, than the host-to-JTAG interface may not be common between stand-alone and backplane based.

My original intention was to create an FPGA slave for a Pc. The backplane just used as a saving, supply an housing unit.Altough i have to admit that a standalone solution is interesting i would postpone such for later developments if it is not possible to create a common interface for both solutions. I would like to ask you to clearify if a common interface  for both solutions is possible or not, then we may decide on that point.  


One other point that has been numerously discussed is the use of an FT2232 device simmilar http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=768-1010-2-ND
for Bus system JTAG.
I'd like to know if we can basically decide to use this chip at all or if anybody has any reasonable alternatives ?

[Olaf.Mandel]

[...]

[...]
If yes, then maybe O_Shovah as the original poster should clarify what he intended the purpose of the backplane to be: just cost saving or making a stand-along hashing appliance. If it is the latter, than the host-to-JTAG interface may not be common between stand-alone and backplane based.

My original intention was to create an FPGA slave for a Pc. The backplane just used as a saving, supply an housing unit.Altough i have to admit that a standalone solution is interesting i would postpone such for later developments if it is not possible to create a common interface for both solutions. I would like to ask you to clearify if a common interface  for both solutions is possible or not, then we may decide on that point.  

Edit: I just realised that the word standalone is used in two contexts here: the Ethernet capable backplane that does not need a host computer and the USB-based FPGA board that does not need a backplane. When I write below, I am talking about the FPGA boards.

Thanks for clarifying that. I think a common interface is not impossible, it is actually straight forward! If we decide on using JTAG (at, say 3.3V) as the bus interface for all boards, then it is just a question of how the JTAG signals are generated on the backplane. The first, simpler version could have a dumb USB chip and later people could design a backplane that instead (or additionally!) has some small CPU with Ethernet.

So what signals (apart from power) do we need on the bus? A summary of the current suggestions plus my thoughts about details. This is only suggestions to keep the discussion flowing and provide a central point to criticise to comment:

• JTAG (4-pin should suffice, i.e. no TRST). Connects all FPGA on the DIMM in a ring. Voltage should be something common, e.g. 3.3V
• I2C bus. Connects to all FPGA plus a small EEPROM on the DIMM. Voltage should be something common, e.g. 3.3V, pullups on the backplane (this means a bit of extra work for hybrid DIMMs: they need to enable their pullups in standalone mode).
• A board presence detection (two pins shorted). Can be used to have the backplane automatically connect TDI to TDO on unpopulated slots.

In case one wants to build hybrid DIMMs, i.e. boards that can either work standalone with a USB chip or as a DIMM on the backplane, the following signals may also be interesting. These are just extra, so dedicated DIMMs can still be mixed with hybrid boards:

• The RESET# pin for the USB chip. Pullup on the DIMM, connected to ground on the backplane
• The PWREN# pin for the on-DIMM power supplies. Details depend on the USB chip. Either connected to ground on the backplane or connected to the PWREN# signal of the backplane chip.
• There was a suggestion of adding the USB signal of the DIMM to the bus. Not sure if that is necessary, even if I don't see how it could hurt. If used, the RESET# pin may not be pulled low but instead the JTAG lines must not be used (i.e.: completely different backplane design).

A question remains on local power supply: What to provide through the bus connector and what to generate on the DIMM? I think TheSeven has a point when he is vary of sending small voltages at several amperes through these connectors. Does anyone have a resistance per pin for DIMM connectors? I looked at Digikey A97917-ND and 609-1001-ND, and they don't seem to specify it.

[BkkCoins]

This is my reference for DIMM sockets,

http://www.4uconnector.com/online/Itemadrawing.asp?fldseries=04677&seriesno_a=0702

It indicates 1A max and 30 milliohms contact resistance.

Here's the catalog page.

http://www.4uconnector.com/online/seriesphoto.asp?groupdesp=IC+%2F+RAM+Socket&itemnum=64&SeriesNum=10&GroupNo=07&sample=0

I didn't check price lately but they're quite cheap. $1-$2 if I recall from years ago.

RE: JTAG

I don't know about this "ring" thing. I recall it being just a regular bus. If it needs to pass thru a card and out then you just can't skip a slot. But it's hazy now since it's been years for me.

[O_Shovah]

A question remains on local power supply: What to provide through the bus connector and what to generate on the DIMM? I think TheSeven has a point when he is vary of sending small voltages at several amperes through these connectors. ....

I would strongly recommend that each board generates all needed currents and voltages on the DIMM itself as their quaility is essential to the boards stable operation.
The DIMM should be supplied with 12 V and 5 V rail via the DIMM connectors for lower power needs and Molex or barrel connector for higher power needs.
If we use a standart 240 Pin DIMM connector minus the pins for the Bus and comunication system there should remain enough pins to split up the current to a good level.   

[Olaf.Mandel]

This is my reference for DIMM sockets,

http://www.4uconnector.com/online/Itemadrawing.asp?fldseries=04677&seriesno_a=0702

It indicates 1A max and 30 milliohms contact resistance.
[...]

That is important information, thank you. So for 20 pins per voltage and the the same number for GND, we could get to roughly 3 mOhm for one roundtrip and still support 20A (we need 10A for one large FPGA). When calculating 3mOhm I also took the resistance in the GND return path into account. If you have two voltages with 20 pins each, you should have 40 pins on GND.

If we think of number of available pins: 200 pins minus 10 for dedicated DIMMS minus possibly another 10 for hybrids (both rounded up), we have 180 pins. That is enough for roughly 100A of various voltages plus GND, let's put in a fudge factor and say 45A. Now let's assume that the VCCIO and VCCAUX (for Xilinx) don't use much. This leaves enough for 4 XC6SLX150 on one DIMM! With a smaller fudge factor, the 6 FPGAs visible on the copacobana photo seem possible.

So in summary, it seems possible to build DIMMs without any power supply at all. The problem is that there is not much room for putting extra voltages on the bus to accommodate different devices. How to resolve this:

• Put only a relatively high voltage (3.3V or 5V) on the bus and require the DIMMs to have their own local supplies? This is extensible but expensive.
• Design different back planes for different FPGA types, differing only on the number and type of voltages on the bus? This is cheap but locks the user into specific hardware.

If one goes with the cheaper approach (after all: how many different FPGA boards will be available in the near future and how expensive is getting a new backplane?), there might be a way to keep the extra design effort for different boards to a minimum and keep at least a modicum of compatibility:

Partition the supply pins on the bus into different rails, say 10 pin for each rail. This gives a 5A granularity at a fudge factor of 2. That gives 9 rails plus 90 pins GND. The location of these rails are specified and are identical across all FPGA types and backplane types. It is "just" a question of what voltage to put on each rail. The layout of the bus part of the baseplate can stay the same in all cases. It might even be possible to put different smaller FPGA boards (that don't use all rails) together in one backplane because the different voltages are on different rails.

[Olaf.Mandel]

I would strongly recommend that each board generates all needed currents and voltages on the DIMM itself as their quaility is essential to the boards stable operation.

Ok, matches much better with my desired standalone/hybrid FPGA boards: basically it costs very little effort to transfer a design from a DIMM to a standalone board with an USB chip. One could even build the hybrid version as a standard then (just leave the clunky power connectors unpopulated by default).

The DIMM should be supplied with 12 V and 5 V rail via the DIMM connectors for lower power needs and Molex or barrel connector for higher power needs.
If we use a standart 240 Pin DIMM connector minus the pins for the Bus and comunication system there should remain enough pins to split up the current to a good level.   

What do you need two voltages for? If you decide now on only one, you have the simplest of all designs. The only disadvantage is that you use only half of your ATX power supply... And if you go the Laptop power supply way, then you have only one voltage, anyway.

I would opt for 5V on the bus: yes, the current is higher, but how big a DIMM do you want to build that it requires 5V*45A > 200W? On the other hand, it means one less voltage supply ion the DIMMs: I used the National Semiconductor Webbench to design the power supply for my prototype and when starting from 12V, it (nearly) always goes the two-stage way. It first steps down the 12V to either 4V or 5V and in a second step it steps that further down to 1.2V, and 2.5V. So unless there is FPGAs out there that require 5V stabilised input, then I opt to remove the 12V from the bus.

For a hybrid card, this would mean it needs a 5V input or you need to add the extra step down stage which remains unused if the card is plugged into the bus.

[lame.duck]

Well, the often shown picture from the Copacobana  shows 6 FPGA in the FT256 BGA, this is 17mmx17mm. The successor with  Virtex4 has bigger chips and as far i remember the photos from a prototype photo showing a more than oversized dimm card.

Having no heatsink on the photos is no argument to me that no one is needed. Maybe they did produce the bitstreams without exceeding the allowed power dissipation by using a reduced clock rate rather than the theoretical limit with cooling.

[TheSeven]

3 milliohms for the roundtrip? Including inner resistance of the voltage regulator, PCB traces and FPGA pins, you just can't manage to stay below 5 milliohms this way. Power supply stability is absolutely critical for those FPGA boards. Putting this on the backplane probably isn't going to be cheaper in the end. You seriously want to put a 1.2V 300A power supply on the backplane? Good luck routing the PCB. So you'll have to split the power supply anyway, and whether you do this on the card or on the backplane doesn't matter price-wise. On the power efficiency side you're definitely better off with higher voltages on the DIMM connector.

Also, why on earth would you want a two-step power supply? (12V => 5V => 1.2V)
This greatly increases the load on (and thus cost of) the 5V supply and reduces efficiency.
Either you'll get both 12V and 5V from an ATX PSU (in which case you'll want to use 12V for FPGA supply voltage generation, as it provides the higher wattage on modern PSUs and 5V for all those small FTDIs or whatever), or you'll want to do 12V => 5V and 12V => 1.2V if you want a barrel connector.

Ideally the 12V rail would actually be 10-20V, or at least 12-16V, to allow for some of those laptop power supplies to be used.

Oh, and there is definitely no way to dissipate up to 15 watts from such an XC6SLX150 without a heatsink.

[Olaf.Mandel]

3 milliohms for the roundtrip? Including inner resistance of the voltage regulator, PCB traces and FPGA pins, you just can't manage to stay below 5 milliohms this way. [...]

I was only talking about the connector, not the rest. And it seems like the decision to add power supplies to the DIMMs has been made.

Also, why on earth would you want a two-step power supply? (12V => 5V => 1.2V)
This greatly increases the load on (and thus cost of) the 5V supply and reduces efficiency.
Either you'll get both 12V and 5V from an ATX PSU (in which case you'll want to use 12V for FPGA supply voltage generation, as it provides the higher wattage on modern PSUs and 5V for all those small FTDIs or whatever), or you'll want to do 12V => 5V and 12V => 1.2V if you want a barrel connector.

I am not an electrical engineer, so I rely on what National Semiconductors Webbench tells me about the capabilities and limits of their products. I looked there because of their handy Webbench program: if someone knows what they are doing power-supply-wise, a different manufacturer may be better. But at least for National, going in two steps and having a rail voltage of 4V or 5V seems to increase efficiency by 10% to 15%. I cannot say why that is. Maybe their product portfolio is not optimised for 12V to 1.2V in one step?

Ideally the 12V rail would actually be 10-20V, or at least 12-16V, to allow for some of those laptop power supplies to be used.

Oh, and there is definitely no way to dissipate up to 15 watts from such an XC6SLX150 without a heatsink.

Agreed, having at least some of the cheap power supplies compatible to this design seems like the thing to do.

So if the DIMMs need heatsinks, that means no tight spacing of DIMMs is possible. That means it doesn't hurt to add the power connectors to hybrid boards by default.