Modular FPGA Miner Hardware Design Development

[Olaf.Mandel]

[...]
I was under the impression that the use of DIMM sockets for power distribution wouldn't work well and that power connectors on each card are going to be used. Unless "power distribution" means power distribution for interface logic in this context.

No, you can nominally put 1A through each of the 240 pins of the thing. Even leaving a 50% safety margin, this equates to 720W per DIMM at 12V, which should be more than enough . I calculated only 120 pins because you need the same number for GND return. And with 720W provided and 10W needed, I think we can spare a few pins for other unnecessary things like communication and such...

Are these DIMMS going to be keyed such that any memory chips placed in them won't get fried?

No. It would be nice to be able to do so, after all there are many different key positions. The problem is finding sockets that support a key position that is not used by a memory DIMM. So you are basically limited to the sizes given e.g. in this Micron TN-04-55. As for pinout: a memory DIMM has not a single pin that can stand 12-20V.

[1714235164]

1714235164

[TheSeven]

[...]
I was under the impression that the use of DIMM sockets for power distribution wouldn't work well and that power connectors on each card are going to be used. Unless "power distribution" means power distribution for interface logic in this context.

No, you can nominally put 1A through each of the 240 pins of the thing. Even leaving a 50% safety margin, this equates to 720W per DIMM at 12V, which should be more than enough . I calculated only 120 pins because you need the same number for GND return. And with 720W provided and 10W needed, I think we can spare a few pins for other unnecessary things like communication and such...

Nevertheless, routing that much current on the PCB won't be trivial. You would basically have to place five 8-pin PCIe power connectors around the slot!

I'd say that a DIMM may draw up to 35 watts from the backplane. This should be sufficient for all 2-FPGA and some 4-FPGA cards. If a card needs more, it should use dedicated Molex/PCIe connectors. This allows for 2-card backplanes with just an ATX20 connector, 4-card with ATX24, or up to 7 cards with ATX20 + P4, up to 9 cards with ATX24 + P4, or up to 13 cards with ATX24 + EPS12V.

Are these DIMMS going to be keyed such that any memory chips placed in them won't get fried?

No. It would be nice to be able to do so, after all there are many different key positions. The problem is finding sockets that support a key position that is not used by a memory DIMM. So you are basically limited to the sizes given e.g. in this Micron TN-04-55. As for pinout: a memory DIMM has not a single pin that can stand 12-20V.

They may not be specified for 12V, because nobody in the memory industry would ever want to do that, but I'd expect them to be able to withstand 12V perfectly well. Some extra precautions (all 12V pins on one end, ending on a key, or something similar) can't hurt though.

[makomk]

The reason why I reneged on my original suggestion of using the FT2232H: there is a ton of free software to use the FT2232D as a JTAG interface. There seems to be little to none that uses the FT2232H, though. We couldn't use existing JTAG software with the FT2232H. At that time, other protocols in addition to JTAG were only being discussed as a way to read out the EEPROM, and I figured that could be done by bitbanging. But if you want to send workloads to the FPGA via I²C, then a dedicated MPSSE seems in order.

You sure about that? I know off-hand that at least the Dangerous Prototypes Bus Blaster uses the FT2232H, and it can be wired up to act as one of several common JTAG interfaces.

[Olaf.Mandel]

[...] there is a ton of free software to use the FT2232D as a JTAG interface. There seems to be little to none that uses the FT2232H, though. We couldn't use existing JTAG software with the FT2232H. [...]

You sure about that? I know off-hand that at least the Dangerous Prototypes Bus Blaster uses the FT2232H, and it can be wired up to act as one of several common JTAG interfaces.

Didn't know what chip they used. I just found a lot of hits for the FT2232D when I searched for JTAG adapters and none for the FT2232H. The "little to none" in my second sentence was more a "(little to) none". Seems I was too hasty. And the third sentence is then clearly wrong.

[lame.duck]

They may not be specified for 12V, because nobody in the memory industry would ever want to do that, but I'd expect them to be able to withstand 12V perfectly well. Some extra precautions (all 12V pins on one end, ending on a key, or something similar) can't hurt though.

But this would require massive power traces on the DIMM to the Voltage regulators, and massive power traces to the Vint Pins and the ground traces should be largish too ...

[Olaf.Mandel]

They may not be specified for 12V, because nobody in the memory industry would ever want to do that, but I'd expect them to be able to withstand 12V perfectly well. Some extra precautions (all 12V pins on one end, ending on a key, or something similar) can't hurt though.

But this would require massive power traces on the DIMM to the Voltage regulators, and massive power traces to the Vint Pins and the ground traces should be largish too ...

How is this related to putting the 12V on one side of the board? You have the same issue for the on-DIMM connectors. As for power traces: the 12V traces shouldn't be so bad (only a 10^th of the current relative to V_int). And the V_int supply should probably be done as a large supply polygon, same as GND.

[lame.duck]

Its common and good  practice to lay out the power (and ground) traces in a tree like fashion with the root at the output of the voltage regulator output so the RF ripple of large power drains doesnt affect the other devices. Having massive polygons is the right way, but if you have to to lay out some other traces this would make holes into this polygon. While the impact on the resistance will be low, i don't know if it's still true for the impendance. While learning by doing isn't a  bad idea, the cost per miss are substanaly in this case.

[fizzisist]

I'm finally out of the newbie purgatory, so I wanted to mention that I'm following this thread closely. I use FPGAs for medical imaging systems (also use GPUs for image reconstruction) so the applications in Bitcoin mining make for an interesting hobby for me. I'd like to contribute to this project if I can in any way.

[fpgaminer]

I go away for a few days and you guys make me read 5 more pages   

I am confirming makomk's work with the EP4CE75F23C7 chips ($227.46 USD); I just compiled his fully unrolled design and it fit into 73,400 LEs at the correct Fmax performance. That is with a JTAG interface, and 2K extra LEs for interface changes. JTAG is fairly area inefficient, so there is plenty of room for alternative interfaces like I2C or SPI. Personally I think the FPGA needs a small work queue anyway, regardless of the interface.

I will attempt a compile for the EP4CE75F23C8N ($181.97 USD) and EP4CGX150CF23C8N ($295.60 USD) some time later. I will probably run the 150 overnight, because I am sure that compile will take a very long time.

I'm sure that someone in the community would love to set up such a service, where you upload your design, get back the bitstream a couple of hours later if it succeeded, and pay per processing time.

I would be happy to compile FPGA mining designs for this platform, and others that people create. Luckily I have an extra machine to do the compiles on  I own an LX150 dev kit, and the necessary license to do compiles for it, so as long as the license allows me to distribute bitstreams everything should be fine. At the very least I can verify designs until someone is confident enough to fork out the cash for their own license.

[Olaf.Mandel]

[...] I own an LX150 dev kit, and the necessary license to do compiles for it, so as long as the license allows me to distribute bitstreams everything should be fine. At the very least I can verify designs until someone is confident enough to fork out the cash for their own license.

Is that for an LX150 or for an LX150T? The cheapest devkit I found (AES-S6DEV-LX150T-G) seems to have the software device-locked for the LX150T. The price-difference between the chips at digikey is a bit over 17EUR, and just under 16EUR at avnet.

[fpgaminer]

Is that for an LX150 or for an LX150T? The cheapest devkit I found (AES-S6DEV-LX150T-G) seems to have the software device-locked for the LX150T. The price-difference between the chips at digikey is a bit over 17EUR, and just under 16EUR at avnet.

It's an LX150T, and you may be right that the license won't let me compile for the preferred LX150. These are things I dislike about Xilinx ... :/

On a related note, for the specific device, there is a -N3 variant:
http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=XC6SLX150-N3FGG484C-ND

It's cheaper, because it doesn't have the on-chip memory controlled (which an FPGA miner does not need); $158.75 USD, 60 MOQ, as of this post. That's $111 Euros in a direct conversion, so a few dollars cheaper than the one listed in the OP right now. The same applies to the other Xilinx chips, so look for the -N3 variants of those as well.

[O_Shovah]

Hello Everybody

The poll is finished.We have got a clear decission. The Xilinx Spartan 6 LX150 is the winner with 17 votes angainst 7 votes for the Altera Cyclone IV 75k.

So the FPGA is decided on. We shall proceed with the BUS system development.

JTAG and I2C in combination seem to be the most promising aproaches as i've read so far.
But details are to be disscussed further i guess.

[Olaf.Mandel]

[...]
It's an LX150T, and you may be right that the license won't let me compile for the preferred LX150. These are things I dislike about Xilinx ... :/

Can you check, please? And also look which variants (transceivers, memory controller, speed grade, package) are supported. If the license is good for all LX150 then I might also be interested in purchasing the developer board... If the license is not valid, you probably don't even see the other chips in the drop-down lists for the device settings. But if they are there, please do a test-compile to make sure. Thanks!

On a related note, for the specific device, there is a -N3 variant:
[...]
It's cheaper, because it doesn't have the on-chip memory controlled (which an FPGA miner does not need) [...]

Didn't think of that one. So we have six parameters to select the correct chip, right?

Code:

       XC6SLX   150   T   -   N   3   CSG484   C
                 ^    ^       ^   ^     ^      ^
                 |    |       |   |     |      |
# of LEs       --+    |       |   |     |      |
Serial Transc. -------+       |   |     |      |
Memory contr.  ---------------+   |     |      |
Speed grade    -------------------+     |      |
Package        -------------------------+      |
Temp range     --------------------------------+

And we want the XC6SLX150-N3CSG484C, right? It's 111.29EUR at Avnet, but is's non-stock. The cheapest on-stock chip with speed grade 3 I can find is the XC6SLX150-3FGG484C: 344 available at Avnet for 122.68EUR and 120 available at Digikey for 134.75EUR.

[O_Shovah]

Hello

Having decided on the FPGA (XC6SLX150-N3CSG484C),we should decide our BUS system now.
Seems it has been mostly dicussed to use a combination of iC2 and JTAG.We should determine in wich way and wich extend we will use them and wich hardware needed.

Please proceed with this step without me.
I will be back with you on friday afther my next test.

[fizzisist]

[...]
It's an LX150T, and you may be right that the license won't let me compile for the preferred LX150. These are things I dislike about Xilinx ... :/

Can you check, please? And also look which variants (transceivers, memory controller, speed grade, package) are supported. If the license is good for all LX150 then I might also be interested in purchasing the developer board... If the license is not valid, you probably don't even see the other chips in the drop-down lists for the device settings. But if they are there, please do a test-compile to make sure. Thanks!

I can't speak to the license you get with that board, but I can build designs for the LX150-N3. I have a full license for the Xilinx suite. If anyone needs test designs built, let me know.

[Olaf.Mandel]

Let me try to collect the different suggestions and comments made on bus design till now:

Protocols suggested to be on bus:

• JTAG
• I²C
• SPI
• USB

Voiced preferences (by name, alphabetically sorted), Edited:

• lame.duck: Have I²C or SPI, as JTAG produces errors sometimes. SPI is preferred. Opinion on USB?
• O_Shovah: Have JTAG, SPI and USB on the bus.
• TheSeven: Put JTAG, I²C and USB on the bus, use I²C as an option to communicate with FPGAs to have some flexibility. Use of busses on backplane: USB exclusively at first, the others later for intelligent backplanes. Don't connect all DIMMs into one long JTAG chain, use several independent ones. Don't use SPI because of pin-count and single-master design.

Please correct me if I misrepresented your preferences!

My own opinion after some waffling back and forth: USB as the preferred connection. No strong opinion of SPI vs. I²C. Adding JTAG and any serial interface to the bus complicates the DIMM design, as there is no easy way to tri-state the FT2232D/Hs outputs (the datasheet is not very clear on this: can someone confirm this?). So while not adamantly against JTAG and serial on bus, I think that implementing USB in the CPU of an intelligent backplane is probably better than adding tristate drivers to the DIMMs.

[O_Shovah]

Hello

I've read into the BUS matter a littel and came to the conclusion i will prefere the SPI system with independent slaves over the I2C.
The dulplex could be used for flashing I/O and sensor (temperature?) communication.
It provides plenty of bandwith overhead and higher frequencies(lower latencys)at a fairly low price.
Also i've been told by some collagues at university that SPI is a lot easier to implement in comparison to I2C.


I found a IC wich would provide us with JTAG, SPI (also I2C if needed) and USB:  http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=296-27306-6-ND

Please have a look and tell me if it seems suitable (i dont know if its overpowered).

@Olaf.Mandel    I would also like to put the USB, SPI and JTAG  on the BUS

[Olaf.Mandel]

[...]
The dulplex could be used for flashing I/O and sensor (temperature?) communication.

Can you rephrase that? Which duplex do you mean and what does "flashing an I/O (port?)" mean?

[...]
I found a IC wich would provide us with JTAG, SPI (also I2C if needed) and USB:  http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=296-27306-6-ND

Please have a look and tell me if it seems suitable (i dont know if its overpowered).
[...]

Either I missed something, or it only has a JTAG slave, not a JTAG master. You could emulate a master by bit-banging GPIOs, but that is slower (compared to a dedicated serial engine). As I see it:

Advantages over FT2232H:

• Can handle things like buffering the next workload until FPGA is ready (minimum dead time). This could be done in VHDL on the FPGA, but in C it is easier (for me at least).
• Tri-stating any IOs should be easy.
• Includes ADC for voltage monitor and temperature guard.
• Can act as a client towards the DIMM bus (e.g. via SPI) and pass the data through to internal busses (whatever format).
• No need for a dedicated EEPROM (I²C or otherwise): functionality can be integrated here.
• USB example code available from TI after registering.
• IO voltage of 2.5V possible.

Disadvantages compared to FT2232H:

• Not ready made: you need to read through the example code and write your own firmware.
• Not sure about this, but: Possibility to brick your DIMM by accidentally overwriting the BSL (can rescue with JTAG pins?).
• Did I mention you have to write your own firmware? This is worth two entries in this list
• No JTAG master serial engine (as mentioned above).

If using an MSP, maybe the MSP430F5528IRGCT may be a better choice: the housing is smaller and it is a few cents less.

Any other comments about using an MPU instead of a purpose-made USB interface? Maybe someone wants to add their favourite MPU family? I did a very little programming with a smaller MSP430 before, but this would be a much larger code base for me.

All in all it is interesting, if we find example schematics in an application note and if the programming turns out to be manageable. If we have an intelligent chip on each DIMM, though, why do we need so many protocols on the bus?

[O_Shovah]

Either I missed something, or it only has a JTAG slave, not a JTAG master. You could emulate a master by bit-banging GPIOs, but that is slower (compared to a dedicated serial engine).

Ok, i didn't see that  i only had a brief look into the documentation. This would mean we need another IC as the masterfor JTAG.

Disadvantages compared to FT2232H:

• Not ready made: you need to read through the example code and write your own firmware.
• Not sure about this, but: Possibility to brick your DIMM by accidentally overwriting the BSL (can rescue with JTAG pins?).
• Did I mention you have to write your own firmware? This is worth two entries in this list
• No JTAG master serial engine (as mentioned above).

• Not ready made: you need to read through the example code and write your own firmware.    ->I haven't had a look on the volume of code needed but i asume this may be bearable
• Not sure about this, but: Possibility to brick your DIMM by accidentally overwriting the BSL (can rescue with JTAG pins?).  -> I asume this may be rewriten by the JATG
• Did I mention you have to write your own firmware? This is worth two entries in this list    -> same as above i'm not aware of the workamount needed yet
• No JTAG master serial engine (as mentioned above).  -> if used on every DIMM as slave device we could use a IC either just supporting JTAG master or both JTAG and SPI master on the motherboard

Using an MSP like yours also seems to be a good idea.I would also like to have somebody more experienced making a suggestion.

[Olaf.Mandel]

Either I missed something, or it only has a JTAG slave, not a JTAG master. You could emulate a master by bit-banging GPIOs, but that is slower (compared to a dedicated serial engine).

Ok, i didn't see that  i only had a brief look into the documentation. This would mean we need another IC as the masterfor JTAG.

No, just use a software implementation of JTAG! If you use a fancy general purpose chip like the MSP430, you should only use extra chips for interfacing if you really have to. Let's assume 20 clock cycles for each state change of a bit-banged interface (may be low, may be high: does anyone have a more educated guess?): if you clock the MSP430 with its max clock of 25MHz, this gives you a JTAG clock of 625kHz. Even for the default MCU clock of 8MHz, you still get 200kHz (f_JTAG=f_MCU/(2*20)). That may be good enough, if you can pack the bit-banging into that tight a loop.

[...]

• Not sure about this, but: Possibility to brick your DIMM by accidentally overwriting the BSL (can rescue with JTAG pins?).  -> I asume this may be rewriten by the JATG

I looked into this a bit further: the BSL is write protected, but the protection can be removed by software. After removing it, any garbage can be written into the BSL. This can normally be recovered via JTAG, unless you wrote garbage into the BSL!!! Specifically, the JTAG interface gets permanently disabled if memory-locations 0x17FC-0x17FF of the BSL contain anything but all 0s or all 1s. I have never had an MSP430F5xx in hand, can someone else confirm this?

• No JTAG master serial engine (as mentioned above).  -> if used on every DIMM as slave device we could use a IC either just supporting JTAG master or both JTAG and SPI master on the motherboard

[...]

I still feel that if an MCU is put on the DIMM, you shouldn't route the low level communication to the FPGAs onto the bus. Only connections to the MCU should go there. Let's say: USB for the dumb backplanes and SPI for backplanes with their own CPU, or no USB at all. It's a decision between putting a USB hub chip on the simple backplanes or putting an MCU on the backplanes. In general I just want to reduce the ways how we communicate with the DIMM, as more protocols means more testing required.