Modular FPGA Miner Hardware Design Development

[Olaf.Mandel]

Personally it doesn't look too bad to me, I just need to check that we are meeting requirements for decoupling capacitors.

GND on layer 2 would be best, however we'd have to put in blind vias to link up the top to 2nd in order to still make the rest of the space usable. I might have gone overboard when testing it out though, I had vias going from almost every node to the 2nd layer.

I'll add in the PSUs to the new board layout later.

Edit: Ok added it in, we're gonna have to touchup the IO routing abit, let me finish up the rest in the next few hrs.

@li_gangyi: I looked at your FPGA_pin_combined.brd. and I have a few questions:

• When you changed the routing for the termination resistors, is there a specific reason why you went in from the other side? Both seemed equally good from my point of view, so I am confused.
• Do you think the individual schematics and board layouts are at a level that merging them makes sense? I thought there is quite a bit to do for my board at least, so there are a few iterations there. And once it is merged we cannot work on the thing at the same time.
• Otherwise we have to find a way to collaborate while working on the same files. Working on different files is a first step, because then we don't undo the other persons work. If we work on the same file, maybe we can ask here when a file is "free" to work on?

I wanted to do the following to the FPGA board, maybe this evening:

• Shift GND to layer 2 and VCCINT to layer 15 as suggested by pusle, li_gangyi and fizzisist and shift VCCIO and routing to layer 4.
• Look at the bus routing to move the wires further apart

[li_gangyi]

No specific reason why I went in from the other side, for the termination resistors. I've done up the PSUs for both Vccints, feel free to move them around if there is a need to.

We need to roughly combine the work together, else there's no way to put in the rest of the parts, i.e I need to ripup alot of your work to get stuff to fit in.

I've also reworked the PSU and the FPGAs to bond their GNDs to the 2nd layer pls check the layout. Vccio is mostly routed except for the SMPS module itself.

[O_Shovah]

Maybe just to prevent one of you to overwrite te work of the other .

I created two folders for each of you respectivly so you may save your variant in you folder.

I also propose we  add date and time to the file name so we may backtrack them a bit.

[Olaf.Mandel]

Changed the daughterboard block diagram on dropbox a bit:

• Added a separate power supply for the MSP (delete again if we decide to just reset the FPGAs instead)
• Explicitly wrote SPI instead of data for the bus

[grid]

I'm taking the plunge and getting the ZTEX LX150 board to teach myself about FPGAs, and provide feedback on the real-life performance of the LX150.

I'm a software developer and electronics hobbyist at heart, but FPGAs are a different animal completely so support by the experienced people here will be much appreciated   

[fizzisist]

I finally had time to do some reading about the differences between all of the MSP430F5xx's. To summarize, the primary differences is the size of the flash memory and the SRAM. The amount of memory needed should to be estimated by someone who will write the code (most likely not me, I've never programmed an MSP430 before). If we don't have an estimate, we should err on the side of more memory, because it probably doesn't influence the price all that much.

The other differences basically boil down to the number of ADCs and IOs. ADCs will be useful for monitoring temps and voltages.

The 5504 has only 8 kB flash memory and 4 kB ram, most likely not enough. If you move up to the 5507, we change nothing except increase the flash memory to 32 kB. There are 8 ADCs and 31 total IOs on these.

If we need more memory or IOs, the next up is really the 5524. This one has 64/4 kB (flash/ram), 10 ADCs and 47 IOs.

The top of the line is really the 5528. This one has 128/8 kB, 10 ADCs, 47 IOs. This one would definitely work for us, even if it was a little bit overkill. 

Just looked at the prices and it seems like the 550x series is all out of stock, but is around $5. The 552x series is available, and is more like $10. The extra $5 is negligible on this board, and gives us much more wiggle room, I think.

O_Shovah, I unfortunately don't have Eagle available so I can't look at your schematic. Can you make a PDF or image of it?

[TheSeven]

I finally had time to do some reading about the differences between all of the MSP430F5xx's. To summarize, the primary differences is the size of the flash memory and the SRAM. The amount of memory needed should to be estimated by someone who will write the code (most likely not me, I've never programmed an MSP430 before). If we don't have an estimate, we should err on the side of more memory, because it probably doesn't influence the price all that much.

The other differences basically boil down to the number of ADCs and IOs. ADCs will be useful for monitoring temps and voltages.

The 5504 has only 8 kB flash memory and 4 kB ram, most likely not enough. If you move up to the 5507, we change nothing except increase the flash memory to 32 kB. There are 8 ADCs and 31 total IOs on these.

If we need more memory or IOs, the next up is really the 5524. This one has 64/4 kB (flash/ram), 10 ADCs and 47 IOs.

The top of the line is really the 5528. This one has 128/8 kB, 10 ADCs, 47 IOs. This one would definitely work for us, even if it was a little bit overkill. 

Just looked at the prices and it seems like the 550x series is all out of stock, but is around $5. The 552x series is available, and is more like $10. The extra $5 is negligible on this board, and gives us much more wiggle room, I think.

8 ADCs should be plenty. What's more interesting is the number of USARTs (at least 2, 3/4 would be optimal). I'd also think that 64kiB of flash should be plenty, as this one really doesn't have to do anything complex, just relay data between the backplane/USB and the FPGAs.
Oh, and how can these MSPs be flashed, or how easily can they be bricked? Please provide a standard 2.54mm header (or at least the PCB pads for one) for the pins needed for reprogramming.
Is there a readily-made USB bootloader for them that could be enabled either via USB itself or via a jumper? How big is that one?

[fizzisist]

Well, limiting the amount of flash memory could end up being a pain, and the difference in price between the 5524 and 5528 is something like 10% more, I think the added flexibility is well worth it. Who knows what kind of functionality might be added to these boards in the future!

These come preprogrammed with the USB bootloader. We should certainly break out the pins needed to reprogram it in case it is somehow overwritten. I think that to enable the bootloader, you need to pull the /RST pin low, so we need to connect that to a button or a jumper. Someone please make sure I'm understanding that correctly, don't take my word for it!

[fizzisist]

Anyway, the important thing is that we move from the 5504 to a 552x, so the schematic will need to change.

[li_gangyi]

I don't know how much memory the final design will need, but at the start if we want to just use the MSP430 as a glorified USB to Jtag adapter the current part should be fine, if we wanna say load the bitstream and everything from the uC, I don't think it'll be a good idea, we should just leave that for the motherboard design stage.

Basically I don't see a need to move to a bigger/more powerful MSP.

[magik]

I'd like to try to help with this project, as I'm a firmware engineer ( FPGA programmer - Xilinx ) by trade, and I'd love to dabble in the FPGA bitcoin mining world.  If you guys want I could run some of your schematic designs by our mechanical engineer that does our layouts, see if you guys are missing anything or anything else that jumps out.  It seems a bit early for the FPGA work, and that stuff already seems a bit worked out by others ( Open source FPGA project, etc... ).  But I'd love to help in any way I can - this project just sounds magical, and I don't think I've ever really seen an open source FPGA/hardware project of this magnitude before ever.

I find it a bit hard to figure out where you guys are right now without access to this shared dropbox folder and without reading this entire thread.  But what's the current status of everything right now?  Where do you guys need help/work right now?  I am also a bit of a software/hardware engineer as well and might be able to help out in some of those areas as well?

In regards to the uC talk above - if all the uC is going to do is route work from the USB comm path to the FPGAs, and the reverse - I'm not sure why a powerful uC is even needed.  You could probably get away with doing away with the uC completely and doing the USB chip interface on the FPGA?  Maybe something like the "master" platter controls the USB comms and then communicates to the other FPGAs via bus.

I probably need to brush up a bit on what exactly you guys have planned for the design, how you plan on implementing things from a higher level perspective.

But I'd love to help in any way I can.

[magik]

if we wanna say load the bitstream and everything from the uC, I don't think it'll be a good idea

At my work we use a lot of Spartan 3E 1600s and the bit files are somewhere around 1.6 megabytes....  No idea how large the bit files are for the 6 series....  USB could definitely handle it - but then you have to also think about the uC being able to handle it in a reasonable amount of time too.

From what I've seen typically for every FPGA you would have a SPI flash memory for holding the programming - and on boot, the FPGA's bootloader can be told to talk to the SPI flash chip and unload it's programming from there.

I was dabbling around with some code to field-upgrade the firmware in the flash and the Spartan 3E itself doesn't even have enough BRAM to hold the entire flash image on the FPGA itself - I'm going to have to go down the road of possibly having two flash chips, one with some "bootloader" code and the other with the actual running program - and to field upgrade the firmware, I would have code that would be able to access the 2nd flash to write directly to it.

But I guess if you program the chips through the USB comm interface that's 1 less part you'll need ( flash memory ), but it also means much more complicated uC code to be able deal with programming the FPGAs.  And it also means that if the device loses power, that it would need to be reprogrammed again - and the software communicating through USB would need to be able to catch this, or be restarted.  I guess it could work, but in my mind it just sounds kind of hacky.

[phillipsjk]

It's worse than that: we want to be able to bring down the FPGAs deliberately if they are drawing too much power. How much data would need to be moved across the bus every time we do this? So far the assumption is that bandwidth is negligable. Edit: 1.6MB can take minutes at standard serial speeds.

Another reason for using the MCU is that it saves space on the FPGA for hashing. The MCU can translate between USB and SPI for example.

Edit2: MCUs can also be reprogrammed without software costing $3000 to produce the bitstream. This makes things like minor protocol changes or calibrating the built-in temperature sensor easy.

[magik]

It's worse than that: we want to be able to bring down the FPGAs deliberately if they are drawing too much power. How much data would need to be moved across the bus every time we do this? So far the assumption is that bandwidth is negligable. Edit: 1.6MB can take minutes at standard serial speeds.

And that's a spartan 3E i'm talking about - I imagine the 6 series has a lot more cells that need to be programmed.

Another reason for using the MCU is that it saves space on the FPGA for hashing. The MCU can translate between USB and SPI for example.

Edit2: MCUs can also be reprogrammed without software costing $3000 to produce the bitstream. This makes things like minor protocol changes or calibrating the built-in temperature sensor easy.

It's fairly easy to get an evaluation license for Xilinx - and I think it's even pretty easy to just re-generate more if you have more e-mail addresses....  Nobody in the industry really pays for Xilinx licenses - they just give them away when you buy chips typically - but the bad news is that it's unlikely they will even give you a license if you aren't purchasing semi-bulk.  I would guess it'd be kind of hard for hobbyists to nab a non-eval license.

So what exactly is the purpose of powering down FPGAs?  If this was my mining rig, I'd be running it at 100% at all times.  I see you mentioned temperature sensor - so maybe in places without adequate cooling or if a cooling system breaks you might want to turn things off... again I think I need to see a bit more of what you guys have designed so far.

The amount of FPGA space you will use for simple things like comm protocols is nearly negligible - especially when these modules are running at much reduced clock rates compared to the important stuff ( the hashing engine ).  Another thing to think about - see what other FPGAs in the family have the same foot-print.  A prototype board doesn't need to have the most expensive FPGA if a cheaper smaller FPGA can handle 1 hash engine, and this may help a little in the first round of prototyping.

I guess if the MCU is very cheap it's not that much of a hit to have one... but you are adding one more layer of complexity to the design. PC <==> USB chip <==> MCU <==> FPGA.  And the MCU itself will need to have code written for it...  I guess it matters what you really want the MCU to be able to accomplish.  Or what you think the MCU may be able to accomplish in the long run.

I think I need a bit more info on what the high level design architecture is going to be.  How does the backplane/motherboard fit into this if each DIMM module has it's own MCU and USB controller?  I would imagine ideally you would put these type of comm things onto the backplane/mobo itself - but I understand in terms of prototyping and development it'd be easier to have it on the module itself so you wouldn't need a backplane to debug/develop with it.

Also, I'd like to mention that I'd be willing to throw up a bit of cash ( couple hundred dollars ) when prototyping time comes.  And I am interested in helping with the project.  I have access to Xilinx ISE tools, and all kinds of hardware equipment ( logic analyzers, power supplies, o-scopes, sig gens, etc... ).  I am also a hobby programmer with experience in C/C++, Java, python, perl, PHP, etc... Trained as an EE in school, but focused more on computer engineering, hence why I'm now a firmware engineer.  But yeah I'd love to help develop this with you guys and I have some money I could pool together with you guys to get the first round running.  I don't do much of the hardware layout - but I can always ask my coworkers to take a look at things.

[phillipsjk]

I suppose to quickly answer questions like yours, the first post of the thread should be kept up to date.

The back-plane is supposed to be optional. There will be a USB port on the modules that is inaccessible while plugged into the back-plane. This is to allow stand-alone operation.

AFIAK, the point of the back plane is to make managing several boards easier. I envision powering down the chips being useful in off-grid applications where power may be at a premium, depending on weather conditions. It may also be useful if you want "standby" hashing power to protect the network, but you would probably use (cheaper, more power hungry) general-purpose machines for that.

I was going to say my post was the first mention of a temperature sensor, but it actually wasn't. I don't think any temperature-sensing scheme is finalized yet. I just noticed the MCU has that feature when reading the datasheet (+- 20C).

I then would like to state that for the current design we are going to use 2 FPGA's per DIMM PCB at maximum, not more !
In addition the motherboard will hold up to 5 DIMM boards

- Post 92

I'd say that a DIMM may draw up to 35 watts from the backplane. This should be sufficient for all 2-FPGA and some 4-FPGA cards. If a card needs more, it should use dedicated Molex/PCIe connectors. This allows for 2-card backplanes with just an ATX20 connector, 4-card with ATX24, or up to 7 cards with ATX20 + P4, up to 9 cards with ATX24 + P4, or up to 13 cards with ATX24 + EPS12V.

- Post 184

Let me try to collect the different suggestions and comments made on bus design till now:

Protocols suggested to be on bus:

• JTAG
• I2C
• SPI
• USB

- Post 198

I wanted to get an idea what the space would look like on this board. This shows two LX150s in the FG484 package and the MSP430 in the QFN64 package. This FPGA is available in a smaller package, CSG484, but I think this one fits nicely, and a larger pitch means easier job routing the signals to the pins. For comparison, this package is 23 mm square, the smaller one is 19 mm square.

- Post 235

I had a qoute from pcbcart for a 4 layer board of our size with gold finish and 240 connection pinns for the DIMM ind 1.2 mm thikness resulting in 18 euro per board  for a quatity of 10 boards.

- Post 280

As for license preferences: I would be happy with many of the OSI approved licenses, but GPLv3+ or maybe GPLv2+ would be my preferences.

- Post 351
(I don't think the preceding has been formally decided yet.)

I uploaded a new version of the board to both dropbox and github:

- Post 374 Update: Post 416

I don't think anybody has volunteered to program the MCU yet. In theory, I could do it if I get my computer situation straightened out in the near future.

[Olaf.Mandel]

It's worse than that: we want to be able to bring down the FPGAs deliberately if they are drawing too much power. How much data would need to be moved across the bus every time we do this? So far the assumption is that bandwidth is negligable. Edit: 1.6MB can take minutes at standard serial speeds.
[...]

This is not quite right: the bandwidth needed for mining is negligible. The bandwidth available through either of the two interfaces still in the designs (JTAG and SPI) is much higher. Assuming 2MHz clockrate on SPI (the default value in master mode), we need 17s to bootload the FPGA. As the FPGAs have the same bitstream, they can be loaded in parallel. For a higher clock speed we are faster, e.g. for 33MHz we need only 1s.

[Olaf.Mandel]

So what exactly is the purpose of powering down FPGAs?  If this was my mining rig, I'd be running it at 100% at all times.  I see you mentioned temperature sensor - so maybe in places without adequate cooling or if a cooling system breaks you might want to turn things off... again I think I need to see a bit more of what you guys have designed so far.

It's for cases of having a network outage or your computer crashing, ...

The amount of FPGA space you will use for simple things like comm protocols is nearly negligible - especially when these modules are running at much reduced clock rates compared to the important stuff ( the hashing engine ).  Another thing to think about - see what other FPGAs in the family have the same foot-print.  A prototype board doesn't need to have the most expensive FPGA if a cheaper smaller FPGA can handle 1 hash engine, and this may help a little in the first round of prototyping.

A prototype board needs at least the XC6SLX75, because of the housing and because of the way you need to roll the miner up to fit inside. And the price difference to the 150 is small enough compared to the constant board costs (power supply, ...) that I would suggest going full hog on the first try. Maybe not both FPGAs, but at least the correct size.

I guess if the MCU is very cheap it's not that much of a hit to have one... but you are adding one more layer of complexity to the design. PC <==> USB chip <==> MCU <==> FPGA.  And the MCU itself will need to have code written for it...  I guess it matters what you really want the MCU to be able to accomplish.  Or what you think the MCU may be able to accomplish in the long run.

The MCU is the USB chip, so there is no overhead there. The main ideas for not putting interface logic into the FPGA: the debugging is easier for non-HDL specialists, the FPGA can be brought up without needing a programmer cable (which hobbyists may not have!), the MCU can run on a different power rail than the FPGA and power up and down the FPGA, it can act as a brownout detector and temperature guard, there is no need for a dedicated USB chip when using a suitable MCU.

I think I need a bit more info on what the high level design architecture is going to be.  How does the backplane/motherboard fit into this if each DIMM module has it's own MCU and USB controller?  I would imagine ideally you would put these type of comm things onto the backplane/mobo itself - but I understand in terms of prototyping and development it'd be easier to have it on the module itself so you wouldn't need a backplane to debug/develop with it.
[...]

Hybrid design: reduces entry level investment because you don't need the backplane for your first DIMM and the cost for the extra MCU, Mini-USB and Molex connectors together is small compared to the DIMM cost.

[O_Shovah]

Im not here one evening and i get a whole new page to read

Just looked at the prices and it seems like the 550x series is all out of stock, but is around $5. The 552x series is available, and is more like $10. The extra $5 is negligible on this board, and gives us much more wiggle room, I think.

O_Shovah, I unfortunately don't have Eagle available so I can't look at your schematic. Can you make a PDF or image of it?

Maybe we should decide on a MSP we will use first. I case we take something bigger than the 550x series (wich seems very likely now) i will have to reroute it anyway.
As far as i know there is a free version of eagle avaidable. I doesnt have many features but should be sufficient for inspecting the files.
I case it doesn't work for our files i will upload a png of the new schematic. 

@ magik:
Also for you, just send me  pm with you email adress so i may invite you to the dropboxfolder .

[O_Shovah]

I just rewired most of the MSP430 in two versions now.

I uploaded a MSP430f550x and a MSP430F552x version into the layout MSP430 folder so we may choose the one wich seems best fit.

Currently i would prefere using the 5528 version. It certainly is an overkill but wouldn't make for a big price difference ang gives us plenty of overhead recources.

I also added a PNG for thos without Eagle.

[Olaf.Mandel]

I uploaded a new version of the FPGA schematic, board and PDF+PNG to github and dropbox. Commit log:

Defined layers 2 and 15 as supply layers, removing them from the routing. This required merging VCCINT0 and VCCINT1 again. Additionally, it now was cheap to add a net NC for all unconnected pins of the FPGA.
    
All big components are on the mil 25 grid, just the small caps on the back of the FPGAs are on a local 0.5mm grid around their FPGA.
    
Unfolded the busses. Now they need more space but should be "safe". Moved around the termination resistors.

Unfortunately, after I finished I realised that I had an anulus that was slightly too small (0.2mm instead of 0.25mm). And that was the straw that broke the camels back... So I reduced the anulus to 0.1mm (the next pcbcart size).

Do people like this layout with the supply layers or should we go back to using polygons in layers 2 and 15?