Modular FPGA Miner Hardware Design Development

[ngzhang]

@ ngzhang:

No personal offence, but your global setup highly differs from the one we decided on(Power supply, USB, Bus, routing). Turning to this one now would throw us back to stoneage.

There must be some misunderstanding.
Our purpose is slightly different here. The works of yours is facing directly to a simplest platform for FPGAmining, easily to development. But I am a Ph.D candidate, majored in high performance computing, so in my concept, go a little bit further  for universal use, and cost reduction.
But there are still most of the important parts keep alike. So I think we could push each project forward in the meantime, share each problems and discovers, and discuss them.

It is a great pleasure for me to discuss with you.
For not disturbing this thread, I'll update my design in #579.

[fizzisist]

- completing MSP 430 integration (fizzist claims that we might run out of IO in case we use debugging leds. I personally would vote for the 5529 MSP as it provides maximum performance without high price raises) 

Regarding IOs, I didnt mean that we are close to running out, only trying to come up with an idea for how many we need. Actually, all the 55xx series have more than 30, I believe.

I agree that it makes sense to just go with the beefiest MCU for now, at least. It will give us the most flexibility for the prototype. If we see that it's extreme overkill, we can downgrade for the first production model. I'll find some time later to switch to the 5529 in the schematic.

[O_Shovah]

@ ngzhang:

No personal offence, but your global setup highly differs from the one we decided on(Power supply, USB, Bus, routing). Turning to this one now would throw us back to stoneage.

There must be some misunderstanding.
Our purpose is slightly different here. The works of yours is facing directly to a simplest platform for FPGAmining, easily to development. But I am a Ph.D candidate, majored in high performance computing, so in my concept, go a little bit further  for universal use, and cost reduction.
But there are still most of the important parts keep alike. So I think we could push each project forward in the meantime, share each problems and discovers, and discuss them.

It is a great pleasure for me to discuss with you.
For not disturbing this thread, I'll update my design in #579.

Yes it seems i misunderstood you. I just thought you proposed to change our current setup according to your schematic.

But running our two developments in paralel shurly will bring benefits for both of us.I appreciate that.  

[fizzisist]

Updated the MCU schematic in my folder on Dropbox. Changes:

1) Added part for MSP430F552x to project.lbr (copied from official TI library).

2) Replaced MSP430F5507 with 'F552x in the PN80 (aka QFP80) package. This was a big job, so I still need to go through and double check everything for any mistakes.

[O_Shovah]

Updated the MCU schematic in my folder on Dropbox. Changes:

1) Added part for MSP430F552x to project.lbr (copied from official TI library).

2) Replaced MSP430F5507 with 'F552x in the PN80 (aka QFP80) package. This was a big job, so I still need to go through and double check everything for any mistakes.

Tanks fizzist

I ll also check on the new schematic just to be shure.

I will try to add the debugging leds later two. Should be as easy as to asign one output top led and ground or am is mistaken ?


Also maybe for the rest of our fellow developers li_gangyi and OlafMandel.
Are you currently busy with other things to do, or do i just imagine haven't seen you around for some time now ?
Please just give a small sign of live.         

[fizzisist]

Hi O_Shovah,

You're right, it's especially quiet around here lately...

I already added a few LEDs to the schematic. There is one on VUSB to indicate that USB power is present, and 4 on general purpose pins that can be used to indicate anything we want from the MCU firmware. We can certainly add more (as many as we have free pins). Another thing I wanted to add was some temperature monitoring. I think some thermistors placed on the board near the the FPGAs would work. Does anyone else have other ideas for this?

[rph]

Awesome thread and project.

How are you guys planning to assemble it when it's done?
Here in the US, low-vol BGA assembly is very expensive,
unless you can DIY. And that takes some guts w/ 2 $160 FPGAs.

-rph

[ngzhang]

Awesome thread and project.

How are you guys planning to assemble it when it's done?
Here in the US, low-vol BGA assembly is very expensive,
unless you can DIY. And that takes some guts w/ 2 $160 FPGAs.

-rph

yep, assembling BGAs is a problem @ earth. In China, it's about 0.5CNY/pin。a 484 BGA cost a soldering fee for  242CNY(38$).
Also, DIY soldering a 484 BGA is an impossible mission.

[film2240]

This is hte 1st time I've ever contributed to FPGA miner dev posts so I don't know much about FPGAS/DEV stuff.I have many suggestions.

If we are to follow the conventional way of doing things (each board has it's own processsor,I/O,power lead et.c),I think it's essential to reduce the no of needed parts to reduce costs.
1.Instead of the need to add additional power connectors,why not just make each board a PCI-E x4 or PCI-E x8 depending on the power needed rather than simply adding several power connectors.This way we can eliminate the need for a mains lead/PSU (could save $20 off build) as well as the space dedicated to having several possible PSU connectors.This is ok for FPGA boards that are meant to be added into a PC and then programmed easily with a GUI Utility (which relates to my preference for easy flashing/running of binaries of need be).
2.To tackle the issue of I/O,I think to save costs again we simply remove the need for storage connector and replace with a suitable sized flash chip that is soldered straight onto the board,that way we can save PCB space (from where the storage connector used to be) and possible save costs/make it more convenient for us users.
3.Again I do think that all RAM should simply be soldered directly to the board,eliminating the SDIMM sockets thus saving space on the PCB and reducing bill of materials a little.

I think this will work for people like me who wish to have a premade/pre configured board with an internet connection (preferably sharing my PC's ethernet without requiring a seprate port for each FPGA board.

I do wish to see the ultimate board of a FPGA,where it's like several FPGS chips sharing 1 set of I/O,solely powered by PCI-E x8 slots (as the extra lanes provide up to 75Ws on PCI-E v1 spec mobos.),have a 2 position switch,1 stock and 1 max performance like on the Radeon HD6990.

I mainly have ideas for boards that run from the PC and that are not independant.

[fizzisist]

film2240, I think you missed some of the details of this design. I know that there's a lot of information in 30 pages of this thread and it can be hard to digest it all quickly. O_Shovah's first post does a good job to summarize the current state, but I'll reiterate some of it here and add to it a little bit.

We are designing a two FPGA board. The interface between the FPGAs and the outside world will be through a small microcontroller. The MCU will connect directly through a USB port on the board, or down to a bus on a "motherboard" through the DIMM connector. The idea is that these FPGA boards can be operated as a small USB device or as a module riding in a larger array of these boards, a la this. The motherboard in this case is yet to be designed (it isn't a standard PC motherboard), but it is planned to have power distribution, bus connections, and a USB port in the first iteration. Later, we have discussed adding an ethernet interface that will connect directly to the internet, without a host computer.

There are three ways to get power to the board:

• through a 4 pin Molex connector from a standard computer PSU
• through a barrel connector from something like a laptop AC adapter
• through the DIMM connector coming from the motherboard

The key here is modularity and flexibility. This means you can buy as many or as few boards as you like, and set them up in whatever way you like. It also means that the basic design can be modified but different designs can coexist in the same system, as long as the all follow the same communication protocols. Furthermore, the boards can be reprogrammed, so can easily adapt to future design changes.

[sirky]

Looks like li_gangyi has been working on this other board: https://bitcointalk.org/index.php?topic=37904.msg465501#msg465501

Which it's actually pretty cool something is out there.

[fizzisist]

Looks like li_gangyi has been working on this other board: https://bitcointalk.org/index.php?topic=37904.msg465501#msg465501

Which it's actually pretty cool something is out there.

Ah, interesting that someone took newMeat1 up on his offer. That explains why li_gangyi's been so quiet lately! It is very cool that something is out there and mining, though.

I hope this doesn't mean that we lose li_gangyi's help with design and soldering work!

Also, there is clearly a lot of room for improvement on their design. Their cost is $440 + the cost of the Xilinx platform cable ($200 new, but eBay has some for $50... are they knock offs?). We're looking at about $600 for our prototype which does twice the hash rate. Future designs and bulk orders can bring our price down to probably close to $440.

[li_gangyi]

Hi,
Yes I've been working on a separate project, and yes it does work (100Mhash/s at this moment). I can share what I've learnt so far though.
I didn't post for awhile because I've nothing constructive to add (being unfamiliar with the MSP430).

Decoupling: whatever the Xilinx datasheet has spec'd is adequate, I've not had noise issues with the minimum number, never tried going below though.

Power Supply: At 100Mhash the entire board was consuming 6+W of power, real life measurements indicate at least 80% of this is through VccInt. A 5A supply for VccIO and VccInt is more then enough. I stuck with the resistor values previously calculated, and came up with 1.18V on VccINT (measured) and 2.48V on VccIO/AUX (measured).

Heatsinking: In an effort to improve decoupling/capacitor performance, I placed alot of the capacitors on the topside right up to the edge of the FPGAs, the smaller 4.7uF capacitors are fine, they are lower then the chip itself, however the larger 100uF caps can be situated away or on an edge where the heatsink will not 'cross'. In the end with my layout I was forced to use a small heatsink (2.5x2.5cm foot print) with a small fan, couldn't find a heatsink that was super tall to get away with a passive design. The heat output itself at 100Mhash/s is significant, you can actually measure an increase in current consumption as the part heats up (possibly due to leakage increasing), and drop when u cool it.

FPGA itself: I've trouble sourcing for the LX150-N3 part in the Fgg484 package in single quantities, forcing me to use a -3 part and thus raising costs. I'm not sure if it's the same situation where you guys are though.

My help and whatnot: I'm still willing to solder the first few prototypes for free. However right now I don't see a very clear direction and idea of what's to be done next (still no MSP430 guys in the fray), especially with the USB to FPGA interface hardware+software side of things, so I can't say for sure if I'll fund the initial boards.

[O_Shovah]

Thanks a lot for your feedback li-gangyi. I also gave you word at your thread.

Im currently learning to programm the msp 430. Anybody interested is invited to join me as this is the biggest obstacle for our project to continue.

As li_gangyi's attention is reasonably now bound elsewere and we are missing other colleagues like Olaf.Mandel and Bahnfire seemto be not present currently,
this will slow down the development considerably but i aim to complete it anyway.

So maybe just to have a quick roundup of our active personell:
Who is still here to contribute development ?

 

[sirky]

While I have done a lot of development in the past, it was 100% software development. I have done absolutely nothing hardware related in my life, and wouldn't know where to start. I hope this project continues though!

[O_Shovah]

While I have done a lot of development in the past, it was 100% software development. I have done absolutely nothing hardware related in my life, and wouldn't know where to start. I hope this project continues though!

It certainly will contiune, its just a question of speed.

Such people as you are ones we are currently lacking.

We desperatly need a firmware and software part for the MSP 430 or any other simmilar chip providing most of its funcitons so it you be great if you could focus on the software part.

[sirky]

While I have done a lot of development in the past, it was 100% software development. I have done absolutely nothing hardware related in my life, and wouldn't know where to start. I hope this project continues though!

It certainly will contiune, its just a question of speed.

Such people as you are ones we are currently lacking.

We desperatly need a firmware and software part for the MSP 430 or any other simmilar chip providing most of its funcitons so it you be great if you could focus on the software part.

Perhaps you should break down exactly what this software would be doing. I am envisioning some sort of C or assembly giving direct instructions to various hardwares, which seems sort of scary for someone who has mainly done warm and fuzzy C# and some web stuff (CSS and HTML) in the last five years

My problem is that I am so unfamiliar with this sort of project I wouldn't know where to begin.

[fpgaminer]

Im currently learning to programm the msp 430. Anybody interested is invited to join me as this is the biggest obstacle for our project to continue.

I'd be happy to write firmware for the MSP430, and supporting PC software. Heck, I even have three MSP430 kits  I couldn't resist their $4+free shipping price.

Correct me if I am mistaken: The MSP430 you have selected will be present on each DIMM board. It will be an MSP430 with built-in USB support. The MSP430 will be connected by SPI to pins on both of the FPGAs.

If that is correct I can:

1) Write the SPI module and test it on my Spartan-6 dev kit.
2) Use my MSP430 dev kit to write the MSP430 code and talk to my S6 dev kit.
3) Write a Python interface, console/UI, for the PC software to talk to the MSP430.

Caveat: My MSP430 dev kit has the lowest form of life MSP430 on it. USB is supported through extra chips, so I don't have an actual MSP430 with built-in USB support. So I, or someone else, will have to eventually get one of those chips and tweak up the code to adjust for any differences.

Please correct my understanding of the current design I quoted above, and if the steps I listed seem correct. If all is well, I'll go off and get your firmware and software all ready 

[fizzisist]

fpgaminer, your understanding of the design is correct. The plan is to use the MCU in a configuration similar to that described on page 38 of ug380.pdf to both program the FPGAs and for communication. I'm guessing you have an MSP430 Launchpad? I think that'd be a great test platform. Yeah, we'll have to tweak the code later to work with the 'F55xx series chips, but your work will be a huge step forward. If you'd like to see the current design in detail, PM me and I'll invite you to the shared Dropbox folder. Also note the currently empty "Software" folder in there

The actual functionality of the MCU can grow and change as time goes on. The plan for the hardware was to design it in a flexible way, so that it can be easily incorporated into bigger, more complicated systems (i.e., through the DIMM connector to the motherboard). For the bare minimum, it will need to be able to load bitstreams and handle the data transfer to the FPGAs. Incorporating this with the python interface won't be an easy task, I'm sure, but it seems like you are the right person to do this.

We will be lucky to have your help here (I doubt anyone but you could get it done right, based on your work so far)!

[fpgaminer]

The plan is to use the MCU in a configuration similar to that described on page 38 of ug380.pdf to both program the FPGAs and for communication.

To clarify, I think you mean page 26, where they show an MCU hooked up to the FPGA for Slave Serial Mode Configuration. Correct?

And then the MCU will also be hooked up to a SPI bus going to both FPGAs (on general IO pins), with the MCU as master. Correct?

What I can do is hook up my Launchpad to 4 GPIO pins on the Spartan-6 LX150T dev kit, and develop all of the SPI communication portion of the code. However, I don't believe I have access to the pins on this board necessary for Slave Serial Mode Configuration. So, I wouldn't be able to code the MSP430 to load the bitstream into the FPGA.

Besides that, I will probably experiment with having the MSP430 talk to the JTAG port of my devkit. That isn't directly applicable to your design but it will provide the ground work for getting the bitstream over USB to the MCU, and the general flow of loading it into an FPGA.

Unless someone else has some way of testing Slave Serial Mode Configuration, you may have to actually build the board before anyone can write and test that side of the code.

Just throwing this out there, but if I can get my MSP430 to load a bitstream into the FPGA through JTAG, would it perhaps make sense to switch the programming interface on your design to that?