Modular FPGA Miner Hardware Design Development

[fizzisist]

Thanks for the input, old_engineer.

@Olaf
Thanks for putting together these totals. It's good to see how the price is shaping up.

I have a question about the PSU. What is the load on the 2.5 V rail? Aren't we going a little overkill with the 10A supply? Maybe we could save a little money and space by switching to the LMZ12002 or 12003 (2A and 3A, respectively). The package is slightly smaller, too.

[Olaf.Mandel]

[...]
I have a question about the PSU. What is the load on the 2.5 V rail? Aren't we going a little overkill with the 10A supply? Maybe we could save a little money and space by switching to the LMZ12002 or 12003 (2A and 3A, respectively). The package is slightly smaller, too.

Good question, the exact current can probably be given by TheSeven, as he compiled (some version of) the HDL code that runs on the FPGA and the tools can give a power estimate. He insists these numbers are highly unreliable, but they are still the best we have at the moment. That said, we can also look at the datasheet: the maximum current for VCCAUX should be 600mA per FPGA, but there is no power consumption specified for VCCIO. As we don't connect many io pins, my guess is that this is low (100mA, maybe). So a 2A switcher may be sufficient. But a guess is all it is! We have to build the prototype and measure the currents in each path.

[fizzisist]

I ran the xilinx power estimator tool using a utilisation of 105% for all parts avaidable (Logic,DRAM,DSP,DCM,....).
It seems 5.5 W is the maximum the FPGA allows before getting cooked.
This splits to:

- 1.2 V :  4,2 A
- 2,5 V : 0,3A

This numbers heavily scale with temperature. So cooling is a major influence factor as power consumption ranged from 7.7W (no cooling, no heatsink, -T_case 125 °C)
to  4.5 W (500 LFM,huge heatsink,T_case 40°C) using the same setup.

This estmination should cover for the absolut possible maximum, but as i am totaly new to this tool i would like somebody more used to the FPGA design to confirm this results.

Here's O_Shovah's estimate from a few weeks ago, to remind us. 300 mA x 2 + 100 mA for the MCU gives 700 mA total. I think the LMZ12002 should be sufficient then, and it's half the price of the LMZ12010. Does anyone object to switching to this for the 2.5V switcher?

By the way, it would probably be a good idea to include mounting holes for a heatsink around each FPGA. Also nice would be some headers for fans. These could be directly connected to the 12V unregulated rail. Maybe something like this fansink?. It's a little bit large, but I think would do a great job at cooling these FPGAs. It's probably not required that we use a heatsink like this, but it would be nice to include the holes for it so that we have the option. A more compact option would be something like this, but it's too large for our FPGAs. And finally, a passive heatsink that fits the FG(G)484 package perfectly: HS2135DB.

Does anyone else know of other good options? This is the result of 5 minutes searching on my part, I'm sure there are thousands of other heatsinks out there.

[TheSeven]

I ran the xilinx power estimator tool using a utilisation of 105% for all parts avaidable (Logic,DRAM,DSP,DCM,....).
It seems 5.5 W is the maximum the FPGA allows before getting cooked.
This splits to:

- 1.2 V :  4,2 A
- 2,5 V : 0,3A

This numbers heavily scale with temperature. So cooling is a major influence factor as power consumption ranged from 7.7W (no cooling, no heatsink, -T_case 125 °C)
to  4.5 W (500 LFM,huge heatsink,T_case 40°C) using the same setup.

This estmination should cover for the absolut possible maximum, but as i am totaly new to this tool i would like somebody more used to the FPGA design to confirm this results.

Here's O_Shovah's estimate from a few weeks ago, to remind us. 300 mA x 2 + 100 mA for the MCU gives 700 mA total. I think the LMZ12002 should be sufficient then, and it's half the price of the LMZ12010. Does anyone object to switching to this for the 2.5V switcher?

By the way, it would probably be a good idea to include mounting holes for a heatsink around each FPGA. Also nice would be some headers for fans. These could be directly connected to the 12V unregulated rail. Maybe something like this fansink?. It's a little bit large, but I think would do a great job at cooling these FPGAs. It's probably not required that we use a heatsink like this, but it would be nice to include the holes for it so that we have the option. A more compact option would be something like this, but it's too large for our FPGAs. And finally, a passive heatsink that fits the FG(G)484 package perfectly: HS2135DB.

Does anyone else know of other good options? This is the result of 5 minutes searching on my part, I'm sure there are thousands of other heatsinks out there.

Use the maximum possible values from the datasheets and add some safety margin when designing the PSU and cooling. If the chips stay cool, they consume less power and yield higher hashrates. I'd calculate 8-10W per FPGA. 45°C ambient temperature, and I'd like the FPGA to stay below 65-70°C if possible. So if you don't want to use a fan, you would need at least something like this.
If no other components are in the way, I'd consider using one huge passive heatsink that covers both FPGAs. This would also be favorable for a backplane setup in a server case, where there room between the DIMMs is almost filled by the heatsinks, so you can just add some big fans that push some air through the whole thing from one end of the DIMMs to the other one. I'm thinking of something along the lines of this

[O_Shovah]

Hello i'm back

Im happy to see , you made good progress. 

I only have read through the last pages partialy yet so just some thougts on the last points.

- Regarding the parts size: We will need to have a comment from li_gangyi how much hassel he would have soldeirng smaler components but i havent seen him around lately.
Maybe im gonna pm him later.

- Regarding the Power consumption: The Calculations i ran were a total first try for a absolutly theoretical maximum. @ TheSeven maybe you could run the estemination tool again as you are experienced with Xilinx software.
But in the end nothing will replace test cycles with the protoype.

- Regarding the heatsink: the solution http://www.radianheatsinks.com/Products/INM23001-33W/2.6_155.aspx you postet is certainly  good choice for the prototype,
but for series we should consider a solution using mounting holes. I consider the way of snapping it below the BGA a bit risky.
Im on search for a plate fin heatsink wich covers the whole DIMM in order to act as heatsink for both the the FPGA's and the  voltage regulators.But different heiths and themeratures might be an obstacle to that.

[li_gangyi]

Hi, yeah I've been busy lately. I've no problems with 0402s, but it'll probably slow things down abit. Do we really need the 0402s at this stage?

Regarding the heatsinks, I don't see a problem with the clipons as long as it isn't bumped around. Alot of times I see them thermal epoxied or thermal taped to the package. I don't think we'd be able to find a heatsink that'll cover both the FPGA and Vregs, since they have vastly different heights. The 100uF decoupling capacitors around the perimeter of the FPGA will also probably prevent a long heatsink from being used to cover both chips.

[DiamondPlus]

I find it interesting because of the FX2 chip, but apart from that it has too much: extra DDR SDRAM, extra EEPROM, extra CPLD, microSD card slot. The pricetag is only a factor of 2 above what we hope to achieve (note that they are missing the power supply section and need 3 different voltage rails).

I don't know about the others, but I am in it for the fun of developing a board. The board you linked may be a very good start to get FPGA developers to write code, but once our board is go, it won't be cost effective (unless you figure their support and warranty are worth the price). I am not arguing against the board: it is good for early adopters, but eventually we should beat it.

[O_Shovah]

Maybe just to have a little roundup,

wich task are still to be fullfilled for finalising our prototype on the hardware stage ?

- completing MSP 430 integration ?

- I asume the power section is complete so far

- clarify if the 0402's are nessecary ( i personally would use them to stick to the specification as close as possible)

- intergration of debugging devices ( pin headers, leds ,peepers) ?

- completing routing on the DIMM


Please correct me and complete this list.

[fizzisist]

O_Shovah, good idea to summarize the list of tasks remaining. I would add to it:

1) Decide if we should switch from the LMZ12010 to the smaller and cheaper LMZ12002 (or 3) for the 2.5V rail.

2) Did Olaf change the feedback resistors to the suggested ones in the power supply datasheet?

3) Decide on cooling method. These could mean leaving mounting holes (or clearance for the Radian EZ-Clip) and fan headers.

[Olaf.Mandel]

[...] Did Olaf change the feedback resistors to the suggested ones in the power supply datasheet? [...]

Not yet. I also wanted to incorporate the suggestion by TheSeven to increase the 1.2V rail to 1.26V, and I haven't thought about how to find the closest matching resistor values for a given ratio.

[fizzisist]

[...] Did Olaf change the feedback resistors to the suggested ones in the power supply datasheet? [...]

Not yet. I also wanted to incorporate the suggestion by TheSeven to increase the 1.2V rail to 1.26V, and I haven't thought about how to find the closest matching resistor values for a given ratio.

Just looked back a couple of pages, and TheSeven actually suggested 1.25V, because 1.26 is the max allowed. For that, R_FBT = 1.07k and R_FBB=1.87k gives 1.24989V.

Also, the question of resistor tolerance came up a while back. With this combination and 1% resistors, the worst cases give 1.2591 and 1.2409, which I think is sufficient.

I made the change to the schematic in your folder, and updated the BOM database with those resistor values.

[Olaf.Mandel]

[...]
Just looked back a couple of pages, and TheSeven actually suggested 1.25V, because 1.26 is the max allowed. For that, R_FBT = 1.07k and R_FBB=1.87k gives 1.24989V.
[...]
I made the change to the schematic in your folder, and updated the BOM database with those resistor values.

Thanks! I just now finished the same calculation and got the same results (didn't check the forum first)   

[fizzisist]

Thanks! I just now finished the same calculation and got the same results (didn't check the forum first)    

Independent verification is always a good thing! Especially when I did the calculation!

[ngzhang]

Errrrrrrrrrrrrr...

I read through this thread, I apologize but the feeling is not good. This project is lack of professional guys to do it.
In my opinion, the daughter board and mother board could develop independently, the daughter board have one 5V  input jack, and a USB to UART bridge, in order to connect to a PC and run standalone. and the DIMM connector is a good idea, but please, please use DDR2 - 184PIN socket instead of DDR3- 240PIN socket. Because the soldering fees is depend on the pin count. Then, let "some" GPIOs of the fpga to the connector is enough.

Power the fpgas is very important, use a DC-DC switch to provide about 15A　VCCINT for the 2 XC6SLX150 -3N s will be enough.

and that is all, maybe 3~5days work for an Experienced engineer.

OK ,lets talk about the cost. Now in my country, a XC6SLX150 is about 1000CNY. sampling a 4 layrs PCB ( in acceptable quality), is about 1000 CNY/ time. and maybe 10-20CNY each.  The DC-DC switch circuit is about 50CNY of BOM cost. soldering fees is very high in low quantity, about 200CNY/ PCB. and something others.

so if we build 20 of them, we need:

40FPGAs : 40,000 CNY
PCB: 1,000 CNY
soldering: 4,000 CNY
all other things on the PCB: 4,000CNY(MAX)

less than 50,000 CNY, we got 20 of them. 2,500CNY  each.

If the fpga coding team works well, 200Mh/s of each chip, we got 4Gh/s with 50,000 CNY. The power is about 200w.

50,000 CNY = 7,700 US$

For 4Gh/s, we must use 15 HD6870s in STD FREQ, this cost about 30,000 CNY here(Include PSUs, mother boards, etc. ). These GPUs consuming 3KW of power.

I'm very very sorry about my language. If there are any misstanding, please point out and I will explain it on my best effort.

Thank you.

[fizzisist]

Use the maximum possible values from the datasheets and add some safety margin when designing the PSU and cooling. If the chips stay cool, they consume less power and yield higher hashrates. I'd calculate 8-10W per FPGA. 45°C ambient temperature, and I'd like the FPGA to stay below 65-70°C if possible. So if you don't want to use a fan, you would need at least something like this.
If no other components are in the way, I'd consider using one huge passive heatsink that covers both FPGAs. This would also be favorable for a backplane setup in a server case, where there room between the DIMMs is almost filled by the heatsinks, so you can just add some big fans that push some air through the whole thing from one end of the DIMMs to the other one. I'm thinking of something along the lines of this

I like that elliptical fin heatsink. As for the idea of one big heatsink, the Radian HS1594EB might work. It's similar in size to the one you linked to, but has mounting screws and the fins are oriented in the correct direction (parallel to the longer dimension). I drew an outline of it over Olaf's FPGA section, to get a sense of the size:



If we make sure to put those holes in the board and keep any tall components out of that area (greater than 2 mm), we'll have the flexibility to use either of these two options.

[kjj]

Because the soldering fees is depend on the pin count.

Really?  I figured they would just use a wave machine or a reflow oven.

I've designed several PCBs, and sent them out for low volume fabrication before, but I've never had one assembled, or really even looked into it.  I just sorta assumed that it was pay per board, or pay per unit area.

[ngzhang]

Because the soldering fees is depend on the pin count.

Really?  I figured they would just use a wave machine or a reflow oven.

I've designed several PCBs, and sent them out for low volume fabrication before, but I've never had one assembled, or really even looked into it.  I just sorta assumed that it was pay per board, or pay per unit area.

Yes, partly you are right. In large quantity, they use wave-soldering(for through-hole comps.) and reflow-soldering(for SMT comps.).
But in low quantity, they use man hand to soldering through-hole comps and discrete comps, use a BGA rework station to soldering BGAs and QFNs. Because the wave-soldering and reflow-soldering cost a huge prepare time. So they start at a minimum fees(maybe 50,000 CNY of soldering fees here).

So, we may have to soldering with a few boards, the cost is depend on the pin-count on the board. Here they are about 0.05 CNY/ pin. And the BGAs are expensive, plus a 0.5CNY/BGApin. Of course, you can cut down some fees when you are soldering 100 boards on time, it's a mid-quantity for some small factory.

PS:
I strongly recommend use through-hole electrolytic capacitors and headers instead of SMTs . Because man hand soldering CAN NOT soldering large SMT comps well. They may easily to drop off.

EDIT:

ok, now I didn't go to school(I'm now a phd student ) and worked on this project for 1 day.
I think, if we want to push this project a little faster, may be we must confirm some thing. 1st is a fixed interface of the daughter board.  I think, #508 's design maybe wasted too much gold-finger pins. We can made the daughter board to be used more universally, such as a Accelerate Unit in high performance computing, fit with other computing project, but no addition costs. I'm working on it and will release it tomorrow. I hope finish it in 1-2weeks and got a example at the beginning of Sep.


EDIT2:
Design concept. UPDATE 2011/8/16



The DIMM socket must have 5V and 12V supply, but the daughter board could select to use 1 of them or both of them, for the maximum  compatibility. In correct design, Only 512V is used.

EDIT3:
Add: FPGA JTAG chain and config circuit.


NOTE that FPGA1_SCK signal also connected to FPGA2 CCLK pin. FPGA1's DOUT pin connedted to FPGA2 CSI_B pin.

[fizzisist]

I'm thinking about the MCU decision again. I'd like to summarize the discussion on this so far.

Package: li_gangyi would prefer to stay away from QFN. As long as there is a non-QFN choice, and we aren't pressed for space, I don't see any reason to go with QFN.

IOs: The current schematic has 15 digital IOs used. We will also need a few debugging LEDs, say 4. I think we also need a couple of analog lines for temperature monitoring. Let's say 30 is the minimum number of IOs.

Flash memory: We still have no good estimate for this. I would guess 32 kB is the bare minimum, and the more the merrier.

Based on those points, we are limited to the following parts (notes are flash+SRAM, price):

MSP430F5510 (32+6kB, Non-stock at Digikey)
MSP430F5521 (32+8kB, $7.30)
MSP430F5525 (64+6kB, $8.06)
MSP430F5527 (96+8kB, $8.52)
MSP430F5529 (128+10kB, $8.73)

Based on that, we only really need to decide on the amount of program space needed. Does anyone have a guess?

[ngzhang]

every xc6slx150 need about 33.9Mbit for config. 2 if them could use 1 flash to store 2 config bitsteam in it. about 68Mbit. use a m25p128 to do it. about 20CNY (2€)1 pcs here.



EDIT


EDIT2？

Why no QFNs?

Hand soldering this board is IMPOSSIBLE. Because soldering the 484pin BGAs need BGA rework station or reflow-soldering system. They also can soldering QFNs very easily.

[O_Shovah]

Ok so to improve the todo list:(EDIT)

- completing MSP 430 integration (fizzist claims that we might run out of IO in case we use debugging leds. I personally would vote for the 5529 MSP as it provides maximum performance without high price raises)  

- clarify if the 0402's are nessecary ( i personally would use them to stick to the specification as close as possible)

- intergration of debugging devices ( pin headers, leds ,peepers) ?

- completing routing on the DIMM

-Decide if we should switch from the LMZ12010 to the smaller and cheaper LMZ12002 (or 3) for the 2.5V rail. (i personally would leave the current setup widely untouched.
 We may meassure the real current while testing and may choose a DC DC converter acording to that for the series)

- resistors in the power supply ?

- Decide on cooling method. These could mean leaving mounting holes (or clearance for the Radian EZ-Clip) and fan headers.


So who is going to aim for wich task please ?


@ ngzhang:

No personal offence, but your global setup highly differs from the one we decided on(Power supply, USB, Bus, routing). Turning to this one now would throw us back to stoneage.