Modular FPGA Miner Hardware Design Development

[Olaf.Mandel]

I thought about the FPGA signals some more and edited my previous table: PROGRAM_B should be dedicated (like SSEL) to be able to reboot each FPGA individually.

[1713292672]

1713292672

[1713292672]

1713292672

[1713292672]

1713292672

[O_Shovah]

There you have 7.7W, which is ~6.5A, which already exceeds the 12A max for two FPGAs. Also, please don't trust those tools. They're estimating something like 3 watts for my virtex 5 design, and judging from the amount of heat generated the real power dissipation must be way higher. You're only really safe if you assume 100% of the gates toggling on each clock cycle. SHA256 is surprisingly close to that...

Well just to explain this again maybe. The consuption of 7.7 W is calculated for a temperature of 125 °C wich would absolutly certain result in a total destruction of the IC.
The last value that would be usable would be at 85 °C an there i got those 5,5 W.

This regulator here, same series as you posted, but would offer up to 15 A at 1.2V wich should be enough for two fpga's in our current setup. http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=LTM4627EV%23PBF-ND
This would give us ~7 A per FPGA minimum resulting in a 65% overhead over my calculation. And for the first DIMM's we will use only one FPGA anyway so there should be plenty of security and the chance to measure the real values.


@TheSeven :  Maybe you also run some esteminations regarding power consumption as you are more experienced with FPGA design than me.

[li_gangyi]

Well just to explain this again maybe. The consuption of 7.7 W is calculated for a temperature of 125 °C wich would absolutly certain result in a total destruction of the IC.
The last value that would be usable would be at 85 °C an there i got those 5,5 W.

This regulator here, same series as you posted, but would offer up to 15 A at 1.2V wich should be enough for two fpga's in our current setup. http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=LTM4627EV%23PBF-ND
This would give us ~7 A per FPGA minimum resulting in a 65% overhead over my calculation. And for the first DIMM's we will use only one FPGA anyway so there should be plenty of security and the chance to measure the real values.

Maybe we can simplify even further for prototyping by using National's simple switchers, eg the LMZ22010

http://www.national.com/pf/LM/LMZ22010.html#Overview

This reg uses very few external parts, houses an internal shielded inductor, each puts out 10A and we can place each part near the FPGA for easier routing/decoupling. The only con I guess is the slightly higher cost per part.

*edit I've looked at digikey pricing, and it doesn't seem all that far apart if we use 2x 8A parts (each feeding the FPGA), I think we need to seriously come up with some reliable power consumption figures, anyone with a demo board/dev kit want to confirm the figures for us? That'd be a great help.

[pdki]

A bit off topic, but has anybody though about simply using Arduino http://arduino.cc/en/ together with an Ethernet shield http://de.rs-online.com/web/cpd/6961661/ and then just making one new shied for every FPGA that shall be plugged on top?

This could be the 100€ "backplane" with Ethernet and each FPGA board would need nothing than the FPGA and power. This could also keep the cost of the daughterboard low.

[fizzisist]

One question, why are we planning for 12V input when we want 2.5V maximum? Wouldn't it be much more efficient to be regulating the 5V rail off of an ATX power supply? For reference, this power supply has a 34A limit on the 5V rail (170W) and is only $16. Actually, it also has 28A on 3.3V (92.4W), which might be useful too. Trying to step down 12V to 2.5V is going to be very wasteful.

It seems like as PSUs get more expensive, they are only increasing the 12V rails, so we could save a lot of money by using these cheap PSUs with low 12V rails.

[li_gangyi]

One question, why are we planning for 12V input when we want 2.5V maximum? Wouldn't it be much more efficient to be regulating the 5V rail off of an ATX power supply? For reference, this power supply has a 34A limit on the 5V rail (170W) and is only $16. Actually, it also has 28A on 3.3V (92.4W), which might be useful too. Trying to step down 12V to 2.5V is going to be very wasteful.

It seems like as PSUs get more expensive, they are only increasing the 12V rails, so we could save a lot of money by using these cheap PSUs with low 12V rails.

The community has already decided to include the option of using laptop power supplies (that run anywhere from 12 to 20V) to power the design, electrically it might not seem like the best of all ideas, the higher input voltages usually mean a lower efficiency. With the reg I have in mind, even at 16V in we'd be getting around 70ish % efficiency, not too shabby. Transient and ripple also look good (at least in the datasheet, layout and parts will have an impact).

It's not a bad thing to have, a person aiming for max efficiency has the option to go out and get a suitable adapter that'll get you better efficiency figures.

[fizzisist]

One question, why are we planning for 12V input when we want 2.5V maximum? Wouldn't it be much more efficient to be regulating the 5V rail off of an ATX power supply? For reference, this power supply has a 34A limit on the 5V rail (170W) and is only $16. Actually, it also has 28A on 3.3V (92.4W), which might be useful too. Trying to step down 12V to 2.5V is going to be very wasteful.

It seems like as PSUs get more expensive, they are only increasing the 12V rails, so we could save a lot of money by using these cheap PSUs with low 12V rails.

The community has already decided to include the option of using laptop power supplies (that run anywhere from 12 to 20V) to power the design, electrically it might not seem like the best of all ideas, the higher input voltages usually mean a lower efficiency. With the reg I have in mind, even at 16V in we'd be getting around 70ish % efficiency, not too shabby. Transient and ripple also look good (at least in the datasheet, layout and parts will have an impact).

It's not a bad thing to have, a person aiming for max efficiency has the option to go out and get a suitable adapter that'll get you better efficiency figures.

My concern is not just with efficiency, but with the high price of those switching regulators. $20 per FPGA is really a significant cost, even higher than the PSU I linked which could supply 10 FPGAs! If we limit the design to not allow laptop adapters, we could potentially reduce the cost.

I haven't read through datasheet completely yet, but the TPS40041 is only $3 and has 90% efficiency. Short summary: 2.25V to 5.5V input, 15A output.

[TheSeven]

One question, why are we planning for 12V input when we want 2.5V maximum? Wouldn't it be much more efficient to be regulating the 5V rail off of an ATX power supply? For reference, this power supply has a 34A limit on the 5V rail (170W) and is only $16. Actually, it also has 28A on 3.3V (92.4W), which might be useful too. Trying to step down 12V to 2.5V is going to be very wasteful.

It seems like as PSUs get more expensive, they are only increasing the 12V rails, so we could save a lot of money by using these cheap PSUs with low 12V rails.

The PSU you quoted provides more power on the 12V than on the 5V rail. The 5V and 3.3V usually have a total power limit for both of them combined, which is only slightly above the 5V limit. Also, when paying thousands of dollars for FPGAs, you won't really mind paying $20 more for a better, more efficient PSU. These cheap PSUs are usually crap efficiency-wise. However, the one you quoted can easily run more than 10 FPGAs on its 12V rail.

When talking about regulator efficiency, you aren't considering cable and connector resistance/voltage drops and current limits. I'm fairly sure that a 12V PSU design will run more efficient than a 5V design. If the regulator we choose can accept lower voltages as well, that would of course be even better. But I'm not sure if there are cheap and efficient 1.2V regulators with a very wide 3.3-20V input voltage range.

[TheSeven]

My concern is not just with efficiency, but with the high price of those switching regulators. $20 per FPGA is really a significant cost, even higher than the PSU I linked which could supply 10 FPGAs! If we limit the design to not allow laptop adapters, we could potentially reduce the cost.

I haven't read through datasheet completely yet, but the TPS40041 is only $3 and has 90% efficiency. Short summary: 2.25V to 5.5V input, 15A output.

This is just a switching regulator controller, which needs several external components, which will definitely cost more than this chip itself. This will also probably need more board space, and you can do much more wrong during the PCB design. The efficiency numbers are probably calculated for a very expensive choice of external components, so this doesn't really sound good to me.

[O_Shovah]

Jay we are back to the power supply disscussion.

I think everything has been already said why we have choosen a 12 V - 20 V input net  ( less current, wide range input for ATX PSU und Laptop supply......)

TheSeven,serveral other people and also i, have posted a variety of voltage regulators fit for our application (11-20V input 1,2 V and 2.5 V output) so i consider this problem solved.


Therefore we can go on with the BUS design and FPGA routing.

I started reading through the documentation of the Spartan 6 and the MSP430f55xx series for my routing.

Has anybody got a more or less complete overview over the pins we need to use on the FPGA and what is to be done with I/o pins not needed ?

[fizzisist]

Fine, I will shut up about the power supply. I understand the importance of keeping the discussion moving, even if I don't agree with every decision.

[O_Shovah]

Fine, I will shut up about the power supply. I understand the importance of keeping the discussion moving, even if I don't agree with every decision.

Sorry, i didn't mean any personal offence.
But experience tought me that development forums looking for the "best" solution end up dead or running in circels, so i prefere not redisscussing points that have been decided for now.


As far as i understand the Xilinx documentation, we are free to connect all I/O pins not needed (almost all in our case exept for JTAG,I2C and Spi) either to Ground or VCC (2.5V).
Leaving them unconnected would increase the chance of noise.

Can anybody please verify the facts about the FPGA pinout. (i am still not so confident i undestood everything correctly)

[makomk]

As far as i understand the Xilinx documentation, we are free to connect all I/O pins not needed (almost all in our case exept for JTAG,I2C and Spi) either to Ground or VCC (2.5V).
Leaving them unconnected would increase the chance of noise.

I think Xilinx's tools default to creating bitstreams that pull down unconnected pins to ground. Not sure what happens if you try and connect them to a different voltage externally, but finding out probably wouldn't be the best idea.

[O_Shovah]

I just created a dropbox folder for sharing and collecting als data needed for this development from now on.

Please give me a pm with your email adresse in case you want to upload something and have a dropbox account so i may add you to the group. https://www.dropbox.com


@mamok:   Thanks,  so it seems its best to just leave them alone/unconnected 

[Olaf.Mandel]

I did a demo of two FPGAs connected by JTAG and SPI. The SPI is connected to the pins used during configuration, so the chips can also be booted from there. The board is two-layer only and uses rather normal specifications, so it should be to make. The problem with that is: the board becomes large because of the high-current busses (see top of board).



Full files at: the Github repository, in the slx150x2 / hw directory.

Current todo:

• There is a choke point in the bottom layer, in the middle of each FPGA. The polygons are too small for 5A+ of current. This can be fixed by routing VCCaux across some unused pins. Will do that tomorrow.
• Check the design! I may be completely out to lunch.
• My understanding was, that the default behaviour of ISE is to pull unused pins up to VCC_O . That is how I connected most pins. If it is pulldown to GND, the design needs to change quite a bit...

As for manufacturing cost: PCB-Pool (a German PCB maker, they populate boards too) shows these prices (they are not really cheap, but it may be Ok for prototyping):

Edit 2:
For 1 board with solder-stop, silkscreening, E-test, 120x65 mm²: 81EUR
For populating that board with 63 SMD parts: 188EUR
Both prices are without tax.

Edit:
If the board had components on the backside, also, the price for populating the board rises to 263EUR.

Edit 3:
Just to put this into perspective: if you order 50, not 1 PCBs, then the per-PCB cost is 10.16EUR + 12.72EUR, so only 10% of the FPGA costs.

[li_gangyi]

The prices for populating are for 1 board? That price is inclusive or exclusive of the parts cost? It may be necessary to goto a 4 layer board, and then put the regs and decoupling caps on the backside. We're probably gonna have problems routing the LTM4627 output traces for 10+A without multiple vias that conduct to a power + ground plane.

[Olaf.Mandel]

The prices for populating are for 1 board? That price is inclusive or exclusive of the parts cost?

I stated two prices: for one board it is 188EUR, for 50 boards it is 12.72EUR per board. Those are, of course, without the cost for the parts: one FPGA costs 112EUR, so getting two plus the word for even 188EUR would be quite a bargain, lt alone the 12.72EUR price tag.   

It may be necessary to goto a 4 layer board, and then put the regs and decoupling caps on the backside. We're probably gonna have problems routing the LTM4627 output traces for 10+A without multiple vias that conduct to a power + ground plane.

2-layer to 4 layer is a question of price and size: the 2-layer board is cheaper per area, but it is also larger. And the board may grow so large in the 2-layer design that it is too high or too wide for our preferences. Comments?

About the vias: probably. I used multiple vias, where possible, also. I may even add a few more below the centre of the FPGA, to not let the traces get so hot.

[O_Shovah]

I had a qoute from pcbcart for a 4 layer board of our size with gold finish and 240 connection pinns for the DIMM ind 1.2 mm thikness resulting in 18 euro per board  for a quatity of 10 boards.

I would like to avoid manufayturing the boards in germany. The liddle to no increase in quality often doesn't justify the price over the asian manufacturers in my experience.

@Olaf.Mandel:

Wich size of the board did you set for your design and your quote ?

I tought of increasing the heith  of the daughterboard above standart DIMM size to 50 maybe 60mm to get more space.

I also think we will have to populate the backside with capacitors anyway.The Xilinx docu tells to put the smallest capacitors right behind the Vint and Vcc vias.
So having to additional layers for ground and signal would be a  plus.

Edit: just to have a frame of capabilitys we are talking about here. The restictions published by pcbcart:

Layers                         1-10 layers
Material                         FR4
Copper Thickness         1/2 to 2oz  (18um-70um)
Board Thickness         .016-.126"  (0.4mm-3.2mm)
Surface finish                 HASL,Ni/Au,OSP
Soldermask                 LPI, different colors
Board Dimension         600x700mm
Min Hole Diameter         8mil (0.2mm)
Min line width                 8mil (0.2mm)
Min line spacing          8mil (0.2mm)
Min SMT pitch             16mil (0.4mm)
Min. Annular Ring         .008" (0.2mm)
Aspect Ratio                 5:1
Surface/hole plating         ave. 25um min. 20um
Tolerance:    
Hole Tolerance (PTH)      .002" (0.05mm)
Hole Tolerance (NPTH)    .003" (0.075mm)
SM Tolerance (LPI)        .003" (0.075mm)
Dimension                .004" (0.1mm)
Electrically test                10V-250V, flying probe or testing fixture
   
   
   
 

[Olaf.Mandel]

I had a qoute from pcbcart for a 4 layer board of our size with gold finish and 240 connection pinns for the DIMM ind 1.2 mm thikness resulting in 18 euro per board  for a quatity of 10 boards.

The thickness of 1.2mm is correct: the supplier I mentioned would have created a board with 1.6mm thickness which is too thick or with 1mm thickness, which is too narrow. The price of your manufacturer sounds very good: do you know someone who can do the soldering?

I would like to avoid manufayturing the boards in germany. The liddle to no increase in quality often doesn't justify the price over the asian manufacturers in my experience.

I just needed someone who gives me an online-quote. I was not looking at a specific country at all.

@Olaf.Mandel:

Wich size of the board did you set for your design and your quote ?

I used 120x65 mm². That is just the size of the demo, unrelated to an actual DIMM.

[...]
I also think we will have to populate the backside with capacitors anyway.The Xilinx docu tells to put the smallest capacitors right behind the Vint and Vcc vias.
[...]

When I wrote that placing them on the front or back makes not much difference according to UG380, I was thinking of a 1.6mm board. I forgot the DIMM-connector, which needs a 1.27mm board. In that case, caps on the backside do seem to make more sense.

[Olaf.Mandel]

[...]
The restictions published by pcbcart:

[...]
Copper Thickness         1/2 to 2oz  (18um-70um)
Board Thickness         .016-.126"  (0.4mm-3.2mm)
[...]
Min Hole Diameter         8mil (0.2mm)
Min line width                 8mil (0.2mm)
Min line spacing          8mil (0.2mm)
Min SMT pitch             16mil (0.4mm)
Min. Annular Ring         .008" (0.2mm)
[...]

The copper thickness can be selected: very good, we may need more than the 35µm default. The 8mil minimal via diameter is even better (though you get less current through a smaller via)! The 8mil line width and line spacing are a problem, though: I cannot get an 8mil wire between two pads of the FPGAs: the clearance is too small. This problem can be solved by using more vias and routing everything on the other layers, but is is less than ideal.

There are "special PCBs" on the pcbcart webpage, though: 4mil for both specs. And they offer much more copper! I wonder how much that costs?

The point I made about the clearance depends on which pad diameter you use. My library used 0.5mm pads, but I read somewhere that 0.4mm may be sufficient. Any insight?