Modular FPGA Miner Hardware Design Development

[TheSeven]

I wanted to get an idea what the space would look like on this board. This shows two LX150s in the FG484 package and the MSP430 in the QFN64 package. This FPGA is available in a smaller package, CSG484, but I think this one fits nicely, and a larger pitch means easier job routing the signals to the pins. For comparison, this package is 23 mm square, the smaller one is 19 mm square.

Maybe just to remember you.

We wanted to use a Molex AND a barrel connector.Of course this will result in some space issues.
The backside will be populated with condensators anyway so we might use it also for some other parts.

I guess we will have to increase heith of the DIMM anyway.

Regarding Voltage supply there were seriously considered (for the FPGA supply):

For 1.2 V http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=LTM4601EV-1%23PBF-ND    5A @ 1.2V

For 2.5V ( multiple of) http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=R-78AA2.5-1.0SMD-ND 1.9A @ 2.5V


Just out of interest. Will one Msp 430 be sufficient for both FPGA's ?  

PS.:
Today i will also start a rough layout in eagle.Maybe we could share eagle files for comparison.

5 Amps on 1.2V? No way. If we want this thing to operate stable and have some headroom for future designs, we should allow more like 10A per FPGA. The regulator you linked can supply 12A, so one of those for each FPGA might be a bit overpowered, but one for both won't work. In that case you'll need to use a bigger one (>18A).
Do we have some data on the actual power consumption on the 2.5V rail?

[Olaf.Mandel]

[...]
5 Amps on 1.2V? No way. If we want this thing to operate stable and have some headroom for future designs, we should allow more like 10A per FPGA. The regulator you linked can supply 12A, so one of those for each FPGA might be a bit overpowered, but one for both won't work. In that case you'll need to use a bigger one (>18A).
Do we have some data on the actual power consumption on the 2.5V rail?

Is there any indication that 10A are needed? My understanding is that the maximum that the chip may use is 5A. Can someone run a power-analysis, assuming a 100% state-change rate for all flip flops? The data-sheet unfortunately does not specify a safe maximum current.

As for 2.5V: we probably consume most for the VCCaux net. We have just a hand full of connections on bank 2. But again: for correct numbers we need to run the actual design through the power-analysis.

[li_gangyi]

Hi guys, I have access to BGA rework/repair equipment, if a prototype batch needs to be made, say about 50 boards max, I can assemble and test them out.

http://fillwithcoolblogname.blogspot.com/

That's my blog. I usually do Xbox 360 reballs. I'm new to the bitcoin scene, but not new to electronics.

At this point I think we just need to iron out the PSU requirements, and then design a board to fit all this in. Once that's done we can start drawing up a schematic and then try routing a board that'll fit (and is not too expensive), whilst the bus decision is being finalised. As for populating the backside, I don't think that's a major problem, if cost is prohibitive I figure we can manually solder them on by hand (if the components are say decoupling capacitors and sockets.). just my 2cents.

[grid]

A side-note, check out this little beast:

Algorithm accelerator:
http://www.dinigroup.com/new/DNBFC_S12_PCIe.php
Featuring 12 LX-150's + LX-150T, on a PCI-E board, independently powered (consumes up to 75W, PCI-E independent power just like a GPU).
https://i.imgur.com/IhcwH.jpg

Pricetag... around 10k USD, so around $833 per FPGA chip.
Power draw - 6.25W per FPGA+DDR3 chip (assuming only the LX-150's are very active).

Posting as food for thought, could be considered as a price and design reference point, esp. given the fact that it uses the same FPGA.

6.25W power draw, at 1.2V = 5.2Amps not counting conversion and ohmic losses.

[fizzisist]

Just out of interest. Will one Msp 430 be sufficient for both FPGA's ? 

I think so, this MCU has 47 GPIO pins, so, I think it will be enough. Would someone be willing to make a table of all the expected signals needed? This could be for a tentative design including the maximum conceivable number of bus interfaces, so that we don't end up short later.

PS.:
Today i will also start a rough layout in eagle.Maybe we could share eagle files for comparison.

Great! Yes, we should share as much as possible. I wish there was a "GitHub" for Eagle files. Would a Dropbox shared folder work well?

Currently, Avnet has 344 of the FGG484 packages in stock and 0 of the CSG484 packages.

I couldn't find any of the XC6SLX150-N3 at AVNET in the FGG484 package, not even listed. Maybe you get different results because you're in Europe?

5 Amps on 1.2V? No way. If we want this thing to operate stable and have some headroom for future designs, we should allow more like 10A per FPGA. The regulator you linked can supply 12A, so one of those for each FPGA might be a bit overpowered, but one for both won't work. In that case you'll need to use a bigger one (>18A).

Once again, the design of this board doesn't have to have any influence on future designs. If a second generation FPGA board is made with higher power requirements, the regulator will increase accordingly. This design needs to serve this FPGA. Period. You'll never get anywhere if you always try to plan for future designs.

Is there any indication that 10A are needed? My understanding is that the maximum that the chip may use is 5A. Can someone run a power-analysis, assuming a 100% state-change rate for all flip flops? The data-sheet unfortunately does not specify a safe maximum current.

As for 2.5V: we probably consume most for the VCCaux net. We have just a hand full of connections on bank 2. But again: for correct numbers we need to run the actual design through the power-analysis.

I agree that this is the most pressing thing to know right now. I'll try to do this, but exactly what design are we using? Can we get a link to the code placed in the first post?

Hi guys, I have access to BGA rework/repair equipment, if a prototype batch needs to be made, say about 50 boards max, I can assemble and test them out.

This would be very helpful! At least for the first prototypes. I looked through your blog and you look very experienced. How would you rate your confidence working with $160 ICs? How much time do you have to work on this? Nothing personal to you, but for a larger order on the scale of 50 boards, it would probably be faster and more reliable to go with a professional board house.

As for populating the backside, I don't think that's a major problem, if cost is prohibitive I figure we can manually solder them on by hand (if the components are say decoupling capacitors and sockets.). just my 2cents.

I agree, as long as we're not talking about only reflowable compenents, I would also volunteer to solder them by hand. I've actually done a batch of boards this way in the past: had the QFN ICs reflowed by the board house and then did all the rest of the soldering myself (I don't do reflow, yet).

Sorry for the very long post, there were a lot of new posts to respond to when I woke up! I'm guessing most of you must be in Europe, or insomniacs 

[li_gangyi]

This would be very helpful! At least for the first prototypes. I looked through your blog and you look very experienced. How would you rate your confidence working with $160 ICs? How much time do you have to work on this? Nothing personal to you, but for a larger order on the scale of 50 boards, it would probably be faster and more reliable to go with a professional board house.

No problems with $160 ICs, so long as the board and part is dehumidified (I put them through an oven if I'm unsure, packing them properly with dessicant helps alot here), I can put them down properly. I have plenty of time till the end of the year to work on this at the moment, I agree that once production rates need to ramp up, it's be alot better to get it done professionally through a board house, but for the 1st few protos, I can volunteer to assemble them up. Even if the FPGAs were say pulled from a working board, I can also reball and solder them down to the new board (Not sure if we'll take this route).

I agree, as long as we're not talking about only reflowable compenents, I would also volunteer to solder them by hand. I've actually done a batch of boards this way in the past: had the QFN ICs reflowed by the board house and then did all the rest of the soldering myself (I don't do reflow, yet).

I can easily do reflow on the back side of the board as well if the need arises, I personally hate QFN packages myself, some of them that have pads that don't wrap around to the top side are a pain to work on, you can't inspect them after you've soldered them down, and you're not always quite sure how much paste to apply on the centre pad.

Sorry for the very long post, there were a lot of new posts to respond to when I woke up! I'm guessing most of you must be in Europe, or insomniacs

Haha, not quite, I'm based in Singapore at the moment, so yeah my posting times might be alittle weird.

[li_gangyi]

I'd like to add that if we were to populate the 1st few boards ourselves, before letting a board house do it, it'd be possible to test out the Vregs beforehand to make sure they work properly before putting down the $$$ FPGAs.

[fizzisist]

No problems with $160 ICs, so long as the board and part is dehumidified (I put them through an oven if I'm unsure, packing them properly with dessicant helps alot here), I can put them down properly. I have plenty of time till the end of the year to work on this at the moment, I agree that once production rates need to ramp up, it's be alot better to get it done professionally through a board house, but for the 1st few protos, I can volunteer to assemble them up. Even if the FPGAs were say pulled from a working board, I can also reball and solder them down to the new board (Not sure if we'll take this route).

This project will be very lucky to have your help!

[O_Shovah]

No problems with $160 ICs, so long as the board and part is dehumidified (I put them through an oven if I'm unsure, packing them properly with dessicant helps alot here), I can put them down properly. I have plenty of time till the end of the year to work on this at the moment, I agree that once production rates need to ramp up, it's be alot better to get it done professionally through a board house, but for the 1st few protos, I can volunteer to assemble them up. Even if the FPGAs were say pulled from a working board, I can also reball and solder them down to the new board (Not sure if we'll take this route).

So wich price you would project for such a DIMM board in production and soldering of one FPGA and the power supply and communication devices dicussed up to now, if we would decide to having you make them?
Up to now i considered Pcbcart (China) http://www.pcbcart.com/ to build and assemble this boards, but i havent asked for a price quote yet concerning the assembley(the board manufacturing would be ~350 Euro for 10 boards)

On the power question:

I ran the xilinx power estimator tool using a utilisation of 105% for all parts avaidable (Logic,DRAM,DSP,DCM,....).
It seems 5.5 W is the maximum the FPGA allows before getting cooked.
This splits to:

- 1.2 V :  4,2 A
- 2,5 V : 0,3A

This numbers heavily scale with temperature. So cooling is a major influence factor as power consumption ranged from 7.7W (no cooling, no heatsink, -T_case 125 °C)
to  4.5 W (500 LFM,huge heatsink,T_case 40°C) using the same setup.

This estmination should cover for the absolut possible maximum, but as i am totaly new to this tool i would like somebody more used to the FPGA design to confirm this results.



In addition it seems the MSP430F24xx is becoming a frequent number in this disscussion.
We should advance with our BUS deciscion anyway so please name more detailed setups on this one from now.

[li_gangyi]

I'll be happy to produce the 1st ten boards for $0, just to get this project up and running and to debug/iron out any issues we might have. Just throw the blank PCB and parts in my direction and I'll get right to work.

At this point of time, it's more of a hobby/fun thing to do then monetary, perhaps later on I'll add a donation link or something somewhere. I hope to get 1 FPGA board myself to keep. As a start I think we need to figure out if the demand is there, so I hope this will help out. I will continue to support this for as long as I possibly can, and if at any time I feel I might not meet datelines and/or the demand is starting to pickup I'll update.

I'd hate to see this turn out into a Funcube Dongle, whereby the hardware is known to exist, but can't be bought because the demand exceeds supply. So yes, I'd say I'll stop at 50 boards, unless the demand is really trickling in and we want to keep production costs low.

[O_Shovah]

I just hat a look into the MSP430F4210 documentation and found it not to support USB.
Seems the MSP430F5504 would support USB,SPI and JTAG so it houses all Bus protocols we currently seem to desire on the DIMM.

@li_gangyi

This would be great. Do i understand correctly;  you would assemble the boards, but we are going to have the PCB manufactured somewhere else ?

[Olaf.Mandel]

[...]
I think so, this MCU has 47 GPIO pins, so, I think it will be enough. Would someone be willing to make a table of all the expected signals needed? This could be for a tentative design including the maximum conceivable number of bus interfaces, so that we don't end up short later.

This is the on-DIMM connection I am working on in my design (not ready for showing, yet; maybe during the weekend). Given are all signals going from the head (FT2232, MSP230, DIMM connector, ?) to the FPGAs:

Edit: Changed topology of PROGRAM_B.

Name	Dir.	Top.	Term.
TCK	in	star	(thev.)
TMI	in	star	(thev.)
TDI	in	ring	-
TDO	out	ring	-
SCLK	in	star	thev.
SSEL	in	dedi.	-
MOSI	in	star	(thev.)
MISO	out	star	(pull.)
PROGRAM_B	in	dedi.	-
(DONE)	out	star	pull.
CLK	in	star	thev.
(IRQ_B)	(in/)out	star	pull.

Direction is as seen from the FPGAs ("input" into FPGAs, "output" from FPGAs, ...). Topology is either star, ring (TDO connected to TDI) or dedicated. Termination is per DIMM and can be either a Thevenin Termination with two 100R resistors (see UG380), a pullup resistor of 4k7 or nothing. Features put in brackets are optional.

The "trick" is to connect the SPI interface to the pins used for loading the bit-pattern. This allows loading the FPGA without using the JTAG interface. This is why IRQ_B can be an input: it is during parts of the configuration process. The IRQ signal suggested here is shared between all FPGAs: there shouldn't be so many that this starts becoming worrisome. The connections are then:

Name	FPGA-Pin
SCLK	CCLK
MOSI	DIN
IRQ_B	INIT_B

[...]

Currently, Avnet has 344 of the FGG484 packages in stock and 0 of the CSG484 packages.

I couldn't find any of the XC6SLX150-N3 at AVNET in the FGG484 package, not even listed. Maybe you get different results because you're in Europe?
[...]

No, I was talking about the -3, not the -N3 variant. The -N3 isnot in stock anywhere (I found). The trick is: you can do the prototypes with the -3 and pay a bit more and have the large run made with -N3 FPGAs.

[li_gangyi]

This would be great. Do i understand correctly;  you would assemble the boards, but we are going to have the PCB manufactured somewhere else ?

Yup, that's right, not currently equipped out at the moment to do PCBs professionally (I can only do double sided). Certaintly something we do not want to skimp on, a good quality properly routed board will make or break the prototyping stage, the layout can be optimised later to be more cost effective. I'd like to, at least for the start, not use any parts smaller then 0603 if there's no real need to, and reduce the use of QFNs, they're harder to inspect visually, and the foot print is not much bigger for QFP parts, neither is it really cheaper.

[fizzisist]

I just hat a look into the MSP430F4210 documentation and found it not to support USB.
Seems the MSP430F5504 would support USB,SPI and JTAG so it houses all Bus protocols we currently seem to desire on the DIMM.

The part number I heard was MSP430F5528, although I think any of the F551x or F552x would be fine. Let's try to pick one in a QFP package as opposed to a QFN, as li_gangyi suggested. Is anyone volunteering to lead the programming effort for this MCU?

Olaf.Mandel, is a FT2232 still necessary if we use this MCU? Is the point to avoid having to bit-bang JTAG with the MCU? Or are you only talking about on the motherboard?

[O_Shovah]

I am currently (trying) to rout a layout using the MSP430f5509 wich should be enough for our purpose or am i mistaken here ?

[Olaf.Mandel]

[...]
Olaf.Mandel, is a FT2232 still necessary if we use this MCU? Is the point to avoid having to bit-bang JTAG with the MCU? Or are you only talking about on the motherboard?

No, the MCU replaces the FT2232. The MCU could be used both on the DIMM and (with a different firmware!) on the "dumb" motherboards.

As for bitbanging: you should probably do it, as it would be completely unreasonable to add an extra chip (that costs nearly as much as the MCU!) to the boards just to avoid bitbanging. It is a bit of a hassle to code, and it can be slower than a hardware engine, but: it should be doable with just the MCU.

[O_Shovah]

I routed some part of the bus system so far.

As im new to the eagle software its hard getting used to alle the tricks you need.
The way the parts library is organised and the issues im having with changing layers are really hard to come by.

Does anyone know a more punctual way to change layers ?

[TheSeven]

5 Amps on 1.2V? No way. If we want this thing to operate stable and have some headroom for future designs, we should allow more like 10A per FPGA. The regulator you linked can supply 12A, so one of those for each FPGA might be a bit overpowered, but one for both won't work. In that case you'll need to use a bigger one (>18A).

Once again, the design of this board doesn't have to have any influence on future designs. If a second generation FPGA board is made with higher power requirements, the regulator will increase accordingly. This design needs to serve this FPGA. Period. You'll never get anywhere if you always try to plan for future designs.

When saying "future designs" I was talking about FPGA designs which might use more power than the current, not really optimized versions for that chip do. I'd rather spend $5 more on voltage regulation than having unstable boards in the end.

So I'd say that the power supply should be designed to be able to easily handle the absolute maximum possible power consumption of the FPGA, and also have some headroom, as usually both efficiency and regulation stability decrease at nearly full load of the voltage regulator. Targeting 80% maximum load might be sensible.

On the power question:

I ran the xilinx power estimator tool using a utilisation of 105% for all parts avaidable (Logic,DRAM,DSP,DCM,....).
It seems 5.5 W is the maximum the FPGA allows before getting cooked.
This splits to:

- 1.2 V :  4,2 A
- 2,5 V : 0,3A

This numbers heavily scale with temperature. So cooling is a major influence factor as power consumption ranged from 7.7W (no cooling, no heatsink, -T_case 125 °C)
to  4.5 W (500 LFM,huge heatsink,T_case 40°C) using the same setup.

This estmination should cover for the absolut possible maximum, but as i am totaly new to this tool i would like somebody more used to the FPGA design to confirm this results.

There you have 7.7W, which is ~6.5A, which already exceeds the 12A max for two FPGAs. Also, please don't trust those tools. They're estimating something like 3 watts for my virtex 5 design, and judging from the amount of heat generated the real power dissipation must be way higher. You're only really safe if you assume 100% of the gates toggling on each clock cycle. SHA256 is surprisingly close to that...

[li_gangyi]

I routed some part of the bus system so far.

As im new to the eagle software its hard getting used to alle the tricks you need.
The way the parts library is organised and the issues im having with changing layers are really hard to come by.

Does anyone know a more punctual way to change layers ?

LAYER <layer number or layer name> will change layers. I prefer to type more then click around.

[fizzisist]

You can also click the middle mouse button when routing.