Modular FPGA Miner Hardware Design Development

[Olaf.Mandel]

We are talking about FPGA temperature at the moment. So here a question: do we add one or more temperature sensors to the DIMMs? E.g. the Digikey 296-27725-1-ND, which has an I2C interface and costs 2.15EUR in prototype quantities.

And I just realised: the backplane needs different I2C interfaces for each DIMM, if there is an I2C bus at all! Or is there something like a breakout chip: if it's address is given on the bus master side, it forwards traffic to the second attached I2C bus, acting as a bus master there?

[1713923912]

1713923912

[1713923912]

1713923912

[1713923912]

1713923912

[1713923912]

1713923912

[O_Shovah]

So think we may agree on the following points:

- We will develop a hybrid solution, usuable for both the Modular and the single board use

- Each board hast it own power supply via both 12 V and 5 V through a 240 pin DIMM connector. And if not connected to the mainboard or needs further power it is supplied by the 12-16 V  barrel input or the 12 V Molex 8981 wich makes nessesary for a chip capable of transforming 12-16 V to 5 V and lower Voltages needed for all components on the DIMM.

The further i surely agree that we will have to apply heatsinks to the FPGA's. But i also think that these will not need such heavy weighted parts as on graphic cards, so the DIMM slots should be capable of holding that weight.(When i look at the RAM DIMM's in my Pc i also wonder how they may hold such big heatsinks)

Edit:

I would appreciate it if we at first may decide on a more or less final concept for the power supply before disscussing other parts and then move on to the next topic  eg Bus systems ?

  

[Olaf.Mandel]

Here is a table that hopefully answers the question if one should use a rail voltage in the power supply or not. I got the table by using National Semiconductor Webbench with the following parameters:

• V_in = 11V to 20V
• T_ambient = 30°C
• 4x Xilinx Spartan-6 XC6SLX150, each using these voltages / currents:
  
  • V_int = 1.2V @ 5A
  • V_aux = 2.5V @ 0.3A
  • V_io2 = 2.5V @ 1.6A
  • Nothing on the other banks

The currents were suggested by Webbench, I don't have the output of ISE power analysis. The 1.6A for V_io2 are probably strongly overestimating our actual usage, I hope V_int is not underestimating it. I didn't use 1.8V for V_io2 in order to have fewer switchers.

Webbench lets you specify if you prefer to use their simple switcher(R) modules or not (though it doesn't exclusively use those). Both settings were tried.

Use modules	Allow rail	Rail [V]	Efficiency [%]	Footprint [mm2]	Price [USD]	Part count	Used parts
N	N	-	80.53	2485	25.60	86	3x LM5117
N	Y	5V	87.52	2162	24.70	76	LM3150, 3x LM21215A-1
Y	N	-	73.86	2294	48.86	32	LMZ12008, 2x LMZ12010
Y	Y	5V	85.52	2203	34.63	68	LMZ12010, 3x LM21215A-1

It seems using a supply with a rail voltage is always cheaper, smaller and more efficient. Don't take the prices to heart, though: these are for large quantities. Assume a factor of 2x in actual prototype price!

Did I or the program make a mistake? If not, I suggest we plan on having a two-stage supply on each DIMM. Then the question is simply: does the backplane supply the rail voltage (and not use the first stage of the DIMMs power supply) or does it supply 12V?

The situation may be different with switchers from other manufacturers. Examples?

[O_Shovah]

I dont want to rush things but i'd like to ask you to decide please.

As i see it, there are two options:

Either we use 12- eg 16 V on both the DIMM powersupply rails as also on the molex and barrel connectors and transform them to all voltages needed on the DIMM board.
This should work for the mainboard based as well the single board solution but does not use the 5 V rail provided by an ATX PSU, but might produce a loss in energy efficiency.


Or we use both a 12 V rail and an additional x Volt rail on the DIMM connections. 12 V for the FPGA supply and the x Volt rail for other parts located on the DIMM.
This should increase power efficiency and use the capabilitys of the ATX PSU to its full extend.


Please give me some kind of vote for one of the solutions or corret me if i got one of them wrong.    

Edited

[Olaf.Mandel]

I dont want to rush things but i'd like to ask you to decide please.
[...]
[using the higher voltage on the DIMM connector] should work for the mainboard based as well the single board solution but does not use the 5 V rail provided by an ATX PSU, but might produce a loss in energy efficiency.

Or we use both 12 V and 5 V on the DIMM rails. 12 V for the FPGA supply and the 5 V rail for other parts located on the DIMM.
This should increase power efficiency and use the capabilitys of the ATX PSU to its full extend.

Please give me some kind of vote for one of the solutions or corret me if i got one of them wrong.     

Hi,

as I wrote before, I don't think that supplying a high voltage to the DIMM means a hit in efficiency. So I vote for supplying only 12V to 19V to the DIMM, both when acting stand-alone as well as when being plugged into the backplane.

[lame.duck]

I doubt if this work, 2200 mm2 are 2,2x10 cm, this is a quarter of a normal DIMM and even if we could use a bigger PCB the FPGA area and the power supply are schould de clearly separated from each other and we  don't want the in rush current route under the FPGA. Of course we could place the power supply down at the connector side but then the heat sinks will add mechanical instability then ...

How about the Euro format, the format is a little largisch but you could mount heavy heat sinks on them. There are also formats other than 160x100 mm, e.g. 100x100 and 220x100, if there are also fomats defined for 1 and 2 HE, i dont know, this would be a exercise to the reader.

[O_Shovah]

I personally want to add that i prefere the first option as far as i understood them both.

Most PSU's may output more than their nominal power on the 12 V rail so leaving the 5 V rail out wouldn't result in a loss of power.

I case the DIMM is sloted in the DIMM socket while operating on the motherboard it processes the voltage for the FPGA core from 12 V to quarantee stability anyway.
the 5 V rail of the DIMM socket is used to supply the FPGA peripherals in this case.

And in case the DIMM is used as a single miner card it has no 5 V supplied via the DIMM rails as it is just supplied via the Molex or the barrel connector so it has to be generated from the 12-16 V anyway. Maybe the USB voltage could be used in this case but im not sure if this is enough.  

[Olaf.Mandel]

[...]
the 5 V rail of the DIMM socket is used to supply the FPGA peripherals in this case.
[...]

All periperals that I can think of need the FPGA supply voltages, not 5V. The oscillator (say, 100MHz) needs to run at V_io. The level shifters need two supply voltages: V_io on the FPGA side and whichever is the signalling standard on the other side. For the mode where the card is slotted into the backplane, the backplane could provide this (say, 3.3V). If the USB chip is used in standalone configuration, this would be USB supplied (either 5V or 3.3V).

[Olaf.Mandel]

I doubt if this work, 2200 mm2 are 2,2x10 cm, this is a quarter of a normal DIMM and even if we could use a bigger PCB the FPGA area and the power supply are schould de clearly separated from each other and we  don't want the in rush current route under the FPGA. Of course we could place the power supply down at the connector side but then the heat sinks will add mechanical instability then ...

This depends on the number of FPGAs to put on one board. It will not scale linearly, though!

How about the Euro format, the format is a little largisch but you could mount heavy heat sinks on them. There are also formats other than 160x100 mm, e.g. 100x100 and 220x100, if there are also fomats defined for 1 and 2 HE, i dont know, this would be a exercise to the reader.

How about this: someone first draws the thing (even with small shapes on a vector-graphics program is fine at this point). Then decide how many FPGAs to put on one board based on size relative to a DIMM slot. Think about FPGA package size: I drew a large package for my prototype because I was not sure about prototype cost for the smaller layout.

[Olaf.Mandel]

[...]
And I just realised: the backplane needs different I2C interfaces for each DIMM, if there is an I2C bus at all! Or is there something like a breakout chip: if it's address is given on the bus master side, it forwards traffic to the second attached I2C bus, acting as a bus master there?

There are suitable multiplexers. An example of one input to 8 outputs is the NXP PCA9547PW, which supports up to 400kHz. Digikey 568-3381-5-ND for 1.90EUR.

[O_Shovah]

 So one additional interesting question is wich voltages are needed onboard the DIMM for FPGA,  EEPROM, Bus chips and other parts ?

So to wich level  do we have to convert the 11-20 V input ?

[Olaf.Mandel]

So one additional interesting question is wich voltages are needed onboard the DIMM for FPGA,  EEPROM, Bus chips and other parts ?

So to wich level  do we have to convert the 11-20 V input ?

I only looked at the Xilinx Spartan-6 FPGAs, specifically the XC6SLX75-3FGG484 and XC6SLX150-3FGG484. Assuming you are not interested in bitstream encryption, there are three different supplies:

VCCINT = 1.2V
VCCAUX = 2.5V or 3.3V
VCCO_# = 1.2V or 1.5V or 1.8V or 2.5V or 3.3V
where # runs from 0 to 3 (or 1 to 4, depending on your reference).

When booting the device, the combination VCCAUX=3.3V and VCCO_2=1.8V is forbidden. Also, VCCO_2 must be one of 1.8V, 2.5V or 3.3V.

As I didn't want to change voltages during operation and as I wanted minimally small voltages, this leaves the combination:

VCCINT = 1.2V
VCCAUX = VCCO_2 = 2.5V

I connected the other banks to 2.5V also, but they will not pull a lot of current. Please note that while both pull 2.5V, the supplies for VCCAUX and VCCO_2 are not identical! You can use the same switcher, but because VCCAUX powers the frequency synthesizers (among other things), it needs an extra filter on the input.

As for other chips: the clock oscillator can run off 2.5V, any level shifters would run off that voltage, too, on their FPGA-facing side. The other side should be powered by the same supply that powers the chip that connects to the level-shifter. So the supply of the USB chip or directly powered by the backplane.

The I2C chips (EEPROM and temperature sensor: we still need to decide what to add to the board) should run off of VCCO_2, also: this way they can be on the same side of the level-shifters as the FPGAs.

This only leaves the USB chip: This can be either powered from the USB bus (my preferred option) or off of the 5V voltage rail. My reason to go bus powered: this way the USB chip can send the FPGAs sleeping if the host computer goes down, saving electricity cost in case of a network outage or a software bug that crashes the host.

[marcus_of_augustus]

watching

[O_Shovah]

I had a look for some voltage regulators asuming we choose the option of a wide range input of ~11-18V and one FPGA not using more than 15W including losses.

Using the values provided by Olaf.Mandal found:

EDITED

Voltage regulator 18 V to 2.5 V  2.5 W  2x per FPGA http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=R-78AA2.5-1.0SMD-ND

Voltage regulator 18 V to 1.2 V 1 A   2x per FPGA http://www.mercateo.com/p/139-1467363/V_REG_LDO_1A_1_2V_SMD_Typ_FAN1112SX.html

The size of the DIMM DDr 2 socket http://portal.fciconnect.com/res/en/pdffiles/doc_search/10005639.pdf

is ~133 mm in length and in our case the height is only limited by the strenght of the PCB material.

I will add a rough drawing of a possible layout later.

[O_Shovah]

So here is a first rough sketch of how the location of the parts could look like with the use of two FPGA's on one board.
It will certainly need aditional capicitors an resistors for voltage regulation.
Additional parts like EEPROM's or sensors havent been accounted yet.
This setup is using both front and back of the board for components.

All sizes are in Millimeters

http://imageshack.us/f/685/fpgaboardsketch.png/ ( i may not post images directly yet)

This is just a first idea though so please feel free to criticise and change it.


I asume we are going to have a 4 layer board or more.

[Olaf.Mandel]

[...]
Voltage regulator 18 V to 2.5 V  2.5 W  6x per FPGA http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=R-78AA2.5-1.0SMD-ND

Voltage regulator 18 V to 1.2 V 1 A   2x per FPGA http://www.mercateo.com/p/139-1467363/V_REG_LDO_1A_1_2V_SMD_Typ_FAN1112SX.html
[...]

Can't say I agree with these values. If you design for a XC6SLX150, then you need:

• 5A @ 1.2V = 5 regulators
• 1.9A @ 2.5V = 2 regulators

So you planned too many of the 2.5V and too few of the 1.2V regulators. Did you mix them up?

[Olaf.Mandel]

[...]
This setup is using both front and back of the board for components.

This would increase the manufacturing cost somewhat: two different masks for paste.

[...]
This is just a first idea though so please feel free to criticise and change it.
[...]

There are actually two comments:

• The mini-USB connector, the barrel connector (and possibly even the Molex connector) are very close to the edge. Don't they interfere with the lever for the DIMM socket lock?
• The mini-USB connector is facing down: I assume this is to discourage plugging anything in while the board is sitting in the DIMM socket. But what about the thickness of the plug? The connector should probably go to the edge of the board and just hope that the user doesn't plug it in...

[TheSeven]

I had a look for some voltage regulators asuming we choose the option of a wide range input of ~11-18V and one FPGA not using more than 15W including losses.

Using the values provided by Olaf.Mandal found:

Voltage regulator 18 V to 2.5 V  2.5 W  6x per FPGA http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=R-78AA2.5-1.0SMD-ND

Voltage regulator 18 V to 1.2 V 1 A   2x per FPGA http://www.mercateo.com/p/139-1467363/V_REG_LDO_1A_1_2V_SMD_Typ_FAN1112SX.html

The size of the DIMM DDr 2 socket http://portal.fciconnect.com/res/en/pdffiles/doc_search/10005639.pdf

is ~133 mm in length and in our case the height is only limited by the strenght of the PCB material.

I will add a rough drawing of a possible layout later.

LDOs for 1.2V? Seriously?
You're talking 6% efficiency and 170W of power dissipation in the regulator for just one FPGA?

[O_Shovah]

I am sorry guys. Seems like it was really to late for me yesterday.

Can't say I agree with these values. If you design for a XC6SLX150, then you need:
    5A @ 1.2V = 5 regulators
    1.9A @ 2.5V = 2 regulators
So you planned too many of the 2.5V and too few of the 1.2V regulators. Did you mix them up?

Yes i did. I changed the numbers accordingly.

[...]
This setup is using both front and back of the board for components.

This would increase the manufacturing cost somewhat: two different masks for paste.

If we want to use multiple FPGA's on one DIMM board without exeeding a certain height we will be forced to use both sides for voltage regulation and IO devices.
I know this will make it more expensive but i'm affraid thats the way to go at least in a further series.   

There are actually two comments:

• The mini-USB connector, the barrel connector (and possibly even the Molex connector) are very close to the edge. Don't they interfere with the lever for the DIMM socket lock?
• The mini-USB connector is facing down: I assume this is to discourage plugging anything in while the board is sitting in the DIMM socket. But what about the thickness of the plug? The connector should probably go to the edge of the board and just hope that the user doesn't plug it in...

The socket lock should just pass by them both. You certainly may move them a little to guarantee they are no obstacle.I will fine tune this once i draw it in 3D CAD.
You are right about the USB connector we may locate it at the edge of the board. But im may not say if it is  nessesary to make it inaccessable while the board is connected via the DIMM bus system.

LDOs for 1.2V? Seriously?
You're talking 6% efficiency and 170W of power dissipation in the regulator for just one FPGA?

You've got a point there. As i allready said i did this by mistake.
I had a look into PMIC DC DC switching regulators with 5 A 1.2A out and 18V input,but they also barely strike 70% eff
Also i asume this PWM signal may be problematik for a constant voltage supply.

I would appreciate it if you could have a look into that subject and name some parts you consider the best for this application.     

[O_Shovah]

As there have been no further objections so far i will fix the following points regarding power supply:

- Each DIMM board features an wide range input ~11-18V over either a Molex 8981 or a Barrel 2.5/5.5 connector.

- In addition the DIMM socket provides a 12 V rail supplied by an ATX PSU via the Mainboard if in Modular use.

- The voltage regulation providing the voltages needed for the components on the DIMM is located on the DIMM itself.