BitFury Design, Licensing, Mass production

[bitfury]

The proposed board. Mechanical drawing. Board size is 190 (width) x 160 height. Total height is about 32 mm (counting board width, chip width, and heatsink height of 28 mm).

1. Thanks to guys who showed me http://www.raspberrypi.org/ controller
    That controller has nice GPIO outputs, that could drive FPGA legs. And that controller can drive multiple boards at once.
    So no microcontroller per-board required.

    Just some GPIO pins would be used for SATA.

    It costs only $25 per item.... Which is nice, so when connected to multiple boards, cost of controller would be little.

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

    SPI interface to talk with bitstream and another SPI to program bitstream plus generic RESET.

    And another SATA interface, again SATA3 input, SATA4 output:

PIN1 - TMS
PIN2 - TCK
PIN3 - TDI
PIN4 - GROUND
PIN5 - UNUSED
PIN6 - RST
PIN7 - TDO

    On board design will contain buffers and resistors connected in a way, that connecting these cables in wrong direction won't cause problems.

3. PSU is TPS40090 based 100 Amp power supply, but used at lower amps to get performance better, in overclocked
    condition performance at about 90% expected, in normal conditions about 92-94%. Amps delivered are quire high.
   Also PSU input would be in 10 - 15 V range, so it can be connected to automotive power supplies / lead acid backup /
   wind energy storage batteries directly if someone would like to do it.

4. Connectivity of 12 V is via MOLEX socket (4-pin) and also can be connected via bolt connection to copper bus / bar,
    which could be used to mount boards as well. (This 12 V connection is specifically for Greg). However in our chassis
    we may put separate cables with molex connectors and add fuses on them. If will be requested - we may add fuse on
    board by adding special PCB layout, that will burn if there will go about 20-30 amps current suddenly.

5. Board has connectors on TOP to not distract airflow, and also they would sit tightly there, when pressed by chassis top.

6. There will be 3 jumpers around PSU that would select voltage level - 1.15 V to 1.5 V in 0.05 V steps. We will set it up @ 1.3 V which is safe by datasheet. That way performance could be improved up to 280-290 Mhz clock depending on chip when voltage rised to 1.5 V. TSP40090 actually useful for that.

7. Every board would have 6 jumpers to select board identifier, that can be later used by software.

8. Communicating with board will be done using 32-bit SPI. Basically you first supply 32-bit command and then either read or write 32-bit word of data. 32-bit command would contain address (6-bit board number 3-bit chip number), command type (read or write 1-bit) and register number (5-bit) totalling 15 bits AND some framesync/error check information for quick bus resetting etc.

This communication can be made practically with every GPIO software, and even with LPT-port via resistors, at any clock rates, which will not produce significant error rates. So board will be as cheap as possible then. WITHOUT any RS-485 or USB interfaces. And without any FLASH chips on it.



In next message I'll write about temperature simulations... For maximum overclocked mode - 18W per chip on 1.5 V / 18W on power supply dissipation and 1 m/s airflow...

Then I'll go drawing draft for 4-U device.

[1713581076]

1713581076

[nedbert9]

In light of BFL's SC model announcement can we discuss realistically what the chances are of protocol change that would obsolete such ASIC solutions?  If such an ASIC solution comes to pass what percentage loss would a miner have to bear, compared to ASIC, to remain with FPGA-based solutions to mitigate protocol change risk.

[bitfury]

Ok. I put this board into single air channel virtual with flow setup - 293.2 K input (that's 20 degrees Celsius).

And here's thermal images I got:

Chips (Temperature of first chip in airflow - 309 K / 36 Celsius second chip - 315 K / 42 Celsius
and power supply - 326 K / 53 Celsius - however this is rude to model PSU that way... there will be much hotter parts like MOSFETs and inductors and much colder like drivers):



Board (about 308K / 35 C beneath second chip and 299K / 26 C  beneath first one ):



Air flow (cuts middle of heatsinks - air at 1 m/s overheats at about 20 C - average outlet temp.):



Now let's do some calculations to know temps inside of chip based on Spartan6 datasheets (spartan6-pack-ug385.pdf)
FG(G)484 23x23 says - 3.7 C/W junction-case and 6.3 C/W junction-board.

Let's assume Tcase = 42 Celsius Tboard = 35 Celsius and calculate Tj. Having P = 18 W.

Solve[{Tj - Pjc*3.7 == 42, Tj - Pjb*6.7 == 35, Pjb+Pjc == 18}, { Tj, Pjb, Pjc}]

Tj = 82.5 ; Pbj = 7 ; Pjc = 11

However that are not exactly true, as in model I've modeled device as piece of copper. So real temperature will be in range of about Tj = 86 degrees.

For INDUSTRIAL chip it will fit perfectly WITH 20 degrees C air input.

With 34 degrees inlet air Tj will be at absolute maximum.

So this rises questions whether we should put two such boards in serial way or not.... As IF COLD AIR SUPPLIED, we definitely can, if not - then we're screwed.

Also if we not overclock it - temperature raise would be proportionally less - to about 65 degrees C from 20 degrees in a chip. So if chips are not overclocked, then 2 such boards can be put in serial way leading to same inlet air requirements.

Also this shows that it is better to use industrial chips, although they are $4 more expensive.

Also we can increase air speed twice, and we would get only about 10 degrees temperature rise and install two boards.

So... it seems to better go to sleep and then think about this concept....

Within single 0.2 m space of 4U device one can fit 12 of such boards consuming max 1.95 kW of power @ max. clock. / max. voltage. In normal condition 1.3 kW only. But only 72 chips there.

In denser design it could be 4 kW / 2.6 kW per 4-U chassis, which seems to be OVERKILL. How to cool them down then....

[rjk]

Could you have the chips offset from one another so that the hot air from the first ones isn't warming the second ones?

Even if they weren't perfectly offset, you could use strips of plastic to duct the air around them.

[bitfury]

Could you have the chips offset from one another so that the hot air from the first ones isn't warming the second ones?

Even if they weren't perfectly offset, you could use strips of plastic to duct the air around them.

Yes, this can be done... But let me save how air temp. looks at output....



You mean shifting second row of boards a bit, so inlet air would be on these darker areas, right ?

11 boards in one row and 12 boards in second one ? This would help a bit I believe, but mostly volume would be hotter.

But still, isn't 2.6 kW for 4U too much ?

About plastic duct - that's bad idea, as this would add assembling difficulties = costs.

[bitfury]

In light of BFL's SC model announcement can we discuss realistically what the chances are of protocol change that would obsolete such ASIC solutions?  If such an ASIC solution comes to pass what percentage loss would a miner have to bear, compared to ASIC, to remain with FPGA-based solutions to mitigate protocol change risk.

Easily. BitCoin is small network actually. And it is small, because it was started as e-currency. "Normal" people wouldn't even understand what these hashes are about. This makes it actually interesting, because here on forum, people are smarter than average. But payment is actually application... Value is in things that you can buy with that e-currency. So if you start with thing, and then get nice e-currency for that thing - then all of your audience will use said e-currency. So when bitcoin matures and ASIC control will be in one hands - then people who would start integrating it with bittorrent world would make choice of another currency, with protected protocol depending on their need. And yes - their audience impact CAN BE MUCH BIGGER :-)

Current our position with BitCoin - we're trying it, digging into deeps What actually scares a bit - that exchanges, and people who do business with BitCoins should BUY HASHING POWER FOREMOST.... But it seems not to happen....

[DiabloD3]

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

    SPI interface to talk with bitstream and another SPI to program bitstream plus generic RESET.

    And another SATA interface, again SATA3 input, SATA4 output:

If you're going to use multiple different connections all using SATA connectors, remember that you can buy SATA cables in different colors so people don't accidentally plug the wrong ones in when doing maintenance.

[bitfury]

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

    SPI interface to talk with bitstream and another SPI to program bitstream plus generic RESET.

    And another SATA interface, again SATA3 input, SATA4 output:

If you're going to use multiple different connections all using SATA connectors, remember that you can buy SATA cables in different colors so people don't accidentally plug the wrong ones in when doing maintenance.

I'll first do precaution that nothing bad happen if they do...
And second point - there will be only one cable from one board to another board...
So one cable per board, short one like 4-inch SATA. When I saw them I loved them.

But - JTAG chain is nice to have for other applications for example.

Anyways thanks for JTAG idea!

[rjk]

Forgive me for hacking up your cool picture, but this is what I meant when I said offset:


Is it worth it?

[bitfury]

Forgive me for hacking up your cool picture, but this is what I meant when I said offset:

// pic eaten by wolves ///

Is it worth it?

We did it in hardware already. Yes it worth it, but not much, and in this design even less, than when we could offset whole daughter boards so they get colder air on their bottom side. It would be non-uniform then, and central chip would be hottest. You may see what we did in following pic... But I really want to remove that chip plug-and-pray technology:

[rjk]

Beautiful. I like the pluggable design.

[bitfury]

Beautiful. I like the pluggable design.

Well, me too. BUT there's big cons... Soldering these plugs is done by hand... and error-prone when a lot of them should be soldered... within some of them were balls of solder inside - so it was not possible to put there daughter card without pressing it too much... Then when daughter card is so small, you can only glue heatsink there and it is very RESPONSIBLE work... We did it in two person for about 3 days, and it can't be given to plant, because they can screw things... As if you put too much glue, Spartans could fry.... And we're not cooks :-)))) Fried spartan is expensive and not tasty dish )))
Then if you get it bigger it becomes 60x60 mm daughter card to put desirable-sized heatsink on it... And it wastes too much space :-( Those small heatsinks that you saw on other boards are not suitable well for such power output from Spartan.

[AzN1337c0d3r]

However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain.

18 AWG wire is allowed to carry 14 amps according to NEC:

http://en.wikipedia.org/wiki/American_wire_gauge

In fact, I'm using them in an application that carries 13A (4x Delta AFB1212GHE-CF00, which draw 3.24A each according to the label).

The wires themselves barely get warm, although if I try to pull 13A off one connector, it gets pretty hot. Probably exceeding some kind of molex pin specification there. I moved each fan onto its own connector after that.

[MrTeal]

However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain.

18 AWG wire is allowed to carry 14 amps according to NEC:

http://en.wikipedia.org/wiki/American_wire_gauge

In fact, I'm using them in an application that carries 13A (4x Delta AFB1212GHE-CF00, which draw 3.24A each according to the label).

The wires themselves barely get warm, although if I try to pull 13A off one connector, it gets pretty hot. Probably exceeding some kind of molex pin specification there. I moved each fan onto its own connector after that.

You can do that, but you really should upgrade the wire for 13A. I doubt the fans are actually pulling that much, but even outside the heating the extra inefficiency is unneeded. At 13A you'll be dissipating a little over 1W/ft just in the wire.

[bitfury]

However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain.

18 AWG wire is allowed to carry 14 amps according to NEC:

http://en.wikipedia.org/wiki/American_wire_gauge

In fact, I'm using them in an application that carries 13A (4x Delta AFB1212GHE-CF00, which draw 3.24A each according to the label).

The wires themselves barely get warm, although if I try to pull 13A off one connector, it gets pretty hot. Probably exceeding some kind of molex pin specification there. I moved each fan onto its own connector after that.

Well if you put 2 of them - then no problem... I use 1.5 sq. mm (about 14 - 15 AWG), and wired are at about 50 C compared to 25-30 C surrounding temperature on 8 Amps. If you consider that losses are square of current - i.e.
when you get say 14 amps - you'll get 14*14/8/8 = 3 times more power dissipation in wires.... You'll get into troubles.
Because device is labelled something does not means they consume all of that current!!! Use test equipment to measure.
Say - there's 3 x 3 kW power supplies on BF-110 - but that does not mean that it consumes 9 kW (rack alone). Or you can see 2.5 kW on wash machine, but it would actually consume on average more like 0.5 kW, because heater element is on for short periods of time. What is labelled usually peak power consumption.

[AzN1337c0d3r]

Use test equipment to measure.

Fans plugged in = 593W
Fans pulled out = 453W

Wattage of 4x Delta = 140W at the wall. Power supply is 89% efficient at that load, so 125W DC. 

About 10.5A on that 12V connector. Like I said the Wire barely gets warm and I'm sure I could put another Delta on it if I'd like without it melting.

[TheSeven]

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

Did you look at the internal layout of a SATA cable? IIUC most of them look like this:

...which doesn't seem to be compatible with your pinout.
You'll have to deal with pins 1, 4 and 7 being connected internally and being laid out fundamentally different (as a shield/foil) than the data pins.

[bitfury]

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

Did you look at the internal layout of a SATA cable? IIUC most of them look like this:
 // CUT IMAGE //
...which doesn't seem to be compatible with your pinout.
You'll have to deal with pins 1, 4 and 7 being connected internally and being laid out fundamentally different (as a shield/foil) than the data pins.

No I didn't ... Big thanks... Saved my time... As otherwise I would deal with it during PCB schematic / layout stage... Now have to guess how to live with that... Interesting to know, whether all grounds connected or no...

With JTAG it seems not be problem to reorder connections, but with 2 SPIs it seems to be.