Klondike - 16 chip ASIC Open Source Board - Preliminary

[Bicknellski]

Just to put a bad diagram to your plans for Klego's.... which orientation are you thinking again a vertical or horizontal arrangement?

Fans at the end of either end of the tunnel would work in either arrangement. The horizontal version can fit in side a rack mount 19" wide I believe. The horizontal version would also less prone to heating I think given the density.

As BkkCoins mentioned, There needs to be more spacing on the front side of the boards. I am working on a stacking design with the boards laying flat, heatsinks facing each other, with standoffs to provide spacing.

Figure at minimum 12mm on the front of the board for clearance, double that if you want to use vertical pcie connectors. Keep in mind the wiring needed for the board, you don't want to design anything that will kink the wires or make them difficult to replace. I'll have some initial designs up in the next couple days.


For the SATA debate. I think it's a very bad idea to use SATA. As mentioned earlier, they aren't designed to pull that much power. Additionally, there is more waste from power supplies that are only going to be powering the boards. Most power from the PSU is on the 12v rails. The best density would be to daisy chain pcie connectors to each set of boards, using as much of the 12v rail as possible.

+1

Thanks for this Steamboat... the SATA idea is something for a 2 to 3 board configuration at best and what BKKCoins was suggesting is that it would have to be redesigned board for a later time. I will take your and BKKCoins design notes and look for off the shelf casing particularly something in the 4U variety as I want 512 chips Klego'ed in a horizontal fashion.

[ecliptic]

BKK,

Please comment allten post about I/O timing.
10X

This did look tricky at first and a software only solution would be difficult or impossible. I went to bed last night thinking about how to make this work. Woke up this morning with the answer.

I'm using a single NOR gate on the REPORT_N / _P outputs. This combines the two data signals to extract a separate CLK. I use the EUSART on the PIC in Synchronous Slave Mode. This accepts stateless data RX, CLK as input and shifts in data at high speed. I take REPORT_N as data (either will do) and the combined CLK. The UART has a 2 char FIFO and interrupt, so the instruction cycle isn't a constraint any more. Oh, and I've added a small capacitor to delay the clock edge slightly because the NOR gate doesn't have much delay. This should make data sampling reliable.

I already had the UART pins routed to the ASIC but was expecting asynchronous data. So had to change plans on that.

Total cost for solution is 0.12 gate, 0.015 cap = $0.135.

I've already discussed this with allten and we also discussed being able to remove the oscillator and use the internal oscillator in the PIC. This saves about $1 cost but will depend on whether the ASIC can live with the jitter as the PIC PLL is not as good as an oscillator. I don't know how sensitive the ASIC PLL will be, but I'll leave the oscillator pads on board so that either method can be used.

 What does 10X mean?

I believe that the spec on the avalon ASIC is 10ps for clock jitter.  that's pretty precise.

[BkkCoins]

Just ordered first batch of 20 - K1 test boards. Yay.

[wrenchmonkey]

Just ordered first batch of 20 - K1 test boards. Yay.

Hooray! Now for some chips! Avalon, if you're reading, we need those developer chips sent out PRONTO! 

[BkkCoins]

I believe that the spec on the avalon ASIC is 10ps for clock jitter.  that's pretty precise.

I'm a little doubtful it can be removed but I'll test it. To allow for the possibility I moved the clk enable line to be the same as the PIC clk out pin. That way when I want to test this I can just remove the oscillator and jump the enable pin to the osc clk pin, feeding the PIC clock to the ASIC.

The oscillator is 50ppm and the PIC is undocumented currently, though only requires 3% for it's own input so may not be nearly as stable.

[ecliptic]

Sorry I've been out of the loop for ~2 weeks.  Is this changing the clock to the ASIC for OC purposes? cost savings?

I thought perhaps the ASIC has a PLL inside to allow you to adjust clock frequency in software

[ecliptic]

One thing to look at with testing is the realibility of contact to such a large heatsink

Sharing one common block between all 16 chips, and trying to move the heat between them just with an exposed copper square per chip, no local mechanical force holding the heatsink in contact there.  Could be very tricky

May want to get an IR thermometer (FLIR cameras are great but expensive), measure the temp on the top of each IC to see if they're close.  May not be able to measure the temp on the heatsink itself because the emissivity of aluminum and reflections can fuck everything up, but it's usually good on the black top of a chip package.  If you can measure the heatsink though, see if there's more or less heat at the areas where it should be connected to the QFN pads.  Some may not have good contact.

Can also try making the screws looser, run it for a while to see if heat goes up (from worse thermal contact), make looser, check temp, repeat until it gets too hot.  the chips with bad contact should heat up first or faster.

I just really have not seen this sort of shared heatsink design before..

Going through the files atm (kicad seems to be unable to open them so going to try gEDA for the gerbers) -- may be able to make the exposed copper area for the QFN chips larger?  I'm looking at Avalon's design and not sure why theirs is so small.. or why the thermal paste is fricken everywhere.

edit : oh i see now, Avalon design has ground over basically the entire bottom layer as a plane.  then they have ~24 screws thermally connected and pull the heat from the plane to the heatsink.  not bad

[fasmax]

BkkCoins, you are doing a great job!

Your project is a first reason I joined this forum. I'm going to buy about 500 chips, so I will need about 30 PCBs.

I feel this summer is going to be hot!

Thanks. It's already averaging around 35-37C here.

BTW I've got the code for the pre-calc hash figured out. After a helpful push from chaoztc I did a little digging and managed to find the needed code and verify it looks like it does the right calcs for the Avalon. Found it in the opencl driver. Just ask if you need this for your own work. Maybe this is old hat to sha experts but all the bit fiddling math cycles are a blur to me.

Does the pre-calc need to be done for each miner?
Any idea how much time the pre-calc takes on a Raspberry PI?
Have you though about using a more power full PIC so you do the pre-calc on the K16 board?
Thanks.

[BkkCoins]

BkkCoins, you are doing a great job!

Your project is a first reason I joined this forum. I'm going to buy about 500 chips, so I will need about 30 PCBs.

I feel this summer is going to be hot!

Thanks. It's already averaging around 35-37C here.

BTW I've got the code for the pre-calc hash figured out. After a helpful push from chaoztc I did a little digging and managed to find the needed code and verify it looks like it does the right calcs for the Avalon. Found it in the opencl driver. Just ask if you need this for your own work. Maybe this is old hat to sha experts but all the bit fiddling math cycles are a blur to me.

Does the pre-calc need to be done for each miner?
Any idea how much time the pre-calc takes on a Raspberry PI?
Have you though about using a more power full PIC so you do the pre-calc on the K16 board?
Thanks.

It would be done for each work unit sent. It shouldn't take much time as it's just 3 rounds of a bunch of rotates, xors, ands etc. It may be feasible on the PIC even but I haven't looked yet. Since work will be queued I doubt it will slow anything down on either a PC or RasPi or PIC, but the PIC has very limited program memory and much is already used for the USB stack. Will look at that some time later.

[mike2kt]

BkkCoins, you are doing a great job!

Your project is a first reason I joined this forum. I'm going to buy about 500 chips, so I will need about 30 PCBs.

I feel this summer is going to be hot!

Thanks. It's already averaging around 35-37C here.

BTW I've got the code for the pre-calc hash figured out. After a helpful push from chaoztc I did a little digging and managed to find the needed code and verify it looks like it does the right calcs for the Avalon. Found it in the opencl driver. Just ask if you need this for your own work. Maybe this is old hat to sha experts but all the bit fiddling math cycles are a blur to me.

Does the pre-calc need to be done for each miner?
Any idea how much time the pre-calc takes on a Raspberry PI?
Have you though about using a more power full PIC so you do the pre-calc on the K16 board?
Thanks.

Simple explanation: The mining process does TWO iterations of SHA256 per work unit. The controller does the first round SHA256 (1 time) then passes the value on to the “hashers” for the second iterations of SHA256 (up to 2^32 or 4,294,967,296 times) looking for a golden nonce (in parallel).

[BkkCoins]

One thing to look at with testing is the realibility of contact to such a large heatsink

Sharing one common block between all 16 chips, and trying to move the heat between them just with an exposed copper square per chip, no local mechanical force holding the heatsink in contact there.  Could be very tricky

May want to get an IR thermometer (FLIR cameras are great but expensive), measure the temp on the top of each IC to see if they're close.  May not be able to measure the temp on the heatsink itself because the emissivity of aluminum and reflections can fuck everything up, but it's usually good on the black top of a chip package.  If you can measure the heatsink though, see if there's more or less heat at the areas where it should be connected to the QFN pads.  Some may not have good contact.

Can also try making the screws looser, run it for a while to see if heat goes up (from worse thermal contact), make looser, check temp, repeat until it gets too hot.  the chips with bad contact should heat up first or faster.

I just really have not seen this sort of shared heatsink design before..

Going through the files atm (kicad seems to be unable to open them so going to try gEDA for the gerbers) -- may be able to make the exposed copper area for the QFN chips larger?  I'm looking at Avalon's design and not sure why theirs is so small.. or why the thermal paste is fricken everywhere.

edit : oh i see now, Avalon design has ground over basically the entire bottom layer as a plane.  then they have ~24 screws thermally connected and pull the heat from the plane to the heatsink.  not bad

Thanks for the input. I have an IR thermometer due any time now courtesy of Bicknellski.
The heat sink will be attached via 4 bolt holes located between each set of 4 ASICs, so they should have good mechanical contact. I expect to test with springs and with just tight bolts to compare. I'll be testing various pad methods - like blue silicone heat pads, copper pads, and just thermal tape/goop. Once I have some hard measured data I'll report and people can decide what they like best.

My design turns out to be very much like the Avalon. Almost exactly, (I think my exposed pads are bigger), without even seeing under the board before last night, but then there is only limited choices given the constraints. It makes little sense to put 16 small heat sinks under the board as the air gap between pads can only lessen the overall dissipation of heat.

[BkkCoins]

Sorry I've been out of the loop for ~2 weeks.  Is this changing the clock to the ASIC for OC purposes? cost savings?

I thought perhaps the ASIC has a PLL inside to allow you to adjust clock frequency in software

It does have a PLL. We're discussing the main input clock to the ASIC which the PLL will use. The hash clock is set by config data shifted in with work start data, and can be adjusted over quite a wide range.

[OtaconEmmerich]

Just ordered first batch of 20 - K1 test boards. Yay.

Woo! Progress! Thanks for all of your work Bkk.

[ecliptic]

The avalon schematics are very interesting.  Just looking at the PDFs now, especially for the power for the chips.

so right now you have 2x 16A 1.2V synch buck regulators [IR3895] , 8 chips per regulator.  dead on with the 2A per chip maximum spec from Avalon. [They should be the only chips using that 1.2V]  Can program the switching freq from 0.3 - 1.5 MHz

Avalon has on hash_unit_power.pdf -- it is a 2 page schematic in altium but they only exported page 2, not page 1.  really common mistake actually.  I will have to look at the avalon files to see what's on page 1..

I see there

* VCC1V2 is the 1.2V node.  this is used on all the schematics, so its "created" on hash_unit_power.pdf/schdoc but it's the exact same when you see it on hash_chip.pdf/schdoc
* I see the 10 chips on hash_unit.pdf.  not sure what is on page 2 of that though
* The FPGA on Avalon's design also uses the 1.2V plane, and they use the same VCC1V2.  Avalon also has a 1.2V LDO connected to this same net! (RT9166A_CXL) - RT9166A looks to be a 0.58V dropout 0.6A LDO.  Assuming it's the fixed 1.2V version.  I'm somewhat suprised they tied both regulators to the same plane.  But i dont think klondike board uses anything else on 1.2V so would not need this.  but it is a delta between the two designs.
* They use the tps40193 , 1.2V 20A buck synch buck regulator, dead on with the 2A per chip max spec.  0.3MHz switching frequency

I think as long as the decoupling capacitance on the chips and the regulators is the same or better it should be fine.  1.2V plane power caps aren't a place to skimp for cost  

If you think about most high power/current devices, you'll see a shitload of very tiny caps on the bottom of the PCB under the chip, and multi-phase buck converters to provide huge current, both spike current and average currrent.  Of course these chips aren't as bad, but if you think about it this is still up to 16x(2*1.2)=38.4W used by the chips.  it looks like Avalon has 1 0.47uF cap for every ~2 1.2V pins on the chip, so clearly they could have added more if needed but did not have to. 


But I believe this is one of the problems BFL ran into with their PCBs.  that core 1.2V can be a real PITA.  if the current draw is too high and voltage drops or there is too much instability on it the chips may malfunction.  You can declock it, lower current draw, and it works, but this kills performance.

This may very well limit OC'ing, far before heating or power would.

If by any chance you have an oscilloscope it would be very interesting to see what that 1.2V plane looks like when testing, and how it changes when you clock up/down.
---

also, any chance you've heard when sample chips might be shipping?

[BkkCoins]

* The FPGA on Avalon's design also uses the 1.2V plane, and they use the same VCC1V2.  Avalon also has a 1.2V LDO connected to this same net! (RT9166A_CXL) - RT9166A looks to be a 0.58V dropout 0.6A LDO.  Assuming it's the fixed 1.2V version.  I'm somewhat suprised they tied both regulators to the same plane.  But i dont think klondike board uses anything else on 1.2V so would not need this.  but it is a delta between the two designs.

That LDO is tied only to the PLL power AVDD. I have the same but used a pin-compatible alternate for that part since they are hard to get. That part has a very high transient response, somewhat higher than my choice, so that is a possible variance. I provide one for each bank of 8 chips.

But I believe this is one of the problems BFL ran into with their PCBs.  that core 1.2V can be a real PITA.  if the current draw is too high and voltage drops or there is too much instability on it the chips may malfunction.  You can declock it, lower current draw, and it works, but this kills performance.

This may very well limit OC'ing, far before heating or power would.

I've tried to keep the 1.2V supply as wide and short as possible from each buck to chips. This is why the regs are in board middle not edge. Hopefully that will improve 1.2V stability and if lucky improve clockibility.

If by any chance you have an oscilloscope it would be very interesting to see what that 1.2V plane looks like when testing, and how it changes when you clock up/down.

I used to have two but sold them when I moved here. I won't have one now unless I can find a local to mooch off. I could find one quite cheap on ebay but the shipping is killer on the weight of them. I thought about getting a DSO-230 Nano 4 ch. unit but I'm not certain it's actually very good. I'd need to research some reviews first. It's around $170 on several sites. I have a small logic analyser for data signals, but it would be nice to scope the power for a picture of noise.

also, any chance you've heard when sample chips might be shipping?

Word from zefir was they'd ship end of May, but arrive for him mid-June and re-ship to arrive at devs around end-June. So not really any time soon. I'll have lots of time to work on firmware. Will be mounting 4-5 2W resistors on ASIC pad at some point to simulate power use/heat.

[daemondazz]

The FPGA in Avalon's design is on a different PCB, I think there's 4 different PCBs in their design: hash unit, hash backplane, power distribution and control (which is where the FPGA is).

[erk]

Cheap Oscilloscope

http://www.ebay.com/itm/UTD2102CEL-100MHZ-Digital-Storage-Oscilloscope-7-LCD-1G-Sa-s-Sampling-Rate-/290672019764

A friend just bought one loves it.

[BkkCoins]

Cheap Oscilloscope

http://www.ebay.com/itm/UTD2102CEL-100MHZ-Digital-Storage-Oscilloscope-7-LCD-1G-Sa-s-Sampling-Rate-/290672019764

A friend just bought one loves it.

I've had multi-meters made by this company and they worked fine. They seem to stock UNI-T at some local shops in Bkk so I'll check that out when I'm there in a couple days. They aren't very competitive here so I suspect the price is better on eBay but the shipping may offset that.

First line in the specs:
"The volume exquisite and it is convenient for carrying;" ha ha, so typical.

[LaserHorse]

Rigol DS1052E 50MHz seems to be the go-to scope for cost-conscious EEs
There was even a simple hack that turned it into a 100MHz version, likely blocked now.
~$350 8lbs - not too heavy, no?

[Noitev]

Cheap Oscilloscope

http://www.ebay.com/itm/UTD2102CEL-100MHZ-Digital-Storage-Oscilloscope-7-LCD-1G-Sa-s-Sampling-Rate-/290672019764

A friend just bought one loves it.

I've had multi-meters made by this company and they worked fine. They seem to stock UNI-T at some local shops in Bkk so I'll check that out when I'm there in a couple days. They aren't very competitive here so I suspect the price is better on eBay but the shipping may offset that.

First line in the specs:
"The volume exquisite and it is convenient for carrying;" ha ha, so typical.

for the oscilloscope user on the go!