BFL ASIC mining board project

[MrTeal]

Alright, I am back in town and back to work.

Good news today, I managed to salvage two of the boards that were defective from the PCB house. Thank god too, because I really didn't want to lose the one with 8 ASICs on it. It's definitely a kludge, one board has Vdd shorted to ground but a good MCU, the other has problems with the JTAG interface that prevent me from programming but a good VRM and set of ASICs.

There were a couple issues, chiefly that the new build of the firmware meant for the 8 chip boards is having some issues and wasn't hashing so I had to hack together some code on the old 4 chip test board to get it to work. It behaves really strangely too, cgminer reports twice the hashrate that it's actually putting out. If I had to hazard a guess I'd say it has something to do with having ASICs 0-3 on SPI1 and 4-7 on SPI2, and my hacked code just has it looking for 8 on each bus. Bitminter reports it nicely at 27GH/s though.

The other issue is that the early code doesn't do a good job up keeping the ASICs feed with work. You can clearly see the dips in output current, which is quite different from the straight line you get when all 8 engines are loaded up at the same time and hashing test cases during startup. Once the issues with the new code base are resolved, that should get resolved. It will be interesting to see what kind of hashrates are achievable with grade A chips and some more optimized code.

The other problem is that the ASICs actually get quite hot, even with the closed loop water cooler. The only TIM pads I had laying around was a roll of 3M 5590H, and while it's decent stuff for most applications the 0.5mm thick 3.0W/mK really isn't effective enough for the kind of heat load these chips put out. I'll have to order some 0.5mm 12-17W/mK material, and I might be adventurous and try it with just grease.

[shmadz]

<snip>
The other problem is that the ASICs actually get quite hot, even with the closed loop water cooler. The only TIM pads I had laying around was a roll of 3M 5590H, and while it's decent stuff for most applications the 0.5mm thick 3.0W/mK really isn't effective enough for the kind of heat load these chips put out. I'll have to order some 0.5mm 12-17W/mK material, and I might be adventurous and try it with just grease.

I cut the other stuff even though it's awesome. I'm curious to see how level the chips are, if you're using paste on an 8 chip design there's a pretty good chance at least one of those chips is not going to get adequate cooling.

I was thinking that regular paste might work with a 4 chip design. I'm somewhat disappointed that a 4 chip board hasn't been explored/tested a little more seriously.

OTOH, if you can get optimum results from an 8 chip board, then that is -of course- better..  

That being said, I have great confidence in this project, pending the actual delivery of chips. Thanks for all your hard work.

[PuertoLibre]

Alright, I am back in town and back to work.

Good news today, I managed to salvage two of the boards that were defective from the PCB house. Thank god too, because I really didn't want to lose the one with 8 ASICs on it. It's definitely a kludge, one board has Vdd shorted to ground but a good MCU, the other has problems with the JTAG interface that prevent me from programming but a good VRM and set of ASICs.

There were a couple issues, chiefly that the new build of the firmware meant for the 8 chip boards is having some issues and wasn't hashing so I had to hack together some code on the old 4 chip test board to get it to work. It behaves really strangely too, cgminer reports twice the hashrate that it's actually putting out. If I had to hazard a guess I'd say it has something to do with having ASICs 0-3 on SPI1 and 4-7 on SPI2, and my hacked code just has it looking for 8 on each bus. Bitminter reports it nicely at 27GH/s though.

The other issue is that the early code doesn't do a good job up keeping the ASICs feed with work. You can clearly see the dips in output current, which is quite different from the straight line you get when all 8 engines are loaded up at the same time and hashing test cases during startup. Once the issues with the new code base are resolved, that should get resolved. It will be interesting to see what kind of hashrates are achievable with grade A chips and some more optimized code.

The other problem is that the ASICs actually get quite hot, even with the closed loop water cooler. The only TIM pads I had laying around was a roll of 3M 5590H, and while it's decent stuff for most applications the 0.5mm thick 3.0W/mK really isn't effective enough for the kind of heat load these chips put out. I'll have to order some 0.5mm 12-17W/mK material, and I might be adventurous and try it with just grease.

http://www.frozencpu.com/products/17499/thr-181/Fujipoly_Ultra_Extreme_System_Builder_Thermal_Pad_-_60_x_50_x_05_-_Thermal_Conductivity_170_WmK.html

[erk]

The other problem is that the ASICs actually get quite hot, even with the closed loop water cooler. The only TIM pads I had laying around was a roll of 3M 5590H, and while it's decent stuff for most applications the 0.5mm thick 3.0W/mK really isn't effective enough for the kind of heat load these chips put out. I'll have to order some 0.5mm 12-17W/mK material, and I might be adventurous and try it with just grease.

Will the grease work if you put enough to fill the gaps effectively? I was under the impression that grease has to be as thin as possible for best effect, and we know that 8 chips wont be perfectly level.

[MrTeal]

The other problem is that the ASICs actually get quite hot, even with the closed loop water cooler. The only TIM pads I had laying around was a roll of 3M 5590H, and while it's decent stuff for most applications the 0.5mm thick 3.0W/mK really isn't effective enough for the kind of heat load these chips put out. I'll have to order some 0.5mm 12-17W/mK material, and I might be adventurous and try it with just grease.

Will the grease work if you put enough to fill the gaps effectively? I was under the impression that grease has to be as thin as possible for best effect, and we know that 8 chips wont be perfectly level.

Depends how much variation in height and angle there is, I suppose. A pad is a better conductor of heat than grease, but grease is (hopefully) much thinner. It's not quite straightforward since the properties (and thickness) of a pad will change depending on pressure, but for a basic estimate you can just use thermal conductivity / thickness. Whichever number is bigger wins.

I'm doing some more testing tonight and actually having a lot better success with cooling. The backplate that came with my Water 2.0 Extreme is basically useless, so I just wrapped a small LGA1156 heatsink in duct tape and mounted it to the bottom of the board to act as a proper back plate. I can get it a lot tighter without worrying about board flex now, so that helps a lot.

[Lucko]

Just to let everyone hire know good news:

BFL chips shipping on schedule!!

Just got this email from BFL:

Dear Customer,

This is notification that it is seven days or less until your chip order is scheduled to ship. To see your remaining amount due, please log in to your account dashboard.

If you purchased via bankwire, please issue the remaining amount due to our bank account at:

Bank name: Commerce Bank
Bank address: 8901 State Line Road, Kansas City, Missouri, United States of America
Swift: CBKCUS44XXX
ABA / Routing: 101000019
Account Name: BF Labs Inc.
Account Number: 000049278
Account Address: 10770 El Monte, Leawood, KS, United States of America

Please add your order number to the memo of your wire transfer.

Once we have confirmed receipt of the wire transfer, we will add your order into the queue for processing.

If you purchased using bitcoin, you will be issued a new BitPay invoice about seven days before your order is supposed to ship which you will be required to pay within 15 minutes after opening. Please look for it in your e-mail inbox or spam folder.

If payment has not been completed, the order will not be shipped to you until it has been confirmed as paid.

Thank you for choosing Butterfly Labs' 65nm ASIC.

Best wishes,
The BFL Team

[Sitarow]

Just to let everyone hire know good news:

BFL chips shipping on schedule!!

Just got this email from BFL:

Dear Customer,

This is notification that it is seven days or less until your chip order is scheduled to ship. To see your remaining amount due, please log in to your account dashboard.

If you purchased via bankwire, please issue the remaining amount due to our bank account at:

Bank name: Commerce Bank
Bank address: 8901 State Line Road, Kansas City, Missouri, United States of America
Swift: CBKCUS44XXX
ABA / Routing: 101000019
Account Name: BF Labs Inc.
Account Number: 000049278
Account Address: 10770 El Monte, Leawood, KS, United States of America

Please add your order number to the memo of your wire transfer.

Once we have confirmed receipt of the wire transfer, we will add your order into the queue for processing.

If you purchased using bitcoin, you will be issued a new BitPay invoice about seven days before your order is supposed to ship which you will be required to pay within 15 minutes after opening. Please look for it in your e-mail inbox or spam folder.

If payment has not been completed, the order will not be shipped to you until it has been confirmed as paid.

Thank you for choosing Butterfly Labs' 65nm ASIC.

Best wishes,
The BFL Team

Congrats! nice to see this project get off the ground.

[Lucko]

This in not my order but I would just like anyone to know that impossible looks to be possible. BFL on time So you know that you are not wait for noting (like in Avalon case) and you will need MrTeal or any other(my ) board...

[PuertoLibre]

This in not my order but I would just like anyone to know that impossible looks to be possible. BFL on time So you know that you are not wait for noting (like in Avalon case) and you will need MrTeal or any other(my ) board...

Uh, you do realize that customer are getting chips that "do" about 2.4 GH/s correct?

If this were a good thing, then no one in the community would be chasing [a single] 28nm ASIC that do about 200Gh/s to 500Gh/s.

The customer should ?*rejoice*? about what again? Getting obsolete hardware at a very high price??

Explain it to me...

[nexus99]

This in not my order but I would just like anyone to know that impossible looks to be possible. BFL on time So you know that you are not wait for noting (like in Avalon case) and you will need MrTeal or any other(my ) board...

Uh, you do realize that customer are getting chips that "do" about 2.4 GH/s correct?

If this were a good thing, then no one in the community would be chasing [a single] 28nm ASIC that do about 200Gh/s to 500Gh/s.

The customer should ?*rejoice*? about what again? Getting obsolete hardware at a very high price??

Explain it to me...

Oh sweet Jesus. There is no thread that is Troll free.

[erk]

This in not my order but I would just like anyone to know that impossible looks to be possible. BFL on time So you know that you are not wait for noting (like in Avalon case) and you will need MrTeal or any other(my ) board...

Uh, you do realize that customer are getting chips that "do" about 2.4 GH/s correct?

If this were a good thing, then no one in the community would be chasing [a single] 28nm ASIC that do about 200Gh/s to 500Gh/s.

The customer should ?*rejoice*? about what again? Getting obsolete hardware at a very high price??

Explain it to me...

Oh sweet Jesus. There is no thread that is Troll free.

PuertoLibre should be banned from the forums, if I were an admin he would be for trolling, he is obvious spreading FUD, on one hand he is critical about a chips that has shipped 100/s of TH/s worth, on the other hand he praises a chip that has shipped none.

[mmmerlin]

PuertoLibre should be banned from the forums, if I were an admin he would be for trolling, he is obvious spreading FUD, on one hand he is critical about a chips that has shipped 100/s of TH/s worth, on the other hand he praises a chip that has shipped none.

I would just like to point out that this still counts as feeding... He's obviously hungry, just starve him out and he'll find a different thread for snacking...

EDIT: God it's hard though, I'm already struggling to take my own advice!

[PuertoLibre]

This in not my order but I would just like anyone to know that impossible looks to be possible. BFL on time So you know that you are not wait for noting (like in Avalon case) and you will need MrTeal or any other(my ) board...

Uh, you do realize that customer are getting chips that "do" about 2.4 GH/s correct?

If this were a good thing, then no one in the community would be chasing [a single] 28nm ASIC that do about 200Gh/s to 500Gh/s.

The customer should ?*rejoice*? about what again? Getting obsolete hardware at a very high price??

Explain it to me...

Oh sweet Jesus. There is no thread that is Troll free.

PuertoLibre should be banned from the forums, if I were an admin he would be for trolling, he is obvious spreading FUD, on one hand he is critical about a chips that has shipped 100/s of TH/s worth, on the other hand he praises a chip that has shipped none.

Erk, until recently there were more than 10 people advocating you be removed from the forums. So I would not speak too loudly.

As for the 100Th/s, because those 100Th/s [multiplied by 9] have shipped, the chip itself has become obsolete. I would have said the same thing of an ASICMiner chip or an Avalon chip. Gen 1 is obsolete in the current platform designs. (Obsolete = Nearing or has passed the line of Not being Profitable to run)

Could you revamp the platform to scale these chips massively? Probably. But you would always run into limits of the chips cost vs the electricity cost vs the ROI.

Notice I am talking about substance. Not just a wild claim. You often used to cheerlead on no substance.

=========================

Edit: I am not trying to "start anything" I just read a stupid comment...and reacted to it. Apparently you didn't like that.

People were plenty angry that a 300Mh/s chip sold at ~0.08btc (Avalon) was delivered "too late" just weeks ago. The same kind of logic applies to a BFL chip that was just about equally priced on a per-Gh/s to dollar ratio.

I am only guilty of pointing out the obvious.

[mmmerlin]

This in not my order but I would just like anyone to know that impossible looks to be possible. BFL on time So you know that you are not wait for noting (like in Avalon case) and you will need MrTeal or any other(my ) board...

Uh, you do realize that customer are getting chips that "do" about 2.4 GH/s correct?

If this were a good thing, then no one in the community would be chasing [a single] 28nm ASIC that do about 200Gh/s to 500Gh/s.

The customer should ?*rejoice*? about what again? Getting obsolete hardware at a very high price??

Explain it to me...

Oh sweet Jesus. There is no thread that is Troll free.

PuertoLibre should be banned from the forums, if I were an admin he would be for trolling, he is obvious spreading FUD, on one hand he is critical about a chips that has shipped 100/s of TH/s worth, on the other hand he praises a chip that has shipped none.

Erk, until recently there were more than 10 people advocating you be removed from the forums. So I would not speak too loudly.

As for the 100th/s, because those 100th/s have shipped, the chip itself has become obsolete. I would have said the same thing of an ASICMiner chip or an Avalon chip. Gen 1 is obsolete in the current platform designs. (Obsolete = Nearing or has passed the line of Not being Profitable to run)

Could you revamp the platform to scale these chips massively? Probably. But you would always run into limits of the chips cost vs the electricity cost vs the ROI.

Notice I am talking about substance. Not just a wild claim. You often used to cheerlead on no substance.

It's not that what you're saying is wrong, it's that this is not the place for it.

Many people here are painfully aware of what a shitty situation they are in, but it is what it is, and it's not clear why you feel the need to rub it in the faces of those suffering the consequences of poor decisions they've made.

This is a thread about a board that is being designed to run chips that have already, at least in part, been paid for. Either contribute to this discussion in a useful way or don't contribute at all, but coming on here to rub people's faces in their losses and de-rail a useful thread about hardware R&D is not on.

[PuertoLibre]

It's not that what you're saying is wrong, it's that this is not the place for it.

Alright, then you could have said so.

I'll take it elsewhere, as long as people like erk aren't too dumbfounded by the obvious.

Many people here are painfully aware of what a shitty situation they are in, but it is what it is, and it's not clear why you feel the need to rub it in the faces of those suffering the consequences of poor decisions they've made.

I just reacted to a very dumb comment made by someone else.

This is a thread about a board that is being designed to run chips that have already, at least in part, been paid for. Either contribute to this discussion in a useful way or don't contribute at all, but coming on here to rub people's faces in their losses and de-rail a useful thread about hardware R&D is not on.

Thanks, I will take my point and place it elsewhere.

[mmmerlin]

It's not that what you're saying is wrong, it's that this is not the place for it.

Alright, then you could have said so.

I'll take it elsewhere, as long as people like erk aren't too dumbfounded by the obvious.

Many people here are painfully aware of what a shitty situation they are in, but it is what it is, and it's not clear why you feel the need to rub it in the faces of those suffering the consequences of poor decisions they've made.

I just reacted to a very dumb comment made by someone else.

This is a thread about a board that is being designed to run chips that have already, at least in part, been paid for. Either contribute to this discussion in a useful way or don't contribute at all, but coming on here to rub people's faces in their losses and de-rail a useful thread about hardware R&D is not on.

Thanks, I will take my point and place it elsewhere.

Fair enough, and it's appreciated! 

[nexus99]

I, for one, cant wait to see a full production board doing its thing! Its going to be SEXY!

[MrTeal]

I, for one, cant wait to see a full production board doing its thing! Its going to be SEXY!

I can't wait either. A small quantity of production boards (50) will be arriving here on Wednesday and will get a couple built up. There's a couple improvements in the production boards (fixing a couple ECOs, improved cooling for the VRMs, new GPU cooler mounting holes) in addition to the new (and better) form factor, but functionally they're basically the same as the existing test board so we're very confident that it will work out of the box.

In testing, things have been going really well. The DC/DC mosfets are a little hotter than I'd like, they're probably going to require airflow over them or heatsinks on the bottom side. We'll see once the production boards get here; there are layout improvements and a move from 1 oz to 2oz copper vs the prototype, so we might be able to avoid needing to heatsink them. With a couple small RAM heatsinks from an old VGA cooler I have on the underside, I'm seeing ~65C MOSFET temps, so cooling temp without deafening fan noise should be pretty trivial.

Other things are coming along. We've had a lot of peripheral code disabled as we debugged some job queue issues, so now we're adding that back in. The quality of sample chips is still hit or miss; I've had one board that's happy to hash away are 32.5GH/s on 113 engines at stock (1V) voltage and staying nice and cool, while another board had chips that simply wouldn't produce valid results from half their cores even at 200MHz.

I took a video of some hashing at 32.5GH/s @ 168W (AC at the wall, random OEM HEC PSU, including the 9W from my cooler) but I'll retake it with a tripod and post it up tomorrow. Oscar quality it isn't.

For now we have a couple pics of less aggressively clocked testing I ran overnight; my normal test board and the aforementioned bad chip board.

[MrTeal]

Ah, and here's a quick one of the outline of the boards that will appear on Wednesday.


Tomorrow I'll post up a quick video of the unit hashing.

[Flashman]

The DC/DC mosfets are a little hotter than I'd like, they're probably going to require airflow over them or heatsinks on the bottom side. We'll see once the production boards get here; there are layout improvements and a move from 1 oz to 2oz copper vs the prototype, so we might be able to avoid needing to heatsink them. With a couple small RAM heatsinks from an old VGA cooler I have on the underside, I'm seeing ~65C MOSFET temps, so cooling temp without deafening fan noise should be pretty trivial.

Not sure if you might wanna test the MOSFETs you're getting to make sure they actually meet the data sheet spec. Reading between the lines of BFLs (mis)adventures, it seemed like they had that issue. I've also heard rumors of counterfeit parts, meaning you think you got a big name 20A part and it's a remarked POS burning up at 5A.