[Archive] BFL trolling museum

[greyhawk]

why aren't they showing us the god damn blown chip then ....:/

They did - it was in the first set of pictures they released showcasing the PCB.

Pictures:  http://bitcoinmagazine.com/butterfly-labs-releases-more-asic-photos/

Someone please point it out for me, Those look like the same photos from awhile ago, I would assume that someone would've gone "LOOK!, ITS BROKEN!"

That is exactly what people did. Phinn has pinpointed it correctly.

[SgtSpike]

why aren't they showing us the god damn blown chip then ....:/

They did - it was in the first set of pictures they released showcasing the PCB.

Pictures:  http://bitcoinmagazine.com/butterfly-labs-releases-more-asic-photos/

Someone please point it out for me, Those look like the same photos from awhile ago, I would assume that someone would've gone "LOOK!, ITS BROKEN!"

They did say something along those lines - Josh even acknowledged that in his post.

(you can see the bubbled chip in one of the pictures, I think someone pointed it out.)

[Fiyasko]

OH it IS!!! bubbled!, I have some more faith in BFL!

[Syke]

I'm really getting tired of hearing pedestrian engineering mistakes being attributed to the difficulty of the bitcoin hashing algorithm as implemented in hardware.

Alright, fair enough.  One could certainly argue that the thermal problems should have been foreseen.  I am no expert, so I couldn't tell you - I can only take someone else's word for it one way or the other.

6.4w per chip should not be a problem. As usual, they're not telling the truth.

[SgtSpike]

I'm really getting tired of hearing pedestrian engineering mistakes being attributed to the difficulty of the bitcoin hashing algorithm as implemented in hardware.

Alright, fair enough.  One could certainly argue that the thermal problems should have been foreseen.  I am no expert, so I couldn't tell you - I can only take someone else's word for it one way or the other.

6.4w per chip should not be a problem. As usual, they're not telling the truth.

In a QFN package?  With 8 chips on the same small PCB?  Do you have proof of this?  Is there a comparable product out in the wild (8 chips on one small PCB using roughly the same wattage in QFN form)?

[Syke]

6.4w per chip should not be a problem. As usual, they're not telling the truth.

In a QFN package?  With 8 chips on the same small PCB?  Do you have proof of this?  Is there a comparable product out in the wild (8 chips on one small PCB using roughly the same wattage in QFN form)?

Sure:

http://www.psitechnologies.com/products/powerqfn5x6.php

7 watts in a 5x6 mm package.

[PuertoLibre]

More details on the QFN packaging and reasons for delays:

We made the decision to go with QFN in December. I can't really talk about our development process itself, but we have gone through extensive design and testing phases... at one point in early December we decided to look at a worst case scenario if the chips were in a really hot environment (you can see the bubbled chip in one of the pictures, I think someone pointed it out.). We paid a company out of California quite a bit of money to run a run of simulations under different scenarios on our boards, as well as if we made changes to various portions of the PCB if we could salvage the QFN package's thermal envelope. We were able to get the thermal loads down about 6C off the current mark, but we were still within single digits of the max temperatures of surrounding components once the heat started to migrate through the ground plane. If someone in a really hot area ran these things, the fan would be on full blast the entire time, and as dust and other detritus collected on the HSF the unit would start to overheat and throttle (or worst case, you'd get bubbled chips). The internal junction temp of our ASICs, if I recall is around 121C, however the MCU and a couple other components are around 100C or less if memory serves and we were butting up against that in some cases, in the 90's.

I've already touched on some of the roadblocks we've had. One of the more annoying ones was the diffraction issue ... for example, at 65nm if you try to make a square shape on a wafer, you can't just make a square shape on the mask, you'll end up with an ellipsis of some sort due to the wavelength of light. So you have to shape the mask to accommodate the wavelength so what ends up on the wafer is a square, though it looks very different on the mask. So you have to go through just about everything, making sure what you want is actually what ends up on the wafer... the delay this caused was not anticipated to the extent it delayed us and since this is a full custom, hand routed chip, basically it had to be gone over by hand from top to bottom.

Another delay we've had to endure is the fact that we have effectively tied the ASIC teams payment to the success of the chip. If the chip were to be a failure they don't get paid... so they have incentive to get it right but that has made them very cautious and slow to approve final masks (This is why we can refund all pre-orders we want and why we have the capital to do what we need to do without a failure putting us in bankruptcy).

Ultimately, it has all boiled down to the incredible complexity of the chip (I mean, look at that beast, it's all black in the shot it's so dense). If the chip were not so complex and so efficient there wouldn't be a heat issue, there wouldn't be the wariness of releasing the mask, etc... This is why I find it patently ridiculous that Tom kept claiming his 90nm sASIC or PnR chip would be 100w, it's ludicrous. Avalons claims are far more reasonable at 400w for their design and is why I haven't given them such a hard time. I think Avalon is going to run into some problems that we've run into, but I don't think they will be anything insurmountable, but I suspect it will delay them a bit while they try to figure out how to mount all the heat sinks or the giant heatsink they are going to need to keep the thing cool, and the board itself has to be massive. Tom was estimating 7 x 9" if I recall for his 16 chip 90nm process... the Avalon is 110nm with at least 80 chips I estimate... though I'm sure the chip footprint is much smaller, we're still talking about a bucket load of chips that all have to be cooled. If their package, and I think they are using QFN, is not letting enough heat out the top they are going to flood their thermal and ground planes with 300w+ of heat and cook everything in sight. We were fighting 60w of heat (granted, on a much smaller surface area) and it was a problem, I can't imagine trying to fight 300w of heat. For their sakes, I hope they have already considered these issues or it's going to be a nasty surprise the first time they turn a unit on and the chips start popping and letting the magic smoke out.

https://forums.butterflylabs.com/bfl-forum-miscellaneous/690-13-jan-2013-asic-update-discussion-thread-6.html

Wasn't the picture of the bubbled chip posted in late-ish October? Did you blow it up in October than then have the simulations done in December, and then decided to switch away from QFN?

It sounds like that's about how it went, yeah.  They got the QFN prototypes, figured out they couldn't cool them properly as they thought they could, did some "worst case scenario" self testing, bubbled a chip, decided to have an outside company do some similar testing in December, and then concluded they couldn't properly cool a QFN chip in a heated environment.

That 121 is way too close to the failure at 125.. Even Xilinx starts to shut its chips down at 85.Then there is the issue of exactly what was the ambient when the die was reading 121... if it was the normal test temp of 85 deg c. then fine there is room in the design
but if it was measured at 25 deg. c... then we are all fucked.. the product is going to last less than a year.

Originally, BFL_Josh said the ASIC devices were capable of running in ambient temperatures of 32F to 95F.

I take this to mean that at 95F the device will reach its thermal limit and pop. 95F is about 35C.
P.S. Avalons' device is tested to work up to 105F (ambient temperatures).

[ldrgn]

They don't want to be the company that releases marginally adequate products, and I can respect that decision.

You've made an excellent post here but I think the discussion needs to be carried on a bit farther.  Remember that BFL's selling shovels and that a customer's profit is greatly dependent on how early they can get mining.  BFL's profit is also greatly increased by shipping early and often with rapid iterations in the product.  There's a reason why you said in your post that you would have preferred a miner in November - it's because you and BFL both win more from it.  BFL's made a poor strategic choice here - they're in a market that calls for the "marginally adequate products" and they are creating something else entirely.

[Inaba]

Originally, BFL_Josh said the ASIC devices were capable of running in ambient temperatures of 32F to 95F.
I take this to mean that at 95F the device will reach its thermal limit and pop. 95F is about 35C.

Without throttling.  Finish the rest of the sentence.  A completely finished unit with programmed with proper MCU code would be very unlikely to destroy itself in a hot environment, it would just throttle constantly.

Also in that lovely 80pt red font you quoted you forgot to mention the fact that 121C is the maximum junction temperature.  The maximum temperature we saw on the simulation was 95C inside the ASIC chips themselves, so it's kind of irrelevant. 

P.S. Avalons' device is tested to work up to 105F (ambient temperatures).

That's a good trick, since they don't even have a device to test with. But don't let facts get in the way of your bullshit, please continue.

[PuertoLibre]

why aren't they showing us the god damn blown chip then ....:/

They did - it was in the first set of pictures they released showcasing the PCB.

Pictures:  http://bitcoinmagazine.com/butterfly-labs-releases-more-asic-photos/

Someone please point it out for me, Those look like the same photos from awhile ago, I would assume that someone would've gone "LOOK!, ITS BROKEN!"

That is exactly what people did. Phinn has pinpointed it correctly.

Uh, if I recall correctly, Josh denied the chips were blown.

People thought the chips had been photoshopped at that time. Some thought (oddly) that the chips were actually black tape. Others speculated it was photoshopped to hide the chip writing. etc.

Josh (if I recall correctly) denied they were blown and then called people <insert names here>. Which then led to him and other BFL reps releasing extra images of the PCB with different chip configurations.

Ultimately, Josh admitted (strangely enough at the time) that they didn't have any prototypes that actually worked as intended. I believe this was caused by BitcoinINV in his pursuit of BFL through an Attorney General.

(This was the guy everyone hated who posted up two different images with blue paper. The first one was a complaint and the next was a response from BFL to the Attorney General showing their business address, a short summary of their current development status and the BFL address along with Jody's real name and title.)

[PuertoLibre]

OH it IS!!! bubbled!, I have some more faith in BFL!

Uh...you guys have some short term memory!

[poon-TANG]

why aren't they showing us the god damn blown chip then ....:/

They did - it was in the first set of pictures they released showcasing the PCB.

Pictures:  http://bitcoinmagazine.com/butterfly-labs-releases-more-asic-photos/

Someone please point it out for me, Those look like the same photos from awhile ago, I would assume that someone would've gone "LOOK!, ITS BROKEN!"

That is exactly what people did. Phinn has pinpointed it correctly.

Uh, if I recall correctly, Josh denied the chips were blown.

People thought the chips had been photoshopped at that time. Some thought (oddly) that the chips were actually black tape. Others speculated it was photoshopped to hide the chip writing. etc.

Josh (if I recall correctly) denied they were blown and then called people <insert names here>. Which then led to him and other BFL reps releasing extra images of the PCB with different chip configurations.

Ultimately, Josh admitted (strangely enough at the time) that they didn't have any prototypes that actually worked as intended. I believe this was caused by BitcoinINV in his pursuit of BFL through an Attorney General.

(This was the guy everyone hated who posted up two different images with blue paper. The first one was a complaint and the next was a response from BFL to the Attorney General showing their business address, a short summary of their current development status and the BFL address along with Jody's real name and title.)

Well I'm glad your on the case nancy drew......

[SgtSpike]

6.4w per chip should not be a problem. As usual, they're not telling the truth.

In a QFN package?  With 8 chips on the same small PCB?  Do you have proof of this?  Is there a comparable product out in the wild (8 chips on one small PCB using roughly the same wattage in QFN form)?

Sure:

http://www.psitechnologies.com/products/powerqfn5x6.php

7 watts in a 5x6 mm package.

Who uses 8 of these chips on the same 3.5" x 3.5" PCB?

They don't want to be the company that releases marginally adequate products, and I can respect that decision.

You've made an excellent post here but I think the discussion needs to be carried on a bit farther.  Remember that BFL's selling shovels and that a customer's profit is greatly dependent on how early they can get mining.  BFL's profit is also greatly increased by shipping early and often with rapid iterations in the product.  There's a reason why you said in your post that you would have preferred a miner in November - it's because you and BFL both win more from it.  BFL's made a poor strategic choice here - they're in a market that calls for the "marginally adequate products" and they are creating something else entirely.

Without knowing all the details, it's tough to make a certain determination, as such a move might have caused BFL undue expense.  But you could be right...!

[ldrgn]

Without knowing all the details, it's tough to make a certain determination, as such a move might have caused BFL undue expense.  But you could be right...!

By "the details" do you mean the state of the bubbled chips from earlier?  I wasn't suggesting ship those chips, I was suggesting a less ambitious and risky strategy from the beginning - small batch sizes, tried and true larger nm processes, etc.

[Inaba]

Once again, it's not the ASIC chips themselves that have the problem... it's the surrounding components.  The maximum temperature we saw (in simulations) in the QFN package was 95C with a max of 121C.  However, the MCU and a few other components had a ~100C maximum temp.  With 8 of these units, the heat simply builds up under the QFN plastic and sheds into the ground plane, transferring all that heat throughout the entire 92mmx92mm board.  So at least 20w or 30w of heat is being dissipated on a 92mm board that has very poor thermal dissipation properties.  

BFL's made a poor strategic choice here - they're in a market that calls for the "marginally adequate products" and they are creating something else entirely.

Possibly... but I think it's more likely that we'd be right back here in several months with people calling us all sorts of names for forcing an upgrade cycle on them again and calling us greedy assholes for forcing people to upgrade so we can grab more cash.  Instead, we chose to provide a product that will be viable for a long time, far longer than any other product.  You'll still be mining with our first gen product well into the third or fourth gen Avalon, if they ever make it that far.

[tvbcof]

...

Ultimately, Josh admitted (strangely enough at the time) that they didn't have any prototypes that actually worked as intended. I believe this was caused by BitcoinINV in his pursuit of BFL through an Attorney General.

(This was the guy everyone hated who posted up two different images with blue paper. The first one was a complaint and the next was a response from BFL to the Attorney General showing their business address, a short summary of their current development status and the BFL address along with Jody's real name and title.)

I remember BitcoinNV throwing BFL into the mix as a side-note when he was on the pirateat40 case.  I thought I was the only person who considered this action to be awesome.  I must have already been pretty suspicious by that time.

I guess maybe this is where it came out that BFL had taken over some defunct companies papers and the best conjecture seemed to be that perhaps it was in order to save like $100 on filing expenses?!?  I mean a company who is supposed to be developing ASICs and they were this tight on funds?  That didn't pass the smell test to me.  One of many things which make one go Hmmm.

[PuertoLibre]

Nancy drew, eh?

Well, if anyone has the time they can go and read Josh's answers to his Q&A and then go back into the forums past and reread/reskim all those positions BFL took as you travel back in time to October (2012). Then you can truly decide if there is a bit of revisionist history going on. (and to what extent with Hindsight)

Beyond that, you might even find contradictory statements.

First they (BFL) stated they were perfectly ready to ship (or so they thought) in October 2012. Yet the latest answers from Josh paint a very strange and surreal picture.  Admitting that they only received a small number of chips in October 2012 and finding problems. You should find it pretty strange that this would contradict their prior position that they had somehow been ready to start mass production with their chips.

Think about that.

As an example, I tell you I am ready to ship and fulfill all pre-orders in October 2012. Then in Mid-January I retell the same tale and say that we only received a small number of chips in October and found a problem. How do you explain that discrepancy? Weren't they supposed to be shipping in mass in October? Why didn't Josh say they received a large number of chips in October to fulfill their massive pre-orders? Is this a typo or a mistake?

Did you all know (by now) how long it takes to bake chips at the Fab? Do you all know that BFL has stated that they were using a technique which involved holding the chips half baked to test the quality of their design...and once sure...complete the other wafers or make adjustments. You do realize (I hope) that BFl was using a dense chip with many more layers (than their competitors) and that each layer takes about a day to deposit in the manufacturing process. (30+ Layers)

So ask yourself this question: If they only received a small batch of "test chips" that turned out to be overheating, then why did they make it seem like they were ready to fulfill pre-orders in October? Why make that promise if all you were expecting in the mail was a tiny batch to do tests on?

Hmm, I think someone needs to be careful about what accidentally slips out....Oops

[Syke]

So ask yourself this question: If they only received a small batch of "test chips" that turned out to be overheating, then why did they make it seem like they were ready to fulfill pre-orders in October?

Ask yourself this question. If they received a small batch, wouldn't they put a single chip on a PCB? How would a single chip overheat? Why wouldn't they demo such a working ASIC? Something just doesn't add up.

[Inaba]

As an example, I tell you I am ready to ship and fulfill all pre-orders in October 2012. Then in Mid-January I retell the same tale and say that we only received a small number of chips in October and found a problem. How do you explain that discrepancy? Weren't they supposed to be shipping in mass in October? Why didn't Josh say they received a large number of chips in October to fulfill their massive pre-orders? Is this a typo or a mistake?

Did you all know (by now) how long it takes to bake chips at the Fab? Do you all know that BFL has stated that they were using a technique which involved holding the chips half baked to test the quality of their design...and once sure...complete the other wafers or make adjustments. You do realize (I hope) that BFl was using a dense chip with many more layers (than their competitors) and that each layer takes about a day to deposit in the manufacturing process. (30+ Layers)

Holy shit PuertoLibre, you've somehow, incredibly, managed to out do yourself this time.  You'e managed to pack more fail into two paragraphs than I thought was even possible.  I'll let someone else explain why everything you said here is so completely idiotic that it may be impossible to know where to even begin correcting you (I would vote for 8.9 months before you were born and it involves a coat hanger and a bottle of JD).

[Cluster2k]

Good to see news of progress from BFL and hopefully a deliverable product some time in February.