BFL board project COINTAMINATION - EU facility - ORDERS OPENED for Chili

[bx8389]

Lucko and Mr. Teal,
...

What now is really needed next is that after cooling down the preheating restarts again so if my boards are interrupted from work e.g. network outage, they yould not resets. In fact I've also a problem with bfgminer. After a reset the /dev inode will change so bfgminer did not found them any more. So it will not restart the mining by themself :-( So preventing the device from reset will be very helpfull.
...

And it would be nice to see which fw is installed by the version number?

Cheers...

I more or less told all his to MrTeal and hi is working on it(I hope). He will not open his fw so we need to wait for him. But now that I have someone else saying that his board has the same problem only at different temperature he will probably be more sure it is not just isolated case...

I confirm this issue. I solved it installing cgwatcher and configure it to restart bfgminner every 6 hours - If this ocours only loose this minning time. Now i have a quite stable setup even using 3 boards with pre heating firmware.

regards

[ronin4bits]

I received my boards today, as well. Unfortunately, they both have same damage as GrapeApes... :-(

Posting from my phone so not sure how to include pics in forum -- will post when home later.

Package does not appear damaged or mis-handled...

EDIT: Not going to give anyone a hard time here, but this is 100% due to bad packing. Boards shipped face-to-face wrapped together in single piece of bubble wrap in a box that they *barely* fit in... No offense Lucko, but these were never going to go all the way around the world undamaged. I can fix these, but definitely suggest better packing for all future shipping. No doubt in my mind.

[Bogart]

GPL questions aside, opening the firmware would be a real nice thing to do for us users.  It would probably speed up debugging and save us from losing some precious hashing time.

Is there an advantage in keeping it closed?  I can't imagine someone taking it and making clone boards.  I don't think BFL even sells 65nm chips anymore...

[GrapeApe]

I received my boards today, as well. Unfortunately, they both have same damage as GrapeApes... :-(

Posting from my phone so not sure how to include pics in forum -- will post when home later.

Package does not appear damaged or mis-handled...

EDIT: Not going to give anyone a hard time here, but this is 100% due to bad packing. Boards shipped face-to-face wrapped together in single piece of bubble wrap in a box that they *barely* fit in... No offense Lucko, but these were never going to go all the way around the world undamaged. I can fix these, but definitely suggest better packing for all future shipping. No doubt in my mind.

Sorry to hear that, I just dropped mine off with ChipGeek to have him repair them. Lucky to live in the same city as him. I should have them back tomorrow.

[Lucko]

EDIT: Not going to give anyone a hard time here, but this is 100% due to bad packing. Boards shipped face-to-face wrapped together in single piece of bubble wrap in a box that they *barely* fit in... No offense Lucko, but these were never going to go all the way around the world undamaged. I can fix these, but definitely suggest better packing for all future shipping. No doubt in my mind.

I turned them face to face to protect pins that were beaned on Mudbankkeith(and some other) boards. It was a single piece of bubble wrap put it was 2 times the length and 3,5 times width of the boards. And there was additional around the sides and top of the box but not warped around. Other side had 3 additional and bottom had additional 7 layers from side boards beeping warped. It had enough of bubble warping around since it was minimal of 5 layers and up to 10 on same places... I agree that box could be bigger but I didn't had them. And you need a lot of force to get capacitors off the board... So I really don't see how could that happened without something strange in transport even with smaller box... Unless lead free solder get that fragile in long transports.

This is how 2 card pack look like. There is one layer of bubble wrap in the between them.

https://www.dropbox.com/s/nbvnvji49snzfcd/2014-01-28%2009.12.03.jpg

I'm pretty sure this is amount of bubble warping used in your package...

https://www.dropbox.com/s/i7t65gchvqxy1kb/2014-01-28%2009.13.20.jpg

One with the card on used to wrap them and another to put around it in a box so it will not move.

My theory is(since you say the is nothing visible) that I used to much and when it gets pressure from top and bottom it somehow move them one into another with enough force to do it... But I think I will adopted back pin bending position for US and probably the rest of the world just to be sure...

[Mudbankkeith]

EDIT: Not going to give anyone a hard time here, but this is 100% due to bad packing. Boards shipped face-to-face wrapped together in single piece of bubble wrap in a box that they *barely* fit in... No offense Lucko, but these were never going to go all the way around the world undamaged. I can fix these, but definitely suggest better packing for all future shipping. No doubt in my mind.

I turned them face to face to protect pins that were beaned on Mudbankkeith(and some other) boards. It was a single piece of bubble wrap put it was 2 times the length and 3,5 times width of the boards. And there was additional around the sides and top of the box but not warped around. Other side had 3 additional and bottom had additional 7 layers from side boards beeping warped. It had enough of bubble warping around since it was minimal of 5 layers and up to 10 on same places... I agree that box could be bigger but I didn't had them. And you need a lot of force to get capacitors off the board... So I really don't see how could that happened without something strange in transport even with smaller box... Unless lead free solder get that fragile in long transports.

This is how 2 card pack look like. There is one layer of bubble wrap in the between them.

https://www.dropbox.com/s/nbvnvji49snzfcd/2014-01-28%2009.12.03.jpg

I'm pretty sure this is amount of bubble warping used in your package...

https://www.dropbox.com/s/i7t65gchvqxy1kb/2014-01-28%2009.13.20.jpg

One with the card on used to wrap them and another to put around it in a box so it will not move.

My theory is(since you say the is nothing visible) that I used to much and when it gets pressure from top and bottom it somehow move them one into another with enough force to do it... But I think I will adopted back pin bending position for US and probably the rest of the world just to be sure...

Hi Lucko
If you post a package, the shippers will find a way to break it.(On long haul flight stopovers, the sights in transit bays are horrendous)
I "borrowed" the good lady's manicure set to straighten the bent pins. Boards ok, manicure set, -1.

[ronin4bits]

Hi Lucko - first, I want to say I am NOT upset, in any way.    These things happen and my feedback is to hopefully help other's still waiting to get their boards without damage.  I am happy I was able to be part of this gb - you have been excellent to work with and I would join you in more projects in future, without doubt.

Thanks to MrTeal, too, for licensing the design to our group. I'm sure the outcome hasn't been what you hoped, but you've been a big help getting things straight.

Back to boards and shipping. I believe that if there was something rigid or semi-rigid placed in between the boards before they are wrapped that this wouldn't happen. A small piece of cardboard, or even a small foam packing sheet would do.  I believe that your packing is a good plan, but it looks like it allowed the boards to shift laterally on top of each other during loading dock football shipping, so the end of one board pushed the caps laterally and off the solder.

I think that's the core problem. It looks like the caps were pushed away from the boards at the top:
EDIT: In reviewing my own pictures, it becomes obvious that the power block is what pushed these same caps off when the boards are rotated to each other...
https://s3.amazonaws.com/www.annagreentechnology.com/lucko-chili/20140127_133615.jpg

That said, the solder doesn't look very good to begin with either, as you can see from this pic. Neither the cap nor board in this pic shows signs of decent soldering, and there's absolutely nothing at all to clean off to redo it... I don't think this was the main issue, but I don't think it helped:
https://s3.amazonaws.com/www.annagreentechnology.com/lucko-chili/20140127_133622.jpg

Then there's the box - like GrapeApe indicated of his, no signs of "abuse". The boards were definitely a tight fit in it and it has clearly traveled a long way (my have even been examined by a friendly government safety representative at some point) - but it doesn't look like it was "abused":
https://s3.amazonaws.com/www.annagreentechnology.com/lucko-chili/20140127_133357.jpg
https://s3.amazonaws.com/www.annagreentechnology.com/lucko-chili/20140127_133406.jpg

Summary:
I hope this helps figure out how this has happened on 2 shipments. I'm not upset, although my boards are not hashing yet. I'll be attempting to fix these myself later today when I have time to warm up the soldering iron. I have the tools and I can do this work, but I am not an electrical engineer, so I can only hope it goes smooth and there is no other unseen damage.

If anyone has any suggestions or guidance on things I should look out for, or be careful of, or general tips on cleaning/re-soldering/testing things - I welcome any help.  Seems simple enough so I'm confident it won't be an issue.

Not as lucky as GrapeApe in having ChipGeek right up the road - I'm in the Charlottesville Virginia area, so if there's anyone around me that has some better thoughts, I'm all ears...

Thanks - sorry for the long-read.

[GrapeApe]

I got my boards back from ChipGeek and got the first one assembled. It does a very slow ramp up on the volts but as soon as it hits 1.1v it crashes. After a bit it resets itself and does it all over again. It doesn't force cgminer to crash though. So forgive me but what "trick" should I be doing to keep it hashing?

[Mudbankkeith]

I got my boards back from ChipGeek and got the first one assembled. It does a very slow ramp up on the volts but as soon as it hits 1.1v it crashes. After a bit it resets itself and does it all over again. It doesn't force cgminer to crash though. So forgive me but what "trick" should I be doing to keep it hashing?

I have 2 boards that need some "draft protection" over the power modules but still need additional underside cooling.

https://www.dropbox.com/s/a5t436j4y788r7p/2014-01-28%2021.46.42.jpg (any moving air over the modules and it crashes)

If its reaching 38Gh it will need the voltage limiting software

[bcollan]

Thanks Lucko , Mr Teal and all the others who have tirelessly worked on these boards.

I received both of mine yesterday in perfect condition, and they both start right up and hash around 35K and 56C with evo212 coolers. I am running both off of a single Seasonic G550 which isn't even breaking a sweat.

Overall, much better than I expected given all the problems. Great job!

[GrapeApe]

I got my boards back from ChipGeek and got the first one assembled. It does a very slow ramp up on the volts but as soon as it hits 1.1v it crashes. After a bit it resets itself and does it all over again. It doesn't force cgminer to crash though. So forgive me but what "trick" should I be doing to keep it hashing?

I have 2 boards that need some "draft protection" over the power modules but still need additional underside cooling.

https://www.dropbox.com/s/a5t436j4y788r7p/2014-01-28%2021.46.42.jpg (any moving air over the modules and it crashes)

If its reaching 38Gh it will need the voltage limiting software

"Draft Protection" nice call I moved it away from the fan and it is holding steady right now at 32GH/s at 1.06v. I got them both assembled and they are both holding at 1.06v the other hashes at 35GH/s.

Question for Lucko: Did you test each board and decide the optimal firmware (I know, a lot of work) or are you putting the same firmware on all boards?

I too am pleasantly surprised I expected a host of issues and really have had none (except for damage from shipping) other than moving one out of the way of the fan.

Thanks...

EDIT: Oh by the way the damn red led lights made my heart jump when I first powered it up. I thought uh oh that doesn't look right...

[bx8389]

I got my boards back from ChipGeek and got the first one assembled. It does a very slow ramp up on the volts but as soon as it hits 1.1v it crashes. After a bit it resets itself and does it all over again. It doesn't force cgminer to crash though. So forgive me but what "trick" should I be doing to keep it hashing?

I have 2 boards that need some "draft protection" over the power modules but still need additional underside cooling.

https://www.dropbox.com/s/a5t436j4y788r7p/2014-01-28%2021.46.42.jpg (any moving air over the modules and it crashes)

If its reaching 38Gh it will need the voltage limiting software

I have changed EVO 212 fan to heatsink side where you have the "box" to prevent aditional cooling on VRM components.
This helped to stability. Boards are now quite stable on 36-39Ghs range.
But still have an adicional fan blowing PCB underside

Regards

[Mudbankkeith]

I have found most of my boards need some underside cooling, when I have an airflow under the board, then the hash rate rises by 2 - 3 Gh

The asic temperature then also rises to about 70c but the other temperature (mosfets?) stays the same 44c to 47c.

One board crashes If I give it underside air, but apart from that it works well.


There is another problem showing up.  If I open the door or window and allow the ambient temperature to drop, then they all fall over.

[Lucko]

I got my boards back from ChipGeek and got the first one assembled. It does a very slow ramp up on the volts but as soon as it hits 1.1v it crashes. After a bit it resets itself and does it all over again. It doesn't force cgminer to crash though. So forgive me but what "trick" should I be doing to keep it hashing?

I have 2 boards that need some "draft protection" over the power modules but still need additional underside cooling.

https://www.dropbox.com/s/a5t436j4y788r7p/2014-01-28%2021.46.42.jpg (any moving air over the modules and it crashes)

If its reaching 38Gh it will need the voltage limiting software

"Draft Protection" nice call I moved it away from the fan and it is holding steady right now at 32GH/s at 1.06v. I got them both assembled and they are both holding at 1.06v the other hashes at 35GH/s.

Question for Lucko: Did you test each board and decide the optimal firmware (I know, a lot of work) or are you putting the same firmware on all boards?

I too am pleasantly surprised I expected a host of issues and really have had none (except for damage from shipping) other than moving one out of the way of the fan.

Thanks...

EDIT: Oh by the way the damn red led lights made my heart jump when I first powered it up. I thought uh oh that doesn't look right...

I think your problem is too low ambient temperature. I'm doing this at home now not in a workshop so I get them tested at lower temperature. I also asked if I could get slow rise 1.1V limiter firmware but didn't get it. I think that one would help your board. I did test boards wit a lot of different PSU(they also affect them how they work) and range of conditions. And put on what looks to be working best. Unfortunately Mrteal did not make the firmware that would work best with yours.

Yes read leds are in BOM. If you had a green ones it was a mistake done in first assembly if I'm not mistaken... You can read that in Chili tread...

[MrTeal]

I got my boards back from ChipGeek and got the first one assembled. It does a very slow ramp up on the volts but as soon as it hits 1.1v it crashes. After a bit it resets itself and does it all over again. It doesn't force cgminer to crash though. So forgive me but what "trick" should I be doing to keep it hashing?

I have 2 boards that need some "draft protection" over the power modules but still need additional underside cooling.

https://www.dropbox.com/s/a5t436j4y788r7p/2014-01-28%2021.46.42.jpg (any moving air over the modules and it crashes)

If its reaching 38Gh it will need the voltage limiting software

"Draft Protection" nice call I moved it away from the fan and it is holding steady right now at 32GH/s at 1.06v. I got them both assembled and they are both holding at 1.06v the other hashes at 35GH/s.

Question for Lucko: Did you test each board and decide the optimal firmware (I know, a lot of work) or are you putting the same firmware on all boards?

I too am pleasantly surprised I expected a host of issues and really have had none (except for damage from shipping) other than moving one out of the way of the fan.

Thanks...

EDIT: Oh by the way the damn red led lights made my heart jump when I first powered it up. I thought uh oh that doesn't look right...

I think your problem is too low ambient temperature. I'm doing this at home now not in a workshop so I get them tested at lower temperature. I also asked if I could get slow rise 1.1V limiter firmware but didn't get it. I think that one would help your board. I did test boards wit a lot of different PSU(they also affect them how they work) and range of conditions. And put on what looks to be working best. Unfortunately Mrteal did not make the firmware that would work best with yours.

Yes read leds are in BOM. If you had a green ones it was a mistake done in first assembly if I'm not mistaken... You can read that in Chili tread...

The LEDs were green throughout the Chili run. We were originally going to switch back to red so they would be brighter, but in the end we decided to just keep it continuous instead of having two different styles. I actually like the green ones in there being dim. It reduces the number of bright LEDs that are on all the time. I love the Antminers I have, but those LEDs are obnoxious.

For the limiting firmware, there is a 14g under development now that besides having several features and enhancements for everyone includes some that should specifically help your run of boards. The slow ramp to help heat the boards will be more efficient and will be active at the time instead of just on the startup. As well, the board will drop to idle voltage (0.9V) if there is no work and then slowly ramp back up. That should help prevent resets in the case where the board cools due to lack of work and then starts back up.
The maximum voltage will also be configurable through a z command, so a user can tweak that parameter on their own rather than having different firmwares for different voltage limits.

[ElitePork]

I have 4 chips on batch 2 and 4 chips on batch 3. any updates on the status?

[Lucko]

I have 4 chips on batch 2 and 4 chips on batch 3. any updates on the status?

One post up(MrTeal). The rest was on a mail...

EDIT: I'm currently "suffering" from sick daughter and girlfriend so I'm alone in taking care of them(not that I'm 100%) and a bit slower...

[ronin4bits]

I have found a local shop that has almost everything I needed to get my boards repaired from shipping.  Getting them interested in bitcoin now, too...

We have one of my boards fixed up and ready to go. Putting cooling on and powering it up tonight - everything looks ok this far.

One of the capacitors for the other one was damaged beyond repair though. The contacts were bent badly when it came off the board. It wouldn't make good contact that way so had to try to straighten it a little. Snapped it the rest of the way trying...

My shop doesn't have these exact caps at hand, but they gave me a part # for it to order a few.

Lucko, MrTeal or ChipGeek - would one of you be able to recommend a proper source/part # for getting these?

EDIT: I'll take any recommendation from this crowd - I trust a lot of you more than myself, but I figured I'd ask the gurus first...

[MrTeal]

I have found a local shop that has almost everything I needed to get my boards repaired from shipping.  Getting them interested in bitcoin now, too...

We have one of my boards fixed up and ready to go. Putting cooling on and powering it up tonight - everything looks ok this far.

One of the capacitors for the other one was damaged beyond repair though. The contacts were bent badly when it came off the board. It wouldn't make good contact that way so had to try to straighten it a little. Snapped it the rest of the way trying...

My shop doesn't have these exact caps at hand, but they gave me a part # for it to order a few.

Lucko, MrTeal or ChipGeek - would one of you be able to recommend a proper source/part # for getting these?

EDIT: I'll take any recommendation from this crowd - I trust a lot of you more than myself, but I figured I'd ask the gurus first...

This is the exact part. PCG1C681MCL1GS They aren't standard eletrolytics, they're aluminum polymer.
http://www.mouser.com/ProductDetail/Nichicon/PCG1C681MCL1GS/?qs=%2fha2pyFaduiCPnNwcYiq3wyCZT%2fFH59hIv%2ftueWLy54uR%252b1NvJOoUQ%3d%3d
http://www.digikey.com/product-detail/en/PCG1C681MCL1GS/493-3100-1-ND/1662570

[ronin4bits]

Damn! That was fast. 

Thank you very much MrTeal!