BFL board project COINTAMINATION - EU facility - ORDERS OPENED for Chili

[Mudbankkeith]

The hairdryer trick works wonders, for the low power crash.

as soon as the board completes initialization and starts to count, I switched the hairdryer from cold blow, to hot blow.  
 The fets temp. went up to 46c and NO CRASH  

Now its hashing the fets temp is steady 43c to 44c  

I just need to test out how this will work with 8 Chili on a rPi and hope it stays stable.  


My thanks to the brave man who tried it the first time with a soldering blower.

[bx8389]

The hairdryer trick works wonders, for the low power crash.

as soon as the board completes initialization and starts to count, I switched the hairdryer from cold blow, to hot blow.  
 The fets temp. went up to 46c and NO CRASH  

Now its hashing the fets temp is steady 43c to 44c  

I just need to test out how this will work with 8 Chili on a rPi and hope it stays stable.  


My thanks to the brave man who tried it the first time with a soldering blower.

I confirm that, yesterday and today I got finally all my sent boards working at some point with hairdryer trick: need some effort to start hashing but now I have a method to overcome reset problems until final solution is found. Many thank asjfdlksfd to share this with us.

My last mission now is investigates one board stop hashing after some hours: but for me is a minor annoyance...

Lucko/MrTeal: If board pre heating seems so reliable to solve this reset is possible simulate this behavior on firmware?
I wonder if after all everything is OK with board and main reason of problems are only some of Lucko's BFL chip batch are slightly out of specs. This happened on cointado board so is not completilly out of question.

My last restart ( even knowing 1 board will became "sick" during this night i am happy for first time
 since entered on this project ) :

bfgminer version 3.9.0 - Started: [2014-01-09 21:43:00] - [  0 days 00:22:39]
[M]anage devices [P]ool management Settings [D]isplay options  [H]elp [Q]uit
Connected to eu-stratum-*****.btcguild.com diff 64 with stratum as user ****
Block: ...4e2576ff #279578  Diff:1.42G (10.15Ph/s)  Started: [22:05:30]
ST:14  F:0  NB:7  AS:1  BW:[ 65/ 55 B/s]  E:511.17  I: 1.92mBTC/hr  BS:293k
4      69.0C | 146.9/135.5/128.1Gh/s | A:641 R:2+6(.34%) HW:2911/6.2%
-------------------------------------------------------------------------------
BFL 0: 69.0C | 35.61/29.63/28.57Gh/s | A:128 R:0+0(none) HW: 681/4.6%
BFL 1: 69.0C | 36.24/34.98/33.74Gh/s | A:175 R:2+0(1.1%) HW: 443/3.9%
BFL 2: 60.0C | 36.94/35.67/33.86Gh/s | A:170 R:0+0(none) HW: 611/5.4%
BFL 3: 69.0C | 34.78/35.25/31.94Gh/s | A:169 R:0+6(.11%) HW:1179/ 10%
-------------------------------------------------------------------------------
[2014-01-09 22:05:25] Accepted 018093b1 BFL 1  pool 0 Diff 170/64
[2014-01-09 22:05:25] Accepted 02310d9c BFL 0  pool 0 Diff 116/64
[2014-01-09 22:05:26] Accepted 0026ec8f BFL 0  pool 0 Diff 1.68k/64
[2014-01-09 22:05:26] Accepted 039f0d4f BFL 3  pool 0 Diff 70/64
[2014-01-09 22:05:27] Accepted 0187d6f1 BFL 0  pool 0 Diff 167/64

[MrTeal]

I'm messing around with the firmware on one of my (known good) boards. The quick hack I'm doing now just runs the self-test on startup for a couple minutes. If there is an issue with the PCB where there is interference at low temperatures, it might be difficult to correct with firmware. My board has a heatsink on the bottom side, but what I’m seeing is the board doesn’t actually get that hot in the self test at 0.85V. I’m finding it’s only drawing 51A and getting to about 25C in a 21C room. If I raise the voltage to 1V I get 85A draw and it stabilizes at 32C or so, which still isn’t really that warm. It’s not until I get up to 1.1V or so that I start pulling ~120A and the temperatures get up to 50C or so.
Basically, I think people are likely getting the boards a lot hotter with a hair dryer than I will be able to by running the board, if I have to run it at less than 1V to keep it stable. I'll give it a shot on my bad board tomorrow.

[bx8389]

I'm messing around with the firmware on one of my (known good) boards. The quick hack I'm doing now just runs the self-test on startup for a couple minutes. If there is an issue with the PCB where there is interference at low temperatures, it might be difficult to correct with firmware. My board has a heatsink on the bottom side, but what I’m seeing is the board doesn’t actually get that hot in the self test at 0.85V. I’m finding it’s only drawing 51A and getting to about 25C in a 21C room. If I raise the voltage to 1V I get 85A draw and it stabilizes at 32C or so, which still isn’t really that warm. It’s not until I get up to 1.1V or so that I start pulling ~120A and the temperatures get up to 50C or so.
Basically, I think people are likely getting the boards a lot hotter with a hair dryer than I will be able to by running the board, if I have to run it at less than 1V to keep it stable. I'll give it a shot on my bad board tomorrow.

Ok Understood the issue, thank you

[Mudbankkeith]

I confirm that, yesterday and today I got finally all my sent boards working at some point with hairdryer trick: need some effort to start hashing but now I have a method to overcome reset problems until final solution is found. Many thank asjfdlksfd to share this with us.

My last mission now is investigates one board stop hashing after some hours: but for me is a minor annoyance...

Lucko/MrTeal: If board pre heating seems so reliable to solve this reset is possible simulate this behavior on firmware?
I wonder if after all everything is OK with board and main reason of problems are only some of Lucko's BFL chip batch are slightly out of specs. This happened on cointado board so is not completilly out of question.

My last restart ( even knowing 1 board will became "sick" during this night i am happy for first time
 since entered on this project ) :

bfgminer version 3.9.0 - Started: [2014-01-09 21:43:00] - [  0 days 00:22:39]
[M]anage devices [P]ool management Settings [D]isplay options  [H]elp [Q]uit
Connected to eu-stratum-*****.btcguild.com diff 64 with stratum as user ****
Block: ...4e2576ff #279578  Diff:1.42G (10.15Ph/s)  Started: [22:05:30]
ST:14  F:0  NB:7  AS:1  BW:[ 65/ 55 B/s]  E:511.17  I: 1.92mBTC/hr  BS:293k
4      69.0C | 146.9/135.5/128.1Gh/s | A:641 R:2+6(.34%) HW:2911/6.2%
-------------------------------------------------------------------------------
BFL 0: 69.0C | 35.61/29.63/28.57Gh/s | A:128 R:0+0(none) HW: 681/4.6%
BFL 1: 69.0C | 36.24/34.98/33.74Gh/s | A:175 R:2+0(1.1%) HW: 443/3.9%
BFL 2: 60.0C | 36.94/35.67/33.86Gh/s | A:170 R:0+0(none) HW: 611/5.4%
BFL 3: 69.0C | 34.78/35.25/31.94Gh/s | A:169 R:0+6(.11%) HW:1179/ 10%
-------------------------------------------------------------------------------
[2014-01-09 22:05:25] Accepted 018093b1 BFL 1  pool 0 Diff 170/64
[2014-01-09 22:05:25] Accepted 02310d9c BFL 0  pool 0 Diff 116/64
[2014-01-09 22:05:26] Accepted 0026ec8f BFL 0  pool 0 Diff 1.68k/64
[2014-01-09 22:05:26] Accepted 039f0d4f BFL 3  pool 0 Diff 70/64
[2014-01-09 22:05:27] Accepted 0187d6f1 BFL 0  pool 0 Diff 167/64

Hi bx8389
possibly your BFL2 needs some extra cooling at the power side  60c is a bit low, 70c seems nice for the asics.  Try cooling the back of the board. below the power modules.

[bx8389]

I confirm that, yesterday and today I got finally all my sent boards working at some point with hairdryer trick: need some effort to start hashing but now I have a method to overcome reset problems until final solution is found. Many thank asjfdlksfd to share this with us.

My last mission now is investigates one board stop hashing after some hours: but for me is a minor annoyance...

Lucko/MrTeal: If board pre heating seems so reliable to solve this reset is possible simulate this behavior on firmware?
I wonder if after all everything is OK with board and main reason of problems are only some of Lucko's BFL chip batch are slightly out of specs. This happened on cointado board so is not completilly out of question.

My last restart ( even knowing 1 board will became "sick" during this night i am happy for first time
 since entered on this project ) :

bfgminer version 3.9.0 - Started: [2014-01-09 21:43:00] - [  0 days 00:22:39]
[M]anage devices [P]ool management Settings [D]isplay options  [H]elp [Q]uit
Connected to eu-stratum-*****.btcguild.com diff 64 with stratum as user ****
Block: ...4e2576ff #279578  Diff:1.42G (10.15Ph/s)  Started: [22:05:30]
ST:14  F:0  NB:7  AS:1  BW:[ 65/ 55 B/s]  E:511.17  I: 1.92mBTC/hr  BS:293k
4      69.0C | 146.9/135.5/128.1Gh/s | A:641 R:2+6(.34%) HW:2911/6.2%
-------------------------------------------------------------------------------
BFL 0: 69.0C | 35.61/29.63/28.57Gh/s | A:128 R:0+0(none) HW: 681/4.6%
BFL 1: 69.0C | 36.24/34.98/33.74Gh/s | A:175 R:2+0(1.1%) HW: 443/3.9%
BFL 2: 60.0C | 36.94/35.67/33.86Gh/s | A:170 R:0+0(none) HW: 611/5.4%
BFL 3: 69.0C | 34.78/35.25/31.94Gh/s | A:169 R:0+6(.11%) HW:1179/ 10%
-------------------------------------------------------------------------------
[2014-01-09 22:05:25] Accepted 018093b1 BFL 1  pool 0 Diff 170/64
[2014-01-09 22:05:25] Accepted 02310d9c BFL 0  pool 0 Diff 116/64
[2014-01-09 22:05:26] Accepted 0026ec8f BFL 0  pool 0 Diff 1.68k/64
[2014-01-09 22:05:26] Accepted 039f0d4f BFL 3  pool 0 Diff 70/64
[2014-01-09 22:05:27] Accepted 0187d6f1 BFL 0  pool 0 Diff 167/64

Hi bx8389
possibly your BFL2 needs some extra cooling at the power side  60c is a bit low, 70c seems nice for the asics.  Try cooling the back of the board. below the power modules.

hello,
yes you have reason, I have adicional cooling for boards but at "shoot" time this board was separate from others to take measurements and not yet cooled: this is the one needing hairdryer to start...
Thanks for warning

[Mudbankkeith]

I have finally got these two bad girls  reasonably stable at asic temp of 68c and power module at 44c/45c

Note the evo 212 coolers are lifted 15mm up. This has reduced the cold airflow over the power modules.

after I used the "Hairdryer Mod" the boards would restart, but the temp. at the power modules would slowly fall.(eventually crashing again)

The cardboard cover is (hopefully) holding the temp. in the power modules and the cold air from the evo is deflected upwards.

The secondary fans are controlling the airflow under the boards, if I move them around then the temp. does change.


https://www.dropbox.com/s/y6dc59y5zzeaub5/IMG_20140110_105218.jpg


EDIT  still hashing after 3 hours

EDIT 2 crashed after 5 hours

[one4many]

Lucko,

can we end this madness now ... A proper/stable solution would be nice! 

Thx

    one4many

[bx8389]

I have finally got these two bad girls  reasonably stable at asic temp of 68c and power module at 44c/45c

Note the evo 212 coolers are lifted 15mm up. This has reduced the cold airflow over the power modules.

after I used the "Hairdryer Mod" the boards would restart, but the temp. at the power modules would slowly fall.(eventually crashing again)

The cardboard cover is (hopefully) holding the temp. in the power modules and the cold air from the evo is deflected upwards.

The secondary fans are controlling the airflow under the boards, if I move them around then the temp. does change.


https://www.dropbox.com/s/y6dc59y5zzeaub5/IMG_20140110_105218.jpg


EDIT  still hashing after 3 hours

EDIT 2 crashed after 5 hours

Hello

I have also some crashs during last days from low voltage reset boards, at this moment ( not yet totally confirmed ) i suspect form internet connection: when it stops during few seconds (maybe or more than 30 seconds)  boards lower the hash and cool down enough to be unable to restart by its own. Normal boards aren't affected. Discovered this during a router restart.
I changed setup putting this board on not cooled area: hash will be lowered but expect more immunity from connection drop-outs

Regards

Edit: 4H+  running: I think 26º Ambient temperature helps too ...
Edit: Exceded 14H hashing:By far longest time without incidents with all received boards.
"the black sheep" - BFL 3: 55.0C | 35.14/35.20/31.91Gh/s | A: 4280 R:21+ 2(.49%) HW:26874/8.6%

Edit 3: 28H hashing without intervention: Decided maintain and use this setup ( one board without direct power converter cooling ). Total hashing at Pool now for 4 Lucko's chilli boards: 131 GHs

[Mudbankkeith]

I have finally got these two bad girls  reasonably stable at asic temp of 68c and power module at 44c/45c

Note the evo 212 coolers are lifted 15mm up. This has reduced the cold airflow over the power modules.

after I used the "Hairdryer Mod" the boards would restart, but the temp. at the power modules would slowly fall.(eventually crashing again)

The cardboard cover is (hopefully) holding the temp. in the power modules and the cold air from the evo is deflected upwards.

The secondary fans are controlling the airflow under the boards, if I move them around then the temp. does change.


https://www.dropbox.com/s/y6dc59y5zzeaub5/IMG_20140110_105218.jpg


EDIT  still hashing after 3 hours

EDIT 2 crashed after 5 hours

Hello

I have also some crashs during last days from low voltage reset boards, at this moment ( not yet totally confirmed ) i suspect form internet connection: when it stops during few seconds (maybe or more than 30 seconds)  boards lower the hash and cool down enough to be unable to restart by its own. Normal boards aren't affected. Discovered this during a router restart.
I changed setup putting this board on not cooled area: hash will be lowered but expect more immunity from connection drop-outs

Regards

For some reason these "low voltage reset" boards need a lot more heat ABOVE the power units, they still need the normal asic cooling and they do benefit from underside cooling(when the board is hashing)

It has to be something to do with the switching state in the power modules.(trying to move up to the next stage before all the temperature conditions are met)

That's why the "hairdryer mod" works.(It brings the temperature up in time for the switching to work)

The problem is that if I remove my "heatshield" after hashing starts, the temperature falls and the "crash happens again".

1. In a normal startup state, the board crashes after just a few seconds(never reaching the stable temperature of the asics or power module)
2. With the "hairdryer mod" the board gets to temperature and starts to hash, then the temperature starts to drop, it crashes after 3 or 4 minutes.
3. Adding the secondary cooling to the bottom of the board, the asic temperature continues to rise towards 70c and the hash speed rises, but the temperature in the power module continues to drop.
    It again crashes after about 3 or 4 minutes.
4. With the "heatshield" placed over the power module, the temperatures stay reasonably stable, but I have not managed to keep it hashing for more than about 5 hours.

They readily restart, with the above "conditioning"


I have 7 boards that need this treatment, so a solution will be desirable.

My 8 other boards are well behaved, apart from the rPi deciding to reset occasionally.

[Lucko]

Hello

I have also some crashs during last days from low voltage reset boards, at this moment ( not yet totally confirmed ) i suspect form internet connection: when it stops during few seconds (maybe or more than 30 seconds)  boards lower the hash and cool down enough to be unable to restart by its own. Normal boards aren't affected. Discovered this during a router restart.
I changed setup putting this board on not cooled area: hash will be lowered but expect more immunity from connection drop-outs

Regards

Edit: 4H+  running: I think 26º Ambient temperature helps too ...

Yes I see the same. If I have connection problems for more then seconds I have resets... But I have them in about 30C ambient temperature and they restart themselves... They just heat up my workshop that much and I don't have a window open over the nonworking hours...

EDIT: I also figure out why after first 20 boards I didn't see any big problem. Ambient temperature was at that time at about 30C... Since I put to work first 10 boards that I have assemble.

[Mudbankkeith]

Hi Lucko

I have found that the troublesome boards completely refuse to re start above 30c ambient. (in fact all my boards refuse to re start at this temperature)
When they are below 30c ambient, the troublesome boards attempt to re start and I have about 30 seconds to start up the "hairdryer" or the crash sequence starts.

The regular boards re start automatically when the cooling sequence completes.

[Lucko]

Hi Lucko

I have found that the troublesome boards completely refuse to re start above 30c ambient. (in fact all my boards refuse to re start at this temperature)
When they are below 30c ambient, the troublesome boards attempt to re start and I have about 30 seconds to start up the "hairdryer" or the crash sequence starts.

The regular boards re start automatically when the cooling sequence completes.

Yes I notice that... It takes really long time at that temperature... Or sometimes they don't start at all... Startup has temperature limit over which will not start selftests... I don't know how hi it is but I would say it is about 35C...

EDIT: Hairdryer trick got all but one of my boards running(that one probably has a short on one of the ASICs) so thanks for figuring it out...

[GrapeApe]

Hey Lucko I got your email and go ahead and make my boards. Let me know if I need to respond via email but go ahead.

[ronin4bits]

Same here, Lucko - got your email. Please go ahead with my boards, as well.  Thanks.

[Min€r]

Same here Lucko - make my boards too! Thanks for your mail & work - I am sure we will finally get a stable solution for everybody.

[Mudbankkeith]

Hi Lucko

I have found that the troublesome boards completely refuse to re start above 30c ambient. (in fact all my boards refuse to re start at this temperature)
When they are below 30c ambient, the troublesome boards attempt to re start and I have about 30 seconds to start up the "hairdryer" or the crash sequence starts.

The regular boards re start automatically when the cooling sequence completes.

Yes I notice that... It takes really long time at that temperature... Or sometimes they don't start at all... Startup has temperature limit over which will not start selftests... I don't know how hi it is but I would say it is about 35C...

EDIT: Hairdryer trick got all but one of my boards running(that one probably has a short on one of the ASICs) so thanks for figuring it out...

Just asking now if this resetting problem is influenced by the type of cooler being used.

My coolers are all evo212,      8 boards act normally, 7 boards need hairdryer.

Does anyone have any data on using alternative cooling?

[one4many]

Hi Lucko,

the 'hair dryer' solution is not acceptable. My boards are going to be in 2 (closed) cases with water cooling. It would be (and probably is) a nightmare to kick-off hashing with a hairdryer in the hope they never loose network connection (so the temp would drop because of missing workload) to keep them hashing.

Please work towards a stable solution with Mr. Teal, who took your/our money for a license for something which has an engineering/design problem and is a closed source firmware project where code fixes take days to get.

I personally start to regret having this project kicked off myself, wishing we would have fixed the initial Cointamination board (which probably has/had the same power problem with 'over-performing' BFL chips).

    one4many

[asjfdlksfd]

the 'hair dryer' solution is not acceptable.

I can confirm that. The method is acceptable so I will not lost all my money because there mining now, more or less.

Please work towards a stable solution with Mr. Teal, who took your/our money for a license for something which has an engineering/design problem and is a closed source firmware project where code fixes take days to get.

A reason why I like open source. Maybe we could have allready an alternative q&d solution by patching the fw by community self ;-)

I need a work around that the board will start up without any problem. If this need much more time to hash a full speed it is ok. Save restart is better than have every day a look on the boards. In a few month we can kick the boards in the trash bin. Up to this time I want to put the boards in server racks to bring them to server room which is much more cooled and power is cheaper than at home. I hope everybody understands it will looks stupid if I stand at my rack with a hairdryer, not to forget the drive-in ;-).

Mr Teal, it is an option to open the source for the firmware if you have problems with time to fix/extend it? I guess most of chips are allready sold and your should not lost so much money by selling new boards ;-)

Cheers...

[MrTeal]

the 'hair dryer' solution is not acceptable.

I can confirm that. The method is acceptable so I will not lost all my money because there mining now, more or less.

Please work towards a stable solution with Mr. Teal, who took your/our money for a license for something which has an engineering/design problem and is a closed source firmware project where code fixes take days to get.

A reason why I like open source. Maybe we could have allready an alternative q&d solution by patching the fw by community self ;-)

I need a work around that the board will start up without any problem. If this need much more time to hash a full speed it is ok. Save restart is better than have every day a look on the boards. In a few month we can kick the boards in the trash bin. Up to this time I want to put the boards in server racks to bring them to server room which is much more cooled and power is cheaper than at home. I hope everybody understands it will looks stupid if I stand at my rack with a hairdryer, not to forget the drive-in ;-).

Mr Teal, it is an option to open the source for the firmware if you have problems with time to fix/extend it? I guess most of chips are allready sold and your should not lost so much money by selling new boards ;-)

Cheers...

Probably not, though we've talked about it. I have given Lucko a version that slowly raises the maximum permitted voltage over the span on 5 minutes, which could help. It doesn't work on mine, though Lucko did indicate mine was the worst of the batch.