S19 repeated fan failures

[Oracle7star]

I am seeing repeated fans failures on several of my S19s.
Not receiving much insight from Bitmain support.
Looking for insight to common (or uncommon) causes of fan failure.

Log entry looks like:

2021-04-16 05:13:19 Error, fan lost, only find 3 (< 4)
2021-04-16 05:13:19 fan_id = 0, fan_speed = 5880
2021-04-16 05:13:19 fan_id = 1, fan_speed = 0
2021-04-16 05:13:19 fan_id = 2, fan_speed = 5880
2021-04-16 05:13:19 fan_id = 3, fan_speed = 6000

another example:
 
2021-09-18 02:50:03 Error, fan lost, only find 3 (< 4)
2021-09-18 02:50:03 fan_id = 0, fan_speed = 13080
2021-09-18 02:50:03 fan_id = 1, fan_speed = 5880
2021-09-18 02:50:03 fan_id = 2, fan_speed = 8520
2021-09-18 02:50:03 fan_id = 3, fan_speed = 5880

I am seeing zero rpm all the way to 25k rpm - which is impossible - on fan failures. 6500 rpm seems to be the max on healthy fans.

The environment is clean air with sub <=23c (73f), clean power. My servers are less than 6 months old.

Are the fans disposable?

Regards. 

[philipma1957]

Fans are replaceable.


https://shop.bitmain.com/product/detail?pid=000202102181332481369Jo8K54n063B

[Oracle7star]

Fans are replaceable.


https://shop.bitmain.com/product/detail?pid=000202102181332481369Jo8K54n063B

Yes they are replaceable. I have 4 month old servers and been replacing fans with increasing regularity.

My fan issues started with one fan failure after 3 months, then 3 fans in month 4 and now 8 in month 5.

Just wondering if this is normal for these servers? 

[Gabrics]

These FANs work a lot, warm places see 100% all the time.
So you can't really compare to case fans in desktop/server machines where they do rotate, but most of the time with much lower RPM.

We see some fans failing after a few months (probably manufacturing errors), but most work for a 1-2 years even under full load.

[kp98]

These FANs work a lot, warm places see 100% all the time.
So you can't really compare to case fans in desktop/server machines where they do rotate, but most of the time with much lower RPM.

We see some fans failing after a few months (probably manufacturing errors), but most work for a 1-2 years even under full load.

We replaced 1 dead fan, and then within 24 hours 2 more fans died. Do you have any idea what may be causing this? Thanks for the help

[HagssFIN]

Sounds like it could be a bad quality fan production batch or a bad quality fan model Bitmain is using for the S19.

[kp98]

Sounds like it could be a bad quality fan production batch or a bad quality fan model Bitmain is using for the S19.

ok thank you

[mikeywith]

Sounds like it could be a bad quality fan production batch or a bad quality fan model Bitmain is using for the S19.

That is a possibility. however, I have personally used some very cheap fans on my gears and they don't die anywhere close to the rate OP is talking about, this is likely a combination of bad quality and some other issues.

OP, what is the design of your farm? pressure plays a major role when it comes to the lifespan of these fans, I remember before fixing the pressure in the farm we were losing fans a lot more often than we do now, ever since we controlled the air pressure we seldom lose fans despite the fact that we run most of them at a constant speed of 90% where temps are usually a lot hotter than your temps.

[philipma1957]

Sounds like it could be a bad quality fan production batch or a bad quality fan model Bitmain is using for the S19.

That is a possibility. however, I have personally used some very cheap fans on my gears and they don't die anywhere close to the rate OP is talking about, this is likely a combination of bad quality and some other issues.

OP, what is the design of your farm? pressure plays a major role when it comes to the lifespan of these fans, I remember before fixing the pressure in the farm we were losing fans a lot more often than we do now, ever since we controlled the air pressure we seldom lose fans despite the fact that we run most of them at a constant speed of 90% where temps are usually a lot hotter than your temps.

to be clear

your farm should be

intake------- gear------- exhaust

the intake needs to be large enough to balance the exhaust


so 1 s19 has 4 fans then pull about 400 cfm they exhaust about 400 cfm

assuming they are all pointed in correct directions they are balanced in an open space

but

intake fans 1000 cfm >>>>> gear fans 400 cfm >>>>>>> exhaust fans 300 cfm  = bad for fans

intake fans 300 cfm >>>>>gear fans 400 cfm >>>>>>> exhaust fans 1000 cfm = bad for fans

intake  fans none just a small vent >>>> gear fans 400 cfm >>>>> exhaust fans 1000 cfm = bad for fans


intake fans  600 cfm >>>>>> gear fans 400 cfm >>>>> exhaust fans 600 cfm = winner balanced


do check to see all fans are pointed correctly

[kp98]

Sounds like it could be a bad quality fan production batch or a bad quality fan model Bitmain is using for the S19.

That is a possibility. however, I have personally used some very cheap fans on my gears and they don't die anywhere close to the rate OP is talking about, this is likely a combination of bad quality and some other issues.

OP, what is the design of your farm? pressure plays a major role when it comes to the lifespan of these fans, I remember before fixing the pressure in the farm we were losing fans a lot more often than we do now, ever since we controlled the air pressure we seldom lose fans despite the fact that we run most of them at a constant speed of 90% where temps are usually a lot hotter than your temps.

to be clear

your farm should be

intake------- gear------- exhaust

the intake needs to be large enough to balance the exhaust


so 1 s19 has 4 fans then pull about 400 cfm they exhaust about 400 cfm

assuming they are all pointed in correct directions they are balanced in an open space

but

intake fans 1000 cfm >>>>> gear fans 400 cfm >>>>>>> exhaust fans 300 cfm  = bad for fans

intake fans 300 cfm >>>>>gear fans 400 cfm >>>>>>> exhaust fans 1000 cfm = bad for fans

intake  fans none just a small vent >>>> gear fans 400 cfm >>>>> exhaust fans 1000 cfm = bad for fans


intake fans  600 cfm >>>>>> gear fans 400 cfm >>>>> exhaust fans 600 cfm = winner balanced


do check to see all fans are pointed correctly

Thanks, I think this may be the case. We haven't put a lot of thought into air pressure but I could certainly see that being the cause. If the problem is the control board, for instance, what would be a signal that that is the case? If you have any other information on why fans may die do let me know.

For instance, is it normal to see a miner's hash power to oscillate significantly - on our s19s the average hashrate is 95th/s but I've seen as low as 50 th/s and as high as 250 th/s. I wondered if this could be blowing out the fan somehow. We are just connected to antpool. Thanks again for the help btw

[HagssFIN]

Among of all the pools,
why the hell Antpool?

[kp98]

Among of all the pools,
why the hell Antpool?

Well, supposedly it has the lowest fees. Is it normal to have s19s get to such a high hash rate, and can that blow out fans, or should we just continue operating with antpool?

[HagssFIN]

Did you know that Antpool take all your shares of tx fees and keep it? So much for the low fee....

I don't think the variance in hash rate is related to your fan issue, and you already said it yourself that you suspect there is something wrong with the external fan setup.

If you get it closer to what Phil suggested, your gear will feel better.

Do you know your external fan setup specs?
If you can post them in same kind of a format Phil did, maybe we can notice something.

[kp98]

Did you know that Antpool take all your shares of tx fees and keep it? So much for the low fee....

I don't think the variance in hash rate is related to your fan issue, and you already said it yourself that you suspect there is something wrong with the external fan setup.

If you get it closer to what Phil suggested, your gear will feel better.

Do you know your external fan setup specs?
If you can post them in same kind of a format Phil did, maybe we can notice something.

I see, what pool do you recommend is best? I found out our fans are not turned on. However, it could be a pressure issue since we have an exhaust fan in the building that exhausts 5000 CFM minimum, and we do not have much of an intake at all

[mikeywith]

Thanks, I think this may be the case. We haven't put a lot of thought into air pressure but I could certainly see that being the cause. If the problem is the control board, for instance, what would be a signal that that is the case? If you have any other information on why fans may die do let me know.

I tried to use all the imagination I can afford and honestly I can't see how your control board will damage the fans, the miner's fan isn't some sort of a sophisticated piece of electronics where a dozen of things can go wrong with it, it is just a .. fan.  So I still suspect it's the air pressure that causes this increase fan failure, before you fix it, there isn't much you need to troubleshoot, not even the quality of the fans.

is it normal to see a miner's hash power to oscillate significantly - on our s19s the average hashrate is 95th/s but I've seen as low as 50 th/s and as high as 250 th/s. I wondered if this could be blowing out the fan somehow.

Where do you see that on the pool status page or the miner itself? hash variance on the pool is pretty normal, on your miner status page, it's a lot less noticeable, however, seeing 50th and 250th from a mining gear that puts on 95th on average is somehow very strange, I have seen my M20s does 80th instead of 64th, sometimes 55th, but nothing close to 50% or 200% as you explain, it certainly is strange.

The hashrate variance is probably caused by a bug on antpool, maybe the difficulty code isn't right, or maybe some issues with the miner itself, but really, before pointing your gears to a proper pool -- it's hard to tell where the problem could be, so if you want PPS+ pool, try Viabtc (i personally use it), it has a bit higher fees than most other PPS+ pools, but to me, it's worth every satoshi.

[philipma1957]

At op try joining viabtc.com pps+

pays flat rate per th every day.  about 30 cents a th or 30 bucks a day for s19

steady predictable payout.

viabtc.com 

[kp98]

Thanks, I think this may be the case. We haven't put a lot of thought into air pressure but I could certainly see that being the cause. If the problem is the control board, for instance, what would be a signal that that is the case? If you have any other information on why fans may die do let me know.

I tried to use all the imagination I can afford and honestly I can't see how your control board will damage the fans, the miner's fan isn't some sort of a sophisticated piece of electronics where a dozen of things can go wrong with it, it is just a .. fan.  So I still suspect it's the air pressure that causes this increase fan failure, before you fix it, there isn't much you need to troubleshoot, not even the quality of the fans.

is it normal to see a miner's hash power to oscillate significantly - on our s19s the average hashrate is 95th/s but I've seen as low as 50 th/s and as high as 250 th/s. I wondered if this could be blowing out the fan somehow.

Where do you see that on the pool status page or the miner itself? hash variance on the pool is pretty normal, on your miner status page, it's a lot less noticeable, however, seeing 50th and 250th from a mining gear that puts on 95th on average is somehow very strange, I have seen my M20s does 80th instead of 64th, sometimes 55th, but nothing close to 50% or 200% as you explain, it certainly is strange.

The hashrate variance is probably caused by a bug on antpool, maybe the difficulty code isn't right, or maybe some issues with the miner itself, but really, before pointing your gears to a proper pool -- it's hard to tell where the problem could be, so if you want PPS+ pool, try Viabtc (i personally use it), it has a bit higher fees than most other PPS+ pools, but to me, it's worth every satoshi.

It's on the pool status. ie antpool login & worker overview. I do not see that variance by going to the subnet then the individual miner. Also thanks for the recommendation we will give it a go!

[mikeywith]

It's on the pool status. ie antpool login & worker overview.

The level of fluctuation you talked about is not usual but it really does not matter at all, your average accepted hashrate is all that matters as far as your payout is concerned, so if your daily or 12 hours hashrate is within -+1 to -+2% of the hashrate you see on the miner status page - everything is just fine and the fluctuation should be ignored, if your average hashrate on the long run also fluctuates as much then you that means something is wrong somewhere.

[kp98]

It's on the pool status. ie antpool login & worker overview.

The level of fluctuation you talked about is not usual but it really does not matter at all, your average accepted hashrate is all that matters as far as your payout is concerned, so if your daily or 12 hours hashrate is within -+1 to -+2% of the hashrate you see on the miner status page - everything is just fine and the fluctuation should be ignored, if your average hashrate on the long run also fluctuates as much then you that means something is wrong somewhere.

do brief departures to 200+ th/s result in more fan failures?

[mikeywith]

do brief departures to 200+ th/s result in more fan failures?

I'd say 99.99% NO.