Antminer S15 with latest firmware consumes 1900W in lower temp env of 24C

[BogdanO]

Hi guys,

I recently purchased a S15 and I ran several tests since, measuring the power consumption on wall. All seemed fine until 2 days ago when I installed the latest firmware Antminer-S15-user-201902251130-sig.tar.gz
In Finland the outside temp is cool enough to allow my S15 to report a "PCB temp" while running of 23-26-27-23C in the 4 rows and first column of the monitoring screen. The chips temp is in the 60-72C range for all tests.

Here's my test case:

1. I start the miner in a 20-21C room, allow it the time to boot. Initially all seems ok, hash rate 28TH and on wall consumption is 1600Wh.
2. 4 mins after this I see this logs in the kernel log (notice the low_bringup_pcb_temp_threshhold):

Code:

2019-04-22 10:03:48 driver-btm-soc.c:3774:check_system_work: Entering low environment handler...
2019-04-22 10:03:48 driver-btm-soc.c:3775:check_system_work: bring_up_pcb_temp = 25, low_bringup_pcb_temp_threshhold = 27, ideal rate = 28000, AVG_rate = 26838 
2019-04-22 10:03:48 driver-btm-soc.c:7929:re_open_core: re_open_core start!
2019-04-22 10:03:50 dspic33ep16gs202.c:572:set_pic_heart_beat: Heartbeat failed on Chain[3], retry_count=1! read_back_data[1] = 0x0, read_back_data[2] = 0x0
2019-04-22 10:03:51 dspic33ep16gs202.c:572:set_pic_heart_beat: Heartbeat failed on Chain[2], retry_count=1! read_back_data[1] = 0xfe, read_back_data[2] = 0xfe

3. The miner consumption drops to some idle 100W for the next few minutes.
4. For the next 5 mins it reinitialises the hash board and finishes with:

Code:

2019-04-22 10:09:07 power.c:198:get_power_voltage_from_iic_value: iic_index for voltage[46] = 19.520524
2019-04-22 10:09:07 power.c:229:set_iic_power_by_iic_data: now setting voltage to : 19.553896 
2019-04-22 10:09:08 power.c:198:get_power_voltage_from_iic_value: iic_index for voltage[47] = 19.503839
2019-04-22 10:09:08 power.c:229:set_iic_power_by_iic_data: now setting voltage to : 19.520524 
2019-04-22 10:09:09 power.c:439:get_chain_voltage: chain[0], voltage is: 19.24 
2019-04-22 10:09:10 power.c:439:get_chain_voltage: chain[1], voltage is: 19.23 
2019-04-22 10:09:10 power.c:439:get_chain_voltage: chain[2], voltage is: 19.28 
2019-04-22 10:09:11 power.c:439:get_chain_voltage: chain[3], voltage is: 19.24 
2019-04-22 10:09:11 power.c:462:get_average_voltage: average  voltage is: 19.25 
2019-04-22 10:09:11 driver-btm-soc.c:3794:check_system_work: low environment handler!
2019-04-22 10:09:11 driver-btm-soc.c:3795:check_system_work: low temperature reopen core and increase voltage 1 times!

5. At this point the miner is up again. Hash rate is up to 28TH again but the on wall consumption jumps to 1920W and stays on like that.
6. I bring the unit down and move it deeper inside the house (23-24C) where the temp PCB reads 36-40-38-36C. I power it up again.
7. This time power consumption stay at 1620W and there is no reinit of the hashboards and last lines in the kernel logs read:

Code:

2019-04-22 10:20:19 dspic33ep16gs202.c:65:volt_clamp: failed on Chain[3], b[0] = 254, b[2] = 254!
2019-04-22 10:20:21 dspic33ep16gs202.c:65:volt_clamp: failed on Chain[3], b[0] = 254, b[2] = 254!
2019-04-22 10:50:25 driver-btm-soc.c:3706:check_system_work: The avg rate is  28206 in 30 mins, PCB temperature between 36 ~ 62

Anyone shares the same experience? I didn't notice this on the old firmware. The new firmware mentions one thing that worries me: "Add low temperature protection function". Anyone knows where to pull the old firmware from (ie the December 2018 one)?

Cheers,
Bogdan

[BitMaxz]

... Anyone shares the same experience? I didn't notice this on the old firmware. The new firmware mentions one thing that worries me: "Add low temperature protection function". Anyone knows where to pull the old firmware from (ie the December 2018 one)?

Maybe you are experiencing low temp protection this might be the reason why you are getting unstable hashrate for the first run did you try to run the miner for a longer time? Let it run first for 3 hours and let see if you will get stable hash rate and miner consumption.

Next time don't upgrade the firmware to the latest version if your miner is running fine without any software issue. Bitmain releases new 2019 firmware that mostly fixed all security issue(Disabled SSH access) if you upgrade the miner to 2019 firmware the miner SSH access is disabled where you can modify the config and bypass temp protection.

There is no other way for now to downgrade the firmware to lower version unless if you decide to run them with modified firmware.

[philipma1957]

Maybe you are experiencing low temp protection this might be the reason why you are getting unstable hashrate for the first run did you try to run the miner for a longer time? Let it run first for 3 hours and let see if you will get stable hash rate and miner consumption.

Next time don't upgrade the firmware to the latest version if your miner is running fine without any software issue. Bitmain releases new 2019 firmware that mostly fixed all security issue(Disabled SSH access) if you upgrade the miner to 2019 firmware the miner SSH access is disabled where you can modify the config and bypass temp protection.

There is no other way for now to downgrade the firmware to lower version unless if you decide to run them with modified firmware.

@ op  Never do a firmware upgrade  from bitmain  until  many people have done it.

op I tossed you some merits for letting us know about this.

[Artemis3]

Is there any modded firmware for the S15?

It is a shame this situation is extending to all their products, but i learned about it first with the S15 when someone announced defeating the "security feature". Now the question is, can the S15 be made to boot from an sd?

OP sadly made the mistake of upgrading the firmware, but Bitmain has removed him the chance to revert back.

[BogdanO]

Maybe you are experiencing low temp protection this might be the reason why you are getting unstable hashrate for the first run did you try to run the miner for a longer time? Let it run first for 3 hours and let see if you will get stable hash rate and miner consumption.

Next time don't upgrade the firmware to the latest version if your miner is running fine without any software issue. Bitmain releases new 2019 firmware that mostly fixed all security issue(Disabled SSH access) if you upgrade the miner to 2019 firmware the miner SSH access is disabled where you can modify the config and bypass temp protection.

Jus FYI, I received the unit 2 weeks ago, loaded with the 2018 Dec firmware in which the SSH was already disabled. I upgraded hoping for a fix to the root cause (the ssh vulnerability) and reactivation of the SSH. The hash rate is stable once a certain operational mode is achieved (ie cold or warm env operation.

Now... my main concern here is the increased power consumption to 1900W in colder env (20-24C 68-75F) (which is not that cold anyway and I guess many datacenters keep it even below this):

If I operate the miner in such "colder environment", hoping for a better performance (lower RPM fan) and efficient cooling, it ends up backfiring by jumping the electricity consumption by 300W (at the same hash rate), which at the current BTC price and electricity rate changes the whole economics quite a little. I bought this one miner because of improved efficiency and all that goes down the drain if it's too cold in the room?!? This cold protection seems to defeat the whole purpose of redesigning S15 as a more energy efficient miner.

I'll continue testing when I'll have more time and report back.

Thanks for all the comments.

[BitMaxz]

Now... my main concern here is the increased power consumption to 1900W in colder env (20-24C 68-75F) (which is not that cold anyway and I guess many datacenters keep it even below this)...

The power consumption is too high it should be around 1600w +/-7% there is something wrong about the power supply. Have you tried to enable the low power mode at miner configuration tab? Let's see what changes in power consumption and hashrate?

I thought something that you should try and maybe you may able to reduce the current power consumption of the miner. Try this method below I know its for s9 but you can try it on s15 and maybe it may work.

- Solution for ANTMINER S9 FANS SPEED Up & Down every 20 Seconds.

Follow the guide above to appear the hidden menu where you can see the fan speed and frequency settings. Just play the settings both fan speed and frequency and let's see if it affects the temp and power consumption.

[BogdanO]

Just to be clear... my experience is that the S15 indeed consumes 1600W +/- 20W as long as the room temperature is high enough that the PCB temp reads above 27C. If not only then the consumption jumps to 1900W.

[BitMaxz]

Just to be clear... my experience is that the S15 indeed consumes 1600W +/- 20W as long as the room temperature is high enough that the PCB temp reads above 27C. If not only then the consumption jumps to 1900W.

Yeah, that is why I suggested you to try the link above to see the hidden fan speed and frequency my thoughts is to try to change the fan speed to lower speed so that the PCB can be reach to 27c or above so that the low temperature protection not be able to activate.

That is the purpose of changing the fan speed to lower since you have colder environment this is the only solution that I know if the method above will work.

[mikeywith]

on the s15 specfication on bitmain it does mention that the ideal room temp for the s15 is 25 °C  > https://shop.bitmain.com/promote/antminer_s15_asic_bitcoin_miner/specification

so anything lower than that is not good.

from the kernel log you posted , it's pretty clear that the firmware has a count down of about 4 + 5 mins that checks the PCB temp , if it does not increase to where it wants it to be (36 ~ 62) it does this

Code:

low temperature reopen core and increase voltage 1 times!

the extra 300w it consumes afterwards is only the effect of voltage increasing which is the only way that miner has to increase the PCB temp ( higher voltage = increase in temps)  if you were to run it at even a colder room, then my guess is that it will increase the voltage twice and power on the wall will go pass 2000w.

so your best shot at this is to follow BitMaxz's advice and lower the fans' rpms to a degree where it does not spin fast enough to cool the miner below the ideal temp, remember you only have a few mins before the miner checks for temps, so you might need to provide some heat , maybe point the exhaust fan of another miner to the intake of this miner to warm it at start up ?

if you can't do that, then please allow more time, chances are the miner will do another temp check to drop the voltage after a certain time has passed, and if the voltage does drop then the power consumption will drop too.

and by the way, i wish i had a problem like yours , i think we all do   , it is summer here and all i worry about is how to cool my gears down, just the exact opposite of what  you are tying to do.

[fyoung55]

Unfortunately, when it autotunes, it runs the fans on max, which is when it identifies the pcb temp, and jacks up the voltage.  The only solution I have found with the s15 is to feed warmer air in during autotune (mixed room air and T15 miner exhaust).  Then when set fans take over it keeps the pbc warm enough to avoid that 300w issue, because it checks pcb temps every 30 minutes or so and at any point, if the boards get too cold, it jacks up the voltage.

A custom firmware that sets voltage and freq to static would likely overcome this issue...

My question is why are temps below 25C a problem?

[philipma1957]

Unfortunately, when it autotunes, it runs the fans on max, which is when it identifies the pcb temp, and jacks up the voltage.  The only solution I have found with the s15 is to feed warmer air in during autotune (mixed room air and T15 miner exhaust).  Then when set fans take over it keeps the pbc warm enough to avoid that 300w issue, because it checks pcb temps every 30 minutes or so and at any point, if the boards get too cold, it jacks up the voltage.

A custom firmware that sets voltage and freq to static would likely overcome this issue...

My question is why are temps below 25C a problem?

 They are not  problem but the firmware was set to pick that number as the trigger point.

If you went into the firmware and alter the 25c tigger point to 18c it would solve the problem.

[fyoung55]

that would work, but can that be done?  I thought the s15s were locked down tight.

[philipma1957]

My bad  if you could do what I said it would work , but you can't.

I think you can do this below:

You could  get weaker fans.

if you have 6000 rpm fans  that move  220 cfm

you could swap to  4000-  4800 rpm fans that move 180 cfm

Delta makes some

https://www.ebay.com/itm/DELTA-AFC1212DE-120-120-38MM-12V-1-6A-12CM-4Pin-Cooling-Fan/263916014114?

[mikeywith]

that would work, but can that be done?  I thought the s15s were locked down tight.

According to vnish.net, they should have their firmware ready for S15 in a few weeks, his/her firmware usually allows you to modify the voltage and frequency of each chip individually, it's also very likely that they will disable the function that causes the problem altogether and you won't really have to do a thing about it, provided that they actually know about this problem, so maybe reach out and let them know about it if you plan to use their firmware.

[lightfoot]

Unfortunately, when it autotunes, it runs the fans on max, which is when it identifies the pcb temp, and jacks up the voltage.  The only solution I have found with the s15 is to feed warmer air in during autotune (mixed room air and T15 miner exhaust).  Then when set fans take over it keeps the pbc warm enough to avoid that 300w issue, because it checks pcb temps every 30 minutes or so and at any point, if the boards get too cold, it jacks up the voltage.

A custom firmware that sets voltage and freq to static would likely overcome this issue...

My question is why are temps below 25C a problem?

I've been thinking about that: The S15's run at a higher DC voltage (I think it's 20v) than the S9's (12) and have 72 chips instead of 63.

The problem with chips is that as temperature increases, current draw will increase and in a series string you'll have the voltages on chips in different parts of the board go out of sync due to resistance dropping on the chip as temp increases.  When you get up to that many chips in series on that long a board you can run into situations where cooler chips will pull less current than the warmer chips, causing the voltages to go very weird very quickly and possibly blow out the hotter chips.

Thus I'm guessing that Antland wants to run the chips at a min temperature to keep this from happening, especially on startup. In the S9 they did this partially by removing the lower heat sinks on the chips that were most in the coolest air stream (those by the intake fans). For the S15 they run the whole bunch in a less efficient mode to keep the temps up. I wonder if they also implemented per chip clocking to assist with this and keep the board at a more constant temp. Hm.

[philipma1957]

This is a good idea if it is  so using the lessor 4000 rpm fans would fix a cold room issue. The chips would run warmer and voltage would be dropped.

[mikeywith]

This is a good idea if it is  so using the lessor 4000 rpm fans would fix a cold room issue. The chips would run warmer and voltage would be dropped.

If that would fix it - wouldn't setting the fans to low fixed/static RPMs fix the problem? according to fyoung55's reply on the fan-related topic, the trick works on S15.

Confirmed on T15 as well!

my concern is the autotune part, assuming it does that every once in a while and fans must spin at 100%, will the tuning time be enough to cool the chips down thus increasing the voltage? I guess it's worth a try.

[philipma1957]

A good point.

It is very hard for me to test as I am warm room 30c most of the time.

[fyoung55]

newest update and info on my T15 and S15 and voltage jumps...

I run them in my basement and the room temp is usually between 15 and 20 C all fall/winter/spring.  Summer may be warmer, we will see...

When auto tune runs at the start, the fans cycle up to a full 100%, as long as the PCBs are above 25 it doesn't jump up the voltage. A slower fan like phillipma suggested might fix that, since autotune overrides the static fan and sets fans to 100%.  I run static fans at 60% right now which is around 3600 rpm.  I'll look into ordering them and get back to you.

Further, it seems from watching the temps and fan speeds that if any CHIPS exceed 80C at any point, the firmware overides my static fan speed, and they spin up to 100% until the chip drops below 81 again.  I was running my s15 at 50% but it kept cycling every few minutes up to 100% for a few seconds.  It annoyed me, so I sat and watched it for about 5 minutes and identified the problem.  Set fans to static 60% and my hottest chips sit at 74-75 - NO FAN JUMPS

anyone use the awesome miner software with these?  the free version doesn't do much, but I wonder if the paid version allow more access?  I have 2018 firmware on my units still, not the super locked down Dec 2019 update.

[mikeywith]

There is no free Awoesomeminer firmware, they have a fee between 2-3%, one consumes a license and other one doesn't, and up to this point in time they don't have any firmware for S15 and T15, neither does Asic.to, the only two firmware distributors that I would trust, I am sure there are some unknown firmware versions out there, but most of them are nothing but scams.

I think your best shot would be getting lower RPM fans as phill suggested or/and wait for that firmware to be ready, supposedly in a few weeks as mentioned on their website.