[ANN] Spondoolies-Tech - carrier grade, data center ready mining rigs

[jtoomim]

5 out of 10 SP30's seem to have failed after about 10 hours with the new firmware 2.5.6
any suggestions?

Look in the Events tab. Post what you find.

Try disabling ASICs. First, disable all but one ASIC. See if it starts up. If it does, then try enabling a few more ASICs, then a few more, etc. Eventually, you may be able to isolate one or two ASICs that are causing the trouble.

If you see "Loop serial failed X" messages, then you can start by disabling loop X, then re-enabling those ASICs one by one. If you see that message, you should also let Zvi know (and preferably also me by Skype; my Skype ID is jtoomim).

You're welcome to contact me on Skype if you want more advice. Zvi's rather busy, and I'd be happy to help if I can until he can get his workload under control again.

[1714770261]

1714770261

[1714770261]

1714770261

[1714770261]

1714770261

[faraway]

5 out of 10 SP30's seem to have failed after about 10 hours with the new firmware 2.5.6
any suggestions?

Look in the Events tab. Post what you find.

Try disabling ASICs. First, disable all but one ASIC. See if it starts up. If it does, then try enabling a few more ASICs, then a few more, etc. Eventually, you may be able to isolate one or two ASICs that are causing the trouble.

If you see "Loop serial failed X" messages, then you can start by disabling loop X, then re-enabling those ASICs one by one. If you see that message, you should also let Zvi know (and preferably also me by Skype; my Skype ID is jtoomim).

You're welcome to contact me on Skype if you want more advice. Zvi's rather busy, and I'd be happy to help if I can until he can get his workload under control again.

Some ASICs can't be activated again. Is this the main sha256 chip that is affected, or a different hardware failure?

[miner454]

Guy,

My two SP31 October batches are defective.  My ROI is basically screwed.  Sh*t happens.

The first SP31 never came up with the latest firmware etc.
The second SP31 with the latest firmware ran for a while and now it's basically as good as dead.

Minersource tech support (the colo that is hosting my SP31's) is waiting for Zvisha's reply.  But I got an email fro sptech today saying Zvisha is waiting for Minersource.  We are going in circles.

I really don't want to deal with two dead SP31's.  Minesource has put too much effort trying to get these SP31's online.  I had already purchased two hosting that will not even be used for this.

Could I get authorization to return these two defective SP31's?

Please reply.

Your Loyal and Return Customer.

[Guy Corem]

We'll RMA both units and improve our support procedures with Minersource.

Guy

[faraway]

5 out of 10 SP30's seem to have failed after about 10 hours with the new firmware 2.5.6
any suggestions?

Look in the Events tab. Post what you find.

Try disabling ASICs. First, disable all but one ASIC. See if it starts up. If it does, then try enabling a few more ASICs, then a few more, etc. Eventually, you may be able to isolate one or two ASICs that are causing the trouble.

If you see "Loop serial failed X" messages, then you can start by disabling loop X, then re-enabling those ASICs one by one. If you see that message, you should also let Zvi know (and preferably also me by Skype; my Skype ID is jtoomim).

You're welcome to contact me on Skype if you want more advice. Zvi's rather busy, and I'd be happy to help if I can until he can get his workload under control again.

Some ASICs can't be activated again. Is this the main sha256 chip that is affected, or a different hardware failure?

One of my ASIC came back to life. I suspect now, a chip undefined state, unaffected by the RESET signal. If this is really the case have you a trick (software, etc..) to set the chip in the proper state, and enforce the RESET sequence? May you fix this bug?

[promojo]

How common is it for these ASIC loops to drop off or need to be disabled?  Essentially that is reducing your mining power....  Expected lifespan?  Do we even know?

[Biffa]

How common is it for these ASIC loops to drop off or need to be disabled?  Essentially that is reducing your mining power....  Expected lifespan?  Do we even know?

I've got one of the first SP10's ever made and no ASIC dropouts.

[raskul]

How common is it for these ASIC loops to drop off or need to be disabled?  Essentially that is reducing your mining power....  Expected lifespan?  Do we even know?

I've got one of the first SP10's ever made and no ASIC dropouts.

ditto. still the world's best miner

[DevonMiner]

How common is it for these ASIC loops to drop off or need to be disabled?  Essentially that is reducing your mining power....  Expected lifespan?  Do we even know?

I've got one of the first SP10's ever made and no ASIC dropouts.

ditto. still the world's best miner

Ditto as well ... but I don't think this problem ever showed on the SP10s, it's a SP30/31 thing and maybe now the SP20. (Stand to be corrected if I'm wrong).

SP10 ... solid as a rock!

[Biffa]

How common is it for these ASIC loops to drop off or need to be disabled?  Essentially that is reducing your mining power....  Expected lifespan?  Do we even know?

I've got one of the first SP10's ever made and no ASIC dropouts.

ditto. still the world's best miner

Ditto as well ... but I don't think this problem ever showed on the SP10s, it's a SP30/31 thing and maybe now the SP20. (Stand to be corrected if I'm wrong).

SP10 ... solid as a rock!

SP30 has been good so far in my experience as well, SP20 well considering the abuse I am subjecting it to its not doing too bad either.

5+ hours on 2.5.12 with no errors.

[Marvell1]

Still having issues with mine.  I'm thing 2.5.7 might have damaged my Asics its super unstable.  Right now I have it reset to factory setting at 70% fan.  When I try to bump up the fan to my normal
80% the problems get worse : constant cgminer restarts asics going into fault loop.  ugh this is not good.

[jtoomim]

Still having issues with mine.  I'm thing 2.5.7 might have damaged my Asics its super unstable.  Right now I have it reset to factory setting at 70% fan.  When I try to bump up the fan to my normal
80% the problems get worse : constant cgminer restarts asics going into fault loop.  ugh this is not good.

Try disabling the problematic ASICs, then wait a few hours or a few days, and try to reenable the ASICs again. You can add a few long power-off cycles as well. That seems to have worked for most of our machines that have had issues with 2.5.7. Hit me up on Skype (jtoomim) if you want me to take a look at it.

[zvisha]

Still having issues with mine.  I'm thing 2.5.7 might have damaged my Asics its super unstable.  Right now I have it reset to factory setting at 70% fan.  When I try to bump up the fan to my normal
80% the problems get worse : constant cgminer restarts asics going into fault loop.  ugh this is not good.

Try disabling the problematic ASICs, then wait a few hours or a few days, and try to reenable the ASICs again. You can add a few long power-off cycles as well. That seems to have worked for most of our machines that have had issues with 2.5.7. Hit me up on Skype (jtoomim) if you want me to take a look at it.

I recommend trying 2.5.12. It looks pretty solid for both SP20 and SP3x. It has annoying redirection issue after reboot, you can not miss it if you reboot (it goes to router after reboot).
In SP20 there are some things that can go wrong because of the physical damage in shipment - we are working on it. Skype me if needed.
P.S. We will update next version to power limit of 288W for SP20 and 1275w for SP31, you can change it if you want.

UPD: If you want to set Murata above 1300, you need to use "/etc/mg_ignore_110_fcc" file.

[edgar]

just to add - i left it being 'lame' for 20 minutes and it has found its way back online at 1456GH..

i will attempt to upgrade FW tonight ...12-14 hrs from now

[jtoomim]

I have had 39 SP30s running 2.5.12 so far for about 12 hours 16 minutes with no apparent issues. If I see no problems with 2.5.12 by next morning, I will probably move the rest of our machines over to it. Otherwise, I will probably move everything back to 2.4.23.

I've had a plethora of "disabled (bist cant start, stuck X)" errors on the remaining 100 or so SP30s we have running 2.5.6 in the mean time. It seems every few hours, I have to restart cgminer on 5% or 10% of our machines to get them back to full hashrate. Very annoying, but so far the issue has always been resolved by restarting minergate.

I've also had several reports of machines on 2.5.6 showing half hashrate at the pool even though I see full hashrate. These issues have always been restarted by a minergate restart.

I saw some much more serious problems on 2.5.7 during the 36 hours we had machines using that version. We saw about 10 machines get "Loop serial failed x" error messages (where x is a loop number, between 0 and 9 inclusive) and fail to start until one or sometimes two ASICs on that loop were disabled. After a few hours, many of these ASICs could be enabled again, but about 25% of the ones that failed are still non-operational. Currently, I think there are 9 ASICs which we lost on 2.5.7 which have not recovered.

It's possible that the potential ASIC-damaging bug in 2.5.7 might still be present in 2.5.12. I haven't seen any indications of it yet, but I also haven't had enough machines running on it for long enough to be certain. On 2.5.7, I saw the loop serial failed ASIC damage thing with about 5% of our machines per day. At that rate, there's only about a 63% chance that I would have seen it if it were present at the same rate in 2.5.12 with 39 machines running half a day.

We moved everything onto 2.5.12 about 5 hours ago (11am PST). Everything looks much better now. When we were running 2.5.6 on most machines, our hashrate would vary between 706 TH/s and around 722 TH/s, depending on how active I had been restarting machines with "bist cant start (stuck x)" errors. Now we're at 724 TH/s (pus another 4.5 TH/s that we added this morning), and everything seems a lot more stable and requires less effort to keep things going. I recommend anyone using an SP30 or SP31 with 2.5.6 to 2.5.10 firmware to upgrade to 2.5.12 immediately. I also recommend to Zvi to mark 2.5.6 as unstable/testing.

[jtoomim]

P.S. We will update next version to power limit of 288W for SP20 and 1275w for SP31, you can change it if you want.

It's really annoying when you do this. We've already tested our SP30s and SP31s and found the PSU limits that work best for each machine. In the past, when firmware changes these settings, we haven't had records of which machines were using which settings and have had to go back and figure them out again from scratch. Not fun. You have more machines deployed successfully with PSU limits higher than 1275W than you have shipping out right now. Overwriting old settings is more disruptive than leaving them be.

Also, 1275W sounds very low.

[bobsag3]

P.S. We will update next version to power limit of 288W for SP20 and 1275w for SP31, you can change it if you want.

It's really annoying when you do this. We've already tested our SP30s and SP31s and found the PSU limits that work best for each machine. In the past, when firmware changes these settings, we haven't had records of which machines were using which settings and have had to go back and figure them out again from scratch. Not fun. You have more machines deployed successfully with PSU limits higher than 1275W than you have shipping out right now. Overwriting old settings is more disruptive than leaving them be.

Also, 1275W sounds very low.

I am in agreement here. Having to retune 50+ machines every firmware update gets very tedious.

[faraway]

To continue on the disabled ASIC issue topic (following the 2.5.7 upgrade).

I've still two ASICs in a 'stuck' state; They operated again, for some time, a few hours ago; But, after a reboot the issue is back on the same ASICs.

If we consider there is no hardware damage (I hope so). Do you know, a reliable method, to get them in a deterministic and operational state?

[ScaryHash]

I will try to contact Zvi or jtoomin friday morning sometime.

After applying 2.5.12 firmware upgrade, I still have a stuck ASIC 4. I did look through the log, and I have seen a new error after applying the firmware.

Error::: PLL stuck:ffffffff asic(4/255) LOOP:127 (Initial freq), 3000
Nov 11 02:33:35 miner local1.warn minergate[1019]: Critical: ASIC ERROR PLL on address 4/255
Nov 11 02:33:35 miner local1.warn minergate[1019]: Discovered stuck PLL 7000001

If that makes that ASIC try to run at 3000 Mhz, than that would be the hangup. I highly doubt these things are rated for 3000 Mhz...

[jtoomim]

Error::: PLL stuck:ffffffff asic(4/255) LOOP:127 (Initial freq), 3000
Nov 11 02:33:35 miner local1.warn minergate[1019]: Critical: ASIC ERROR PLL on address 4/255
Nov 11 02:33:35 miner local1.warn minergate[1019]: Discovered stuck PLL 7000001

That error usually goes away with a power cycle or two. Restarting minergate isn't enough for that one. If power cycles don't fix it, just disable ASIC 4 and your machine should run.

P.S.: It's jtoomim, not jtoomin.