I have had 39 SP30s running 2.5.12 so far for about 12 hours 16 minutes with no apparent issues. If I see no problems with 2.5.12 by next morning, I will probably move the rest of our machines over to it. Otherwise, I will probably move everything back to 2.4.23.
I've had a plethora of "disabled (bist cant start, stuck X)" errors on the remaining 100 or so SP30s we have running 2.5.6 in the mean time. It seems every few hours, I have to restart cgminer on 5% or 10% of our machines to get them back to full hashrate. Very annoying, but so far the issue has always been resolved by restarting minergate.
I've also had several reports of machines on 2.5.6 showing half hashrate at the pool even though I see full hashrate. These issues have always been restarted by a minergate restart.
I saw some much more serious problems on 2.5.7 during the 36 hours we had machines using that version. We saw about 10 machines get "Loop serial failed x" error messages (where x is a loop number, between 0 and 9 inclusive) and fail to start until one or sometimes two ASICs on that loop were disabled. After a few hours, many of these ASICs could be enabled again, but about 25% of the ones that failed are still non-operational. Currently, I think there are 9 ASICs which we lost on 2.5.7 which have not recovered.
It's possible that the potential ASIC-damaging bug in 2.5.7 might still be present in 2.5.12. I haven't seen any indications of it yet, but I also haven't had enough machines running on it for long enough to be certain. On 2.5.7, I saw the loop serial failed ASIC damage thing with about 5% of our machines per day. At that rate, there's only about a 63% chance that I would have seen it if it were present at the same rate in 2.5.12 with 39 machines running half a day.
It is now looking like the issues we've been seeing in our datacenter with the potential ASIC-damaging bug was not related to version 2.5.7 per se, but is instead temperature-related, and we only noticed it now because the temperatures in our DC only recently started getting really close to 0°C. Version 2.5.13 does not appear to fix these issues. Zvi is looking into it further. The other two bugs were definitely firmware related, so you should still avoid 2.5.6 and 2.5.7. Findings are still tentative and provisional, and investigation is ongoing. |
|
|
|
MS recieve my Unit on 31 October
then mailing, waiting , mailing,waiting ,...
normal that after eleven days my unit is not online ?
You may want to post here instead or as well, as this sounds more like a hosting issue than a Spondoolies issue: https://bitcointalk.org/index.php?topic=622998.new#new |
|
|
|
Error::: PLL stuck:ffffffff asic(4/255) LOOP:127 (Initial freq), 3000
Nov 11 02:33:35 miner local1.warn minergate[1019]: Critical: ASIC ERROR PLL on address 4/255
Nov 11 02:33:35 miner local1.warn minergate[1019]: Discovered stuck PLL 7000001
That error usually goes away with a power cycle or two. Restarting minergate isn't enough for that one. If power cycles don't fix it, just disable ASIC 4 and your machine should run. P.S.: It's jtoomim, not jtoomin. |
|
|
|
P.S. We will update next version to power limit of 288W for SP20 and 1275w for SP31, you can change it if you want.
It's really annoying when you do this. We've already tested our SP30s and SP31s and found the PSU limits that work best for each machine. In the past, when firmware changes these settings, we haven't had records of which machines were using which settings and have had to go back and figure them out again from scratch. Not fun. You have more machines deployed successfully with PSU limits higher than 1275W than you have shipping out right now. Overwriting old settings is more disruptive than leaving them be. Also, 1275W sounds very low. |
|
|
|
I have had 39 SP30s running 2.5.12 so far for about 12 hours 16 minutes with no apparent issues. If I see no problems with 2.5.12 by next morning, I will probably move the rest of our machines over to it. Otherwise, I will probably move everything back to 2.4.23.
I've had a plethora of "disabled (bist cant start, stuck X)" errors on the remaining 100 or so SP30s we have running 2.5.6 in the mean time. It seems every few hours, I have to restart cgminer on 5% or 10% of our machines to get them back to full hashrate. Very annoying, but so far the issue has always been resolved by restarting minergate.
I've also had several reports of machines on 2.5.6 showing half hashrate at the pool even though I see full hashrate. These issues have always been restarted by a minergate restart.
I saw some much more serious problems on 2.5.7 during the 36 hours we had machines using that version. We saw about 10 machines get "Loop serial failed x" error messages (where x is a loop number, between 0 and 9 inclusive) and fail to start until one or sometimes two ASICs on that loop were disabled. After a few hours, many of these ASICs could be enabled again, but about 25% of the ones that failed are still non-operational. Currently, I think there are 9 ASICs which we lost on 2.5.7 which have not recovered.
It's possible that the potential ASIC-damaging bug in 2.5.7 might still be present in 2.5.12. I haven't seen any indications of it yet, but I also haven't had enough machines running on it for long enough to be certain. On 2.5.7, I saw the loop serial failed ASIC damage thing with about 5% of our machines per day. At that rate, there's only about a 63% chance that I would have seen it if it were present at the same rate in 2.5.12 with 39 machines running half a day.
We moved everything onto 2.5.12 about 5 hours ago (11am PST). Everything looks much better now. When we were running 2.5.6 on most machines, our hashrate would vary between 706 TH/s and around 722 TH/s, depending on how active I had been restarting machines with "bist cant start (stuck x)" errors. Now we're at 724 TH/s (pus another 4.5 TH/s that we added this morning), and everything seems a lot more stable and requires less effort to keep things going. I recommend anyone using an SP30 or SP31 with 2.5.6 to 2.5.10 firmware to upgrade to 2.5.12 immediately. I also recommend to Zvi to mark 2.5.6 as unstable/testing. |
|
|
|
Still having issues with mine. I'm thing 2.5.7 might have damaged my Asics its super unstable. Right now I have it reset to factory setting at 70% fan. When I try to bump up the fan to my normal
80% the problems get worse : constant cgminer restarts asics going into fault loop. ugh this is not good.
Try disabling the problematic ASICs, then wait a few hours or a few days, and try to reenable the ASICs again. You can add a few long power-off cycles as well. That seems to have worked for most of our machines that have had issues with 2.5.7. Hit me up on Skype (jtoomim) if you want me to take a look at it. |
|
|
|
5 out of 10 SP30's seem to have failed after about 10 hours with the new firmware 2.5.6
any suggestions?
Look in the Events tab. Post what you find. Try disabling ASICs. First, disable all but one ASIC. See if it starts up. If it does, then try enabling a few more ASICs, then a few more, etc. Eventually, you may be able to isolate one or two ASICs that are causing the trouble. If you see "Loop serial failed X" messages, then you can start by disabling loop X, then re-enabling those ASICs one by one. If you see that message, you should also let Zvi know (and preferably also me by Skype; my Skype ID is jtoomim). You're welcome to contact me on Skype if you want more advice. Zvi's rather busy, and I'd be happy to help if I can until he can get his workload under control again. |
|
|
|
Hi all.
Version 2.5.12 is ready. It solves "low hash rate in pool" for some SP20 having this issue.
For SP30 it works well in our DC, but because it might have done some damage to jtoomim (few ASICs stopped working) - let's wait with the upgrade.
Regards` UPpdate: updated wording not to cause panic  I have had 39 SP30s running 2.5.12 so far for about 12 hours 16 minutes with no apparent issues. If I see no problems with 2.5.12 by next morning, I will probably move the rest of our machines over to it. Otherwise, I will probably move everything back to 2.4.23. I've had a plethora of "disabled (bist cant start, stuck X)" errors on the remaining 100 or so SP30s we have running 2.5.6 in the mean time. It seems every few hours, I have to restart cgminer on 5% or 10% of our machines to get them back to full hashrate. Very annoying, but so far the issue has always been resolved by restarting minergate. I've also had several reports of machines on 2.5.6 showing half hashrate at the pool even though I see full hashrate. These issues have always been restarted by a minergate restart. I saw some much more serious problems on 2.5.7 during the 36 hours we had machines using that version. We saw about 10 machines get "Loop serial failed x" error messages (where x is a loop number, between 0 and 9 inclusive) and fail to start until one or sometimes two ASICs on that loop were disabled. After a few hours, many of these ASICs could be enabled again, but about 25% of the ones that failed are still non-operational. Currently, I think there are 9 ASICs which we lost on 2.5.7 which have not recovered. It's possible that the potential ASIC-damaging bug in 2.5.7 might still be present in 2.5.12. I haven't seen any indications of it yet, but I also haven't had enough machines running on it for long enough to be certain. On 2.5.7, I saw the loop serial failed ASIC damage thing with about 5% of our machines per day. At that rate, there's only about a 63% chance that I would have seen it if it were present at the same rate in 2.5.12 with 39 machines running half a day. |
|
|
|
|
It appears there is a user named "horcof" (note only one 'f') who is PMing people and trying to get them to pay for these miners. I'm pretty sure horcof is a scam. The real owner should be horcoff.
|
|
|
|
Is anyone else getting errors like this on firmware 2.5.7 and 2.5.9? LOOP[8] ON
24: disabled (Asic all engines fail BIST)
25: DC2DC/1/:[vlt1:671 vlt2:674(DCl:794 Tl:794 Ul:749) 50W 74A 38c] ASIC:[ 85c (120c) 580hz(BL: 580) 18 (E:193) F:0]
26: DC2DC/1/:[vlt1:669 vlt2:674(DCl:794 Tl:794 Ul:749) 48W 71A 38c] ASIC:[ 85c (120c) 555hz(BL: 555) 12 (E:193) F:0]
They get cleared by restarting minergate, seem to happen at random, and often come with very large (0.5 to 2 TH/s) reductions in hashrate. I started to get them recently on our machines, and they annoy me. In other news, I did another test on the effects of fan speed at lower temperature (around 9C -- it's night here now), and they came out 4 GH/s in favor of 70. My guess is that at intake temps above 17C, fan speeds at 80 are better, and below 16C, fan speeds at 70 are better. |
|
|
|
The manufacturer's CEO, Guy Corem, told CoinDesk that he had visited the site in late June, when it was primarily populated by about 1,000 Dragon miners and around 100 Spondoolies-Tech miners.
Corem said the facility was expanded shortly after his visit, with hundreds of Spondoolies-Tech SP30 miners...
... including about $2m-worth from Spondoolies-Tech alone.
Pay attention to timing please. There were no SP30 in June. Those are SP10 miners... So 2 million for 100 Sp10 miners is 20k each, even more outrageous. LOL There were about 100 SP10s in June, but then they added hundreds (note the plural) of SP30s. 400 SP30s plus 100 SP10s would cost around US$2m. |
|
|
|
You may want to change the fan speeds as well. I'm not sure yet what the optimum is...
I did a little experiment with SP30s. I took 10 machines, and gave the even-numbered ones a fan speed setting of 80 to start with, and the odd numbered ones a fan speed of 70. I then waited an hour, and recorded their temperatures and performance (from monitor.php). I then switched them up (odd 80, even 70), waited another hour, and recorded again. Here's the result: jtoomim@server:~/wissamperf$ python ~/fan_comparison.py
==== phase1 ====
machine-name timestamp hashrate front top bottom is_odd
wissamitter-10 1415393491 4601120 17 70 68 0
wissamitter-1 1415393491 4593400 17 78 75 1
wissamitter-2 1415393491 4585680 16 71 69 0
wissamitter-3 1415393491 4582785 17 79 74 1
wissamitter-4 1415393491 4560590 17 72 68 0
wissamitter-5 1415393491 4576995 17 75 73 1
wissamitter-6 1415393491 4596295 16 70 70 0
wissamitter-7 1415393491 4574100 17 73 71 1
wissamitter-8 1415393491 4562520 17 69 67 0
wissamitter-9 1415393491 4606910 17 78 74 1
==== phase2 ====
machine-name timestamp hashrate front top bottom is_odd
wissamitter-10 1415396747 4591470 17 73 72 0
wissamitter-1 1415396747 4585680 16 73 69 1
wissamitter-2 1415396747 4533570 16 74 73 0
wissamitter-3 1415396747 4576030 16 72 68 1
wissamitter-4 1415396747 4558660 16 74 71 0
wissamitter-5 1415396747 4571205 16 68 66 1
wissamitter-6 1415396747 4596295 16 74 73 0
wissamitter-7 1415396747 4562520 17 68 68 1
wissamitter-8 1415396747 4568310 17 72 71 0
wissamitter-9 1415396747 4581820 16 72 69 1
==== Summaries ====
=== Evens ===
== phase1 ==
{'count': 5, 'front': 16.6, 'rate': 4581.241, 'top': 70.4, 'bottom': 68.4}
== phase2 ==
{'count': 5, 'front': 16.4, 'rate': 4569.661, 'top': 73.4, 'bottom': 72.0}
=== Odds ===
== phase1 ==
{'count': 5, 'front': 17.0, 'rate': 4586.838, 'top': 76.6, 'bottom': 73.4}
== phase2 ==
{'count': 5, 'front': 16.2, 'rate': 4575.451, 'top': 70.6, 'bottom': 68.0}
== High speed ==
{'count': 10, 'front': 16.4, 'rate': 4578.346, 'top': 70.5, 'bottom': 68.2}
== Low speed ==
{'count': 10, 'front': 16.7, 'rate': 4578.2495, 'top': 75.0, 'bottom': 72.7}
That's 4578.346 GH/s at 80, and 4578.2495 GH/s at 70. Conclusion: if your intake temps are around 16C, and you're using September SP30s, fan speeds around 70 or 80 seem to be equally good. Source code available upon request. |
|
|
|
If you are having weird behavior in SP3x : upgrade to 2.4.9. If all works well - don't.
Good day.
I take it you mean 2.5.9. |
|
|
|
what are your temps like. Even the SP30 looses close to 200TH it seems per 10deg temp rise
I've measured it at about 10 GH/s per 1C for SP30s, or around half what you indicate. I've measured this by taking a machine out of our racks (on around 14C air, IIRC) and sticking it into the coldest spot of our hot aisle (about 26C at the time, IIRC), and seeing about 120 GH/s drop in the hashrate. My memory is a little fuzzy on the exact numbers we got from that experiment, but it's accurate to +/- 20%. I've also done other measurements by looking at diurnal variation in performance on SP30s with similar results. SP10s are much more sensitive to temperature. They also vary in performance by about 10 GH/s per 1C, despite having less than 1/3 the total hashrate. I've heard that SP20s are more sensitive to temperature than SP30s, but I don't have any here to test. Preliminary observations with SP31s (as well as predictions based on what I know of semiconductor physics) suggest that they're more temperature sensitive than the SP30s, but I haven't done any systematic measurements yet. |
|
|
|
Anyone with SP31 getting more than 4.7Th/s?
We have 13 SP31s here at Toomim Bros Bitcoin Mining Concern right now. With 16C intake temps, we're currently seeing the lowest performer getting 4743 GH/s, highest 4915 GH/s, mean 4835 GH/s. Do this from the ssh console: echo "1' > /etc/mg_ignore_110_fcc If you have a lot of machines to configure, you can also scp it over. Example if your machines are 192.168.1.5 through 192.168.1.8: echo "1" > mg_ignore_110_fcc
for ip in 5 6 7 8; do scp mg_ignore_110_fcc root@192.168.1.$ip; echo "Finished 192.168.1.$ip"; done You will then want to change the power limit on the PSUs. In my experience, Murata PSUs can handle between 1330W and 1380W when run off of 235-245V in ambient temps no greater than 22C. Keep in mind that when Muratas hit their power limit, they sometimes lock up and need a power cycle to recover them, so you shouldn't increase the limits unless you're going to be nearby for several hours. I also recommend using starting voltages of 650 mV or 645 mV with SP31s, not the default 655 mV, since 655 is too close to the settled voltage for some SP31s. You may want to change the fan speeds as well. I'm not sure yet what the optimum is, but I suspect it's the setting that will eliminate thermal throttling on the loop 4 and 9 ASICs, but no higher. |
|
|
|
Damn! Feeling for those guys.. Surely you'd choose a less hot & humid spot for a DC?
But that's where the good beaches are. |
|
|
|
DC's will be thinking twice before allowing people to host this gear now, at least at unrealistic densities. Imports may receive greater scrutiny.
I'm not thinking twice about it. I bet the fire started in their electrical infrastructure -- transformer, distribution panel, etc -- not in a power supply or a miner. There's not much flammable material inside a miner. However, they may have installed a transformer near to a wood wall, then overloaded the transformer or wired it up without adequate fuse protection. |
|
|
|
They spent at least an hour last night disassembling, troubleshooting and fixing one of my miners.
One of the hashblades in his S2 was a little crooked, and its heatsink was touching some pins on the Beaglebone Black controller, which shorted them out. I used some electrical tape to insulate the Beaglebone, and that got his S2 running again. |
|
|
|
one my psu is dead (the guy who power cycle the rig said that the PSU threw spark out of the fan  ) There is no more support for this SP-10 now. Can SP-T tell me what's the PSU model I shall buy ? Can you send a fresh new one ? The SP10 uses a DS1050-3 power supply. However, I've found in the past that the DS1050-3 PSUs you can get on the market (e.g. ebay) do not work with the SP10, since there is some difference in how they supply 5V to the miner. The DS1200-3 used by the SP30 also works in the SP10. You can get those on ebay too, but the ebay ones also don't work in the SP10 because of the 5V issue. However, in the SP30, the 5V for the control board is shared between the top PSU and bottom PSU, and if either one correctly supplies 5V, then the machine works. As a result, what you can do (and what I've done in the past when we had a SP10 with a failed PSU, before we got spare PSUs from Spondoolies) is you can buy an ebay DS1200-3 PSU, stick it in an SP30, and then put the SP30's DS-1200-3 into the SP10. Spondoolies will probably be able to send you a fresh PSU, though. You'll get better support if you email support@spondoolies-tech.com instead of posting in the forum. |
|
|
|
If you want to host it with us (we have shelf space available in one week!), we should be able to provide power supplies for you for a reasonable price. We've modded server PSUs to work with Antminer S3s before. We've also got a few spare ATX PSUs. |
|
|
|
|
Show Posts
