[ANN] Spondoolies-Tech - carrier grade, data center ready mining rigs

[Marvell1]

Version 2.5.12 became the official, 2.5.6 removed and 2.5.13 is the latest experimental.
2.5.12 - different handling of BIST and Murata flow - improves stability.
2.5.13 - sets default (after factory reset) values to 1275/288 in SP3x/SP2x. Redirection bug fixed. Handling of ambient temperatures lower then 0c added.

 

So which firmware should I install on an Emerson PSU set I currently have 2.5.1 and its still unstable  not getting more than 3TH and asics keep failing with that loop error.

I rue the day I updated my firmware when I was happily hashing at around 4.5 TH for months.

Is there a process to roll back?

you can roll back contact Zivi there is a URL where you can download the firmware directly i.e the version you want , in my case I thnk it was too late some of my asics are permenatly damaged I think.

Is there any possibility of those with damage hardware to get coupons for discounts on the newley priced sp20s ?  I'm down around 250GH of hashing power now

[faraway]

I have had 39 SP30s running 2.5.12 so far for about 12 hours 16 minutes with no apparent issues. If I see no problems with 2.5.12 by next morning, I will probably move the rest of our machines over to it. Otherwise, I will probably move everything back to 2.4.23.

I've had a plethora of "disabled (bist cant start, stuck X)" errors on the remaining 100 or so SP30s we have running 2.5.6 in the mean time. It seems every few hours, I have to restart cgminer on 5% or 10% of our machines to get them back to full hashrate. Very annoying, but so far the issue has always been resolved by restarting minergate.

I've also had several reports of machines on 2.5.6 showing half hashrate at the pool even though I see full hashrate. These issues have always been restarted by a minergate restart.

I saw some much more serious problems on 2.5.7 during the 36 hours we had machines using that version. We saw about 10 machines get "Loop serial failed x" error messages (where x is a loop number, between 0 and 9 inclusive) and fail to start until one or sometimes two ASICs on that loop were disabled. After a few hours, many of these ASICs could be enabled again, but about 25% of the ones that failed are still non-operational. Currently, I think there are 9 ASICs which we lost on 2.5.7 which have not recovered.

It's possible that the potential ASIC-damaging bug in 2.5.7 might still be present in 2.5.12. I haven't seen any indications of it yet, but I also haven't had enough machines running on it for long enough to be certain. On 2.5.7, I saw the loop serial failed ASIC damage thing with about 5% of our machines per day. At that rate, there's only about a 63% chance that I would have seen it if it were present at the same rate in 2.5.12 with 39 machines running half a day.

It is now looking like the issues we've been seeing in our datacenter with the potential ASIC-damaging bug was not related to version 2.5.7 per se, but is instead temperature-related, and we only noticed it now because the temperatures in our DC only recently started getting really close to 0°C. Version 2.5.13 does not appear to fix these issues. Zvi is looking into it further.

The other two bugs were definitely firmware related, so you should still avoid 2.5.6 and 2.5.7.

Findings are still tentative and provisional, and investigation is ongoing.

Jtoomim, thank you for helping debug firmware issues we'll have problem recreating in Isarel
Also for keeping the miners in top condition.

Guy

The issue seems to be really temperature dependent; Indeed at low temperature (e.g. 5°C intake) some ASICs can't start and must be disabled. This is an hardware issue; anyway a good software fix may be to add an option to delay the activation of the problematic ASICs at the time, when the machine is at an acceptable temperature; And turn off these ASICs again when minergate restart to avoid machines crashes.

[wh00per]

.. or keep the windows closed in the winter  .. 15 dec C to 20 deg C is decent enough. One does not need to evacuate all the heat if they don't need to.

[grn]

new SP20 price! $795
with discount code and shipping of 2 units (3.4TH): 4.5 BTC

7* bitmain S3+ units (3.4 TH) with coupons and shipping this week = 4.7 BTC
4* Bitmain C1 (4TH) with watercooling and shipping this week = 5.3 BTC   (4.64 for 3.5TH)

wrong math

1x SP20 price! $795+shipping   1700GH/s
2x C1 price   $800+shipping    2000GH/s

Which one is more GH/s ?

Which one requires extra components not mentioned in the price?

Nice comeback, Bick

BOTH!

[Guy Corem]

... SPTech ARE doing a pretty damn good job ...

We're trying. Thank you.

[jtoomim]

It is now looking like the issues we've been seeing in our datacenter with the potential ASIC-damaging bug was not related to version 2.5.7 per se, but is instead temperature-related, and we only noticed it now because the temperatures in our DC only recently started getting really close to 0°C. Version 2.5.13 does not appear to fix these issues. Zvi is looking into it further.

It's looking more certain that what we've been seeing in our DC is a poor reaction of SP30s to temperatures below about 6°C. We haven't seen any issues with SP10s yet at these temperatures.

Interesting. Did you try NOT to evacuate the heat that miners produce, but instead direct large fans on a stack of miners, so heat is distributed in the warehouse?
This might be a naive question because you certainly have a large density of miners. I am curious because one of my miners is hosted in WA.

This is what we're doing.

I designed our system with the intent of using 100% fresh air -- air would go through our miners once then be exhausted outside. With the revelation that SP30s have problems below about 8°C, we've had to revise that system, as it's currently -4°C outside, with a forecast low for tonight of -11°C. Over the last few days, we have set up a system whereby we can control the amount of fresh air vs. recirculated air we get by turning on and off fans. It's a bit rough still, and produces hotspots when we recirculate too heavily, but it works.

The tricky thing is getting the air to mix where you want it to mix. Our hot air containment system was designed with the intent of having the interior be hot and low pressure. In order to get air to recirculate, we have to make it hot and high pressure. This means we get hot air leaking out at every hole or crack that we have in the containment. Unfortunately, the most common cracks and holes we have in the system are the rack mounting bars for the SP30s, which means we tend to get uneven air temperature within a single SP30. Air on the side tends to go through the SP30, then circle around and come back out the front by the mounting bars, and then go through the SP30 again. On the sides of the machine, the air will make several circuits through the SP30, but in the center, the air goes through only once. We're working on sealing off those little cracks. There are other problems too, like making sure that the eight corners of our pod all are exposed to roughly the same temperature, and making sure that our intake and exhaust air streams get mixed well.

Is there a process to roll back?

You can click on "Firmware Upgrade" then "Manual Selection" to access several firmware versions which Zvi currently thinks are decent. You can download all of the old firmware at http://storage.googleapis.com/spond_firmware/spon_2.X.YY.tar, where X and YY are the version numbers of the firmware you want to use.

[Marvell1]

confused a bit on the stat pages for the sp20 vs sp30 using 2.5.13


Statistics
 

Name          GH/s        5 min GH/s    Accepted shares Rejected shares Errors Utility Last Share
  sp20         1626.025 1765.03359    1197                     0                 0         45.78 08:42:25


as you can see the mining rate is stable at 1626 GH whereas the 5min gh/s stays from 1700 to somtmes 1800 GH

same thing on the SP30 the 5 min rate is conistantly higer sometimes I see like 4700 GH vs the actual of 4400.

which number is the real number ?

[jtoomim]

Name          GH/s        5 min GH/s    Accepted shares Rejected shares Errors Utility Last Share
  sp20         1626.025 1765.03359    1197                     0                 0         45.78 08:42:25

which number is the real number ?

The 1626 is calculated based on the clock speeds of the ASICs, and should be the actual number of calculations per second your ASICs are performing. Your machine only reports wins back to the pool server, or hashes with a value below the current difficulty setting for that share. The second value is an estimate of the number of hashes your machine must be doing based on the number of wins you have sent back to the server over the last 5 minutes. It's noisier and will bounce around a lot, but this is the number that the pool will reward you for.

Basically, the second number is accurate, but the first number is precise. Both numbers have their uses, which is why both numbers are displayed. They should be within about 20% of each other if your machine has been running for 5 minutes or more.

[jtoomim]

It's looking more certain that what we've been seeing in our DC is a poor reaction of SP30s to temperatures below about 6°C. We haven't seen any issues with SP10s yet at these temperatures.

Some machines work fine at low temperatures. I've seen a few operate just dandy at intake temperatures as low as 2°C. However, others will show problems at intake temperatures as high as 8°C.

When a machine has trouble at low temperatures, it's often specific ASICs that show the problem (often #2 and #17). It's not always the ASICs in the front that have issues, though it seems more common to be those. Once an ASIC has trouble at low temperatures, it sometimes has trouble even when warmed up. Sometimes it has to be disabled permanently; other times it can be enabled once the machine is warmer.

[faraway]

It's looking more certain that what we've been seeing in our DC is a poor reaction of SP30s to temperatures below about 6°C. We haven't seen any issues with SP10s yet at these temperatures.

Some machines work fine at low temperatures. I've seen a few operate just dandy at intake temperatures as low as 2°C. However, others will show problems at intake temperatures as high as 8°C.

When a machine has trouble at low temperatures, it's often specific ASICs that show the problem (often #2 and #17). It's not always the ASICs in the front that have issues, though it seems more common to be those. Once an ASIC has trouble at low temperatures, it sometimes has trouble even when warmed up. Sometimes it has to be disabled permanently; other times it can be enabled once the machine is warmer.

Indeed, but a hardware to be reliable, must handle 25°C variation without too much trouble.Here at low temperature some ASICs can't start. It will be a good idea to add a 'delay' option for the troublesome ASICs where they are turn off during the cold start and enabled later. Adding an option to change the Fan speed according to the intake temperature, will be a good too idea too. (I suspect this issue to trigger some machines crashes when minergate restart).

[Guy Corem]

It's looking more certain that what we've been seeing in our DC is a poor reaction of SP30s to temperatures below about 6°C. We haven't seen any issues with SP10s yet at these temperatures.

Some machines work fine at low temperatures. I've seen a few operate just dandy at intake temperatures as low as 2°C. However, others will show problems at intake temperatures as high as 8°C.

When a machine has trouble at low temperatures, it's often specific ASICs that show the problem (often #2 and #17). It's not always the ASICs in the front that have issues, though it seems more common to be those. Once an ASIC has trouble at low temperatures, it sometimes has trouble even when warmed up. Sometimes it has to be disabled permanently; other times it can be enabled once the machine is warmer.

Indeed, but a hardware to be reliable, must handle 25°C variation without too much trouble.Here at low temperature some ASICs can't start. It will be a good idea to add a 'delay' option for the troublesome ASICs where they are turn off during the cold start and enabled later. Adding an option to change the Fan speed according to the intake temperature, will be a good too idea too. (I suspect this issue to trigger some machines crashes when minergate restart).

We're investigating.

[Marvell1]

It's looking more certain that what we've been seeing in our DC is a poor reaction of SP30s to temperatures below about 6°C. We haven't seen any issues with SP10s yet at these temperatures.

Some machines work fine at low temperatures. I've seen a few operate just dandy at intake temperatures as low as 2°C. However, others will show problems at intake temperatures as high as 8°C.

When a machine has trouble at low temperatures, it's often specific ASICs that show the problem (often #2 and #17). It's not always the ASICs in the front that have issues, though it seems more common to be those. Once an ASIC has trouble at low temperatures, it sometimes has trouble even when warmed up. Sometimes it has to be disabled permanently; other times it can be enabled once the machine is warmer.

Indeed, but a hardware to be reliable, must handle 25°C variation without too much trouble.Here at low temperature some ASICs can't start. It will be a good idea to add a 'delay' option for the troublesome ASICs where they are turn off during the cold start and enabled later. Adding an option to change the Fan speed according to the intake temperature, will be a good too idea too. (I suspect this issue to trigger some machines crashes when minergate restart).

We're investigating.

I  see the same issues at higher temps : 24c to 30c , I disabled the troubled asics and still every few hours when I check the machines half the Asics will report failed , sometimes all with the machine not even mining at all.

Its a deeper issue with the firmware than just temps I think.  Getting really frustrating at this point basically tried everything to keep it stable but nothing seems to work.  I have to bounce it every few hours but I don't have time to babysit it I have other stuff to do.

[Biffa]

It's looking more certain that what we've been seeing in our DC is a poor reaction of SP30s to temperatures below about 6°C. We haven't seen any issues with SP10s yet at these temperatures.

Some machines work fine at low temperatures. I've seen a few operate just dandy at intake temperatures as low as 2°C. However, others will show problems at intake temperatures as high as 8°C.

When a machine has trouble at low temperatures, it's often specific ASICs that show the problem (often #2 and #17). It's not always the ASICs in the front that have issues, though it seems more common to be those. Once an ASIC has trouble at low temperatures, it sometimes has trouble even when warmed up. Sometimes it has to be disabled permanently; other times it can be enabled once the machine is warmer.

Indeed, but a hardware to be reliable, must handle 25°C variation without too much trouble.Here at low temperature some ASICs can't start. It will be a good idea to add a 'delay' option for the troublesome ASICs where they are turn off during the cold start and enabled later. Adding an option to change the Fan speed according to the intake temperature, will be a good too idea too. (I suspect this issue to trigger some machines crashes when minergate restart).

We're investigating.

I  see the same issues at higher temps : 24c to 30c , I disabled the troubled asics and still every few hours when I check the machines half the Asics will report failed , sometimes all with the machine not even mining at all.

Its a deeper issue with the firmware than just temps I think.  Getting really frustrating at this point basically tried everything to keep it stable but nothing seems to work.  I have to bounce it every few hours but I don't have time to babysit it I have other stuff to do.

Maybe you just have a dodgy one. They happen even in the best of products. My SP20 is running at near 40°C on the intake with no disabling of ASICs

[zvisha]

It's looking more certain that what we've been seeing in our DC is a poor reaction of SP30s to temperatures below about 6°C. We haven't seen any issues with SP10s yet at these temperatures.

Some machines work fine at low temperatures. I've seen a few operate just dandy at intake temperatures as low as 2°C. However, others will show problems at intake temperatures as high as 8°C.

When a machine has trouble at low temperatures, it's often specific ASICs that show the problem (often #2 and #17). It's not always the ASICs in the front that have issues, though it seems more common to be those. Once an ASIC has trouble at low temperatures, it sometimes has trouble even when warmed up. Sometimes it has to be disabled permanently; other times it can be enabled once the machine is warmer.

Indeed, but a hardware to be reliable, must handle 25°C variation without too much trouble.Here at low temperature some ASICs can't start. It will be a good idea to add a 'delay' option for the troublesome ASICs where they are turn off during the cold start and enabled later. Adding an option to change the Fan speed according to the intake temperature, will be a good too idea too. (I suspect this issue to trigger some machines crashes when minergate restart).

We're investigating.

I  see the same issues at higher temps : 24c to 30c , I disabled the troubled asics and still every few hours when I check the machines half the Asics will report failed , sometimes all with the machine not even mining at all.

Its a deeper issue with the firmware than just temps I think.  Getting really frustrating at this point basically tried everything to keep it stable but nothing seems to work.  I have to bounce it every few hours but I don't have time to babysit it I have other stuff to do.

If you have a bad unit you can use your warranty to replace it.

[Marvell1]

its an End of August sp30 machine not sure if it is still under warranty what is the procedure for replacing it ?  Is it via email or via
the website.

thanks

[Guy Corem]

its an End of August sp30 machine not sure if it is still under warranty what is the procedure for replacing it ?  Is it via email or via
the website.

thanks

Contact support@ to approve RMA

[Marvell1]

its an End of August sp30 machine not sure if it is still under warranty what is the procedure for replacing it ?  Is it via email or via
the website.

thanks

Contact support@ to approve RMA

Done and thanks, hope this gets resloved soon, the SP30 is part of my heating solution in the home and i'm in the progress of purchasing a few more sp20s for garage heating as well.  The Sp20s do not seem to generate enough heat though for my purposes even with the fan throttled to 60%

[zvisha]

its an End of August sp30 machine not sure if it is still under warranty what is the procedure for replacing it ?  Is it via email or via
the website.

thanks

Skype me at zvi_shteingart

[Marvell1]

its an End of August sp30 machine not sure if it is still under warranty what is the procedure for replacing it ?  Is it via email or via
the website.

thanks

Skype me at zvi_shteingart

I'm at my 9-5 right now no access to the unit which is down right now , we skyped last week when I initally contacted you if you remeber we had the issue with trying to switch the firmware where the dropdown of previous versions did not show up.  You provided me the link to download the firmware and I was able to downgrade but
the issues still persisted.  I can ping you again in a few hours if you want to take a look at the unit.

thanks

[opentoe]

I finally saw some large photos of the SP20, looks pretty nice inside, clean and neat. Does that have just one fan? Is it very loud the other SP models? This unit seems to be made for the home, so my dad is asking me about if it is loud or not. Is it tolerable to be in the same room with it and have a phone conversation?