Stay away from ASICSPACE

[ThePhwner]

You're also better off sticking with what you know and not arguing with Jared. I wouldn't put his working knowledge of creating mines too far above the people he left behind who cooked your units, and I think even you can see he's here more to argue than to solve anything. Circular logic and speculation, are exactly that. Good for you for having the balls to stand up to these folks. Of course the twits here are going to poke all kinds of holes in what you're doing. Fair point though, you won't see a criminal prosecution out of this, so go for civil. Good luck.

[CryptoCoin2015]

You're also better off sticking with what you know and not arguing with Jared. I wouldn't put his working knowledge of creating mines too far above the people he left behind who cooked your units, and I think even you can see he's here more to argue than to solve anything. Circular logic and speculation, are exactly that. Good for you for having the balls to stand up to these folks. Of course the twits here are going to poke all kinds of holes in what you're doing. Fair point though, you won't see a criminal prosecution out of this, so go for civil. Good luck.

Thanks for the advice and the heads up !

[SebastianJu]

You're also better off sticking with what you know and not arguing with Jared. I wouldn't put his working knowledge of creating mines too far above the people he left behind who cooked your units, and I think even you can see he's here more to argue than to solve anything. Circular logic and speculation, are exactly that. Good for you for having the balls to stand up to these folks. Of course the twits here are going to poke all kinds of holes in what you're doing. Fair point though, you won't see a criminal prosecution out of this, so go for civil. Good luck.

Thanks for the advice and the heads up !

Did Toomim Brothers already give a timeline when first, or the complete, results will be available?

[jtoomim]

Did Toomim Brothers already give a timeline when first, or the complete, results will be available?

We essentially completed our investigation of the damaged units a week ago, but now we're busy trying to repair them (and the PSUs that ASICSPACE included with them) and to install another 100 S5s, so we haven't had much time to comment here. We kept CryptoCoin2015 updated about the progress of our investigation as we performed it. Here is some information:

Electrical problems

About 40% of the hashboards are damaged and defective, and do not hash at all. My initial count was 89 damaged hashboards out of 200, but I think that was a slight overestimate because I made that count on a weekend and misinterpreted a pile that one of my employees had made. I'll have to make another count soon, but the total is probably closer to 70 damaged hashboards. There may be another half-dozen hashboards that still hash with some dead ASICs. We did not observe any otherwise dead hashboards working if we reduced the clockspeed.

About 30% of the damaged hashboards had visibly burned or popped SMT G337 2V capacitors. This means about 12% of all of the hashboards. Many had several capacitors damaged on a single board. We didn't count all of these; this is just an estimate based on a small random sample.

The resistance across the 12V and Gnd pins for a disconnected hashboard should be around 56 to 58 ohms. This translates to about 4 ohms per stage. (I think it's actually around 4.5 ohms per stage with a parallel resistance of ~330 ohms from 12V straight to Gnd, but that doesn't matter much for this.) All of the working hashboards were in this 56 to 58 ohm range. All of the broken hashboards had lower resistances than this. About 30% had resistances close to 53 ohms, suggesting that they had one bad stage. The rest had resistances below 50 ohms, and about half of them were 30 or below. This suggests several bad stages is typical for the failed hashboards.

We measured the voltage across each stage for a few different hashboards. Here is one such:

15-14:   0.990
14-13:   0.953
13-12:   0.055
12-11:   1.023
11-10:   0.999
10-09:   0.780
09-08:   0.360
08-07:   0.994
07-06:   0.948
06-05:   1.054
05-04:   0.477
04-03:   1.017
03-02:   0.995
02-01:   1.061
01-gn:   0.478

All of the working stages have nearly equal voltages. On this hashboard, those voltages are around 0.980 V +/- 0.070 V. On a good hashboard, they are around 0.800 V, +/- 0.070 V. The voltage abnormalities indicate failed stages. I don't have the resistance measurements recorded, but they were similar, with powered-on voltage per stage corresponding closely with powered-off measured resistance. Several stages were measured at 0.8 ohms, which is the same resistance I measure when I short the leads of the multimeter together. (Multimeters typically are not good at measuring very small resistances, due to the additional resistance of the test probes and leads.)

All of the visibly damaged capacitors were on stages with abnormal stage voltage and stage resistance. Not all stages with abnormal voltage or resistance had visibly damaged capacitors. Removing and replacing damaged capacitors did not fix the abnormal voltage or resistance. (I did not test to see if they hashed after replacing the caps; I just assumed they didn't.) I also tested three severely damaged capacitors after removing them, and despite the crazy physical damage, they still showed the correct electrical characteristics (320 to 370 µF capacitance, DC resistance > 10 Mohm). This suggests that the functionally important damage was to a component other than the capacitors. The most likely candidate is the ASICs themselves.

The G337 2V capacitors are probably aluminum polymer capacitors. Those are typically rated for 105°C operating temperatures. This is lower than the 125°C silicon operating temperature for most mining ASICs and most other discrete power components. I think it's possible that a cooling failure could have caused the capacitors to explode before or at the same time as damage to the ASICs occurred, even though the capacitors were not themselves generating any heat.

We overnighted four bad hashboards to Bitmain Warranty in Denver, CO. I haven't heard back from them about those specifically, but my guess is that they discarded them as unrepairable.

Plastic shield deformations

Most of the S5s we received from ASICSPACE have some deformation of the plastic air guide shield panels. The severity varies widely. There seems to be a correlation between the deformation of the plastic shields and whether the hashboard is broken. There may also be a correlation between deformation and presence of damaged capacitors. I haven't made a close study of that, though. Mostly, all I know is that the shields we took off when investigating broken hashboards tend to be worse than the shields that we left on.

It appears that the worst point on the plastic shields is on the "tail" end (exhaust side) of the miners. The deformation is typically greatest (i.e. shortest radius of curvature) in between the screws. The top edge (where the PCIE connectors and control board are) is also heavily deformed on many machines. In all cases, the deformation causes the shields to bend away from the case of the Ant. In a few cases, the plastic has a wavy appearance in between the screw holes up to approximately 1 cm in from the edge, suggesting that the plastic there had been stretched or elongated and had caused the surface to take a non-linear path between the two fixed screw points. None of the S5s that have been in the Toom.im facility since January show any hints of deformation like this. The only machines like this that we've seen were in ASICSPACE during early Aprli.

Most of the shields have no scuff marks or indications of physical trauma. The deformations are smooth, with no creases. I do not think these deformations were caused by contact with a solid object. Deformation due to air pressure would explain these deformations in every case I've looked at, as long as the plastic were soft enough. I'll describe how I think that happened after I mention some of what I've seen about the environment in which they were operating.

First-person observations

When I visited to pick up the S5s, ASICSPACE's cold aisle was negatively pressurized relative to the outside air. The hot aisle was positively pressurized relative both to outside and to the cold aisle. As a result, the cold aisle was also by no means "cold". Near the cold air supply ducts, the air felt like it was about 15°C. In most of the rest of the cold aisle, it felt like it was 35°C. When I first arrived, I walked past a gap in the cold aisle containment where an Antminer S4 had previously been. The velocity and volume of airflow through that hole was comparable to the airflow coming out of their cold air supply duct outlet, except a lot hotter. I think it felt like about 500 to 1000 CFM through that hole. As we removed S5s, and more holes appeared in their cold air containment, the velocity through each hole decreased substantially. ASICSPACE had noticed this pressure difference, and as a way of mitigating it had set up 9 air ducts (approx. 0.5 m in diameter each) going from their garage door to their cold aisles to serve as supplementary air intake. Note that in proper cold aisle containment design, the cold aisle should be positively pressurized, so ducts like this would normally let air out (and thus normally would not be used). I also spoke with Robert about this, and he was (fortunately) aware of this problem. The ducts looked like they had been hastily added, likely within the previous week. While I was there, their network was experiencing severe problems, causing a large proportion of their machines to stop hashing (and thus not produce heat). It was also not a very hot day, and it was evening when I arrived, making it about 11°C cooler than the daytime high the day before. I can only imagine what their facility was like during the weeks prior.

I did not see any significant exhaust fans installed at ASICSPACE to remove hot air. There were some small fans mounted above the hot aisle, but they were unducted, small, and not very numerous, so I guess they were not significant. There may or may not have been some exhaust fans on the roof of ASICSPACE. From what I've heard through other channels, they chose not to install exhaust fans, and were instead relying on the stack effect to move heat up through their tall (30m?) building and exhaust through natural convection at the top. Unfortunately, the stack effect relies on the interior of the building being hotter than the outside temperature, so when the outside temperatures increase to 30°C, their interior temps would have to rise too. Since they're pre-cooling with evaporative coolers, this means that the air inside their building would have to recirculate several times until it was enough hotter than the outside air for the stack effect to blow it out. However, if they have 300,000 to 800,000 cfm of exhaust fans that I didn't see, this paragraph is irrelevant. (Edit 5/6/2015: We visited ASICSPACE again today, and Robert showed me their exhaust system. They do not have exhaust fans, and are relying on the stack effect plus (in principle) positive pressure from their intake fans.)

The containment sealing system (which looked pretty tight, to be honest --  kudos to ASICSPACE for that) sealed off the exhaust end of each miner, with a gap between the case and the edge of the sealing panel on the order of 1 or 2 cm.

The S5s were installed next to pairs of S4s along both sides of the cold aisle on each of the shelves. This indicates that the S4s and S5s were competing with each other for airflow. The S4 is a sealed tunnel with 4 fans arranged in a push-pull configuration -- i.e., 2 in parallel by 2 in series. Placing fans in series multiplies the amount of static pressure they can produce, and also allows them to maintain their airflow quite well when working against a significant positive pressure gradient, but does very little for the airflow when working in a neutral-pressure regime. This means that the S4 fans were able to to pressurize the hot aisle quite effectively. The S5, on the other hand, is an open semi-tunnel configuration with a single fan in push configuration. The semi-tunnel has large gaps near the exhaust end through which air can escape out the top, as well as small gaps on the bottom.

The power supplies which ASICSPACE had obtained for these S5s were the DPS-800GB using the Gigampz breakout boards. These boards are miswired to connect pin 30 (voltage adjust pin) to the 12V rail, which increases their output voltage to around 12.80 V (no load). Power consumption for the S5s' BM1384 (as with most ASICs) is proportional to frequency times voltage squared, so the 6.6% higher voltage should result in 13.8% more power consumed and heat generated.

What I think happened

As the single S5 fans were unable to compete with the doubled S4 fans in terms of pressure output, the airflow from the S5 fans would instead curve out through the gaps in the S5 pseudo-tunnel. Meanwhile, the positive pressure from the hot aisle may have been strong enough to cause airflow to go in reverse from the hot aisle into the "exhaust" port of the S5, and then out the top of the miner. This retrograde flow was likely strongest in the side areas, in between the hashboards and the plastic shields. These two effects caused the interior and side spaces of the S5s to get very hot and positively pressurized. The heat caused the plastic sides to become soft and "plastic" (in the non-elastic sense of the word). The positive pressure then stretched and deformed the heat shields away from the mounting screws. The deformation caused a small gap to appear at the tail end of the shields in between the two screws. This gap allowed for a very short path -- approximately 3 cm -- for the air to go from the hot aisle to the cold aisle, by entering the tail of the miner through the side slots, making a roughly 150° turn to curve around the steel frame (and partially bouncing off the plastic shield), and then exiting backwards/sideways through the tail/side gaps in between the screws. This air was the closest to the hot aisle, and thus the hottest and highest pressure, and consequently caused the greatest deformation, enlarging these gaps considerably over time. Additional airflow that came in through the side slots would have passed out the top of the miner. Though this airflow was greater, the size of the gaps on the top were much larger, causing lower pressure differences, which made the radius of curvature of the deformation in that area smaller.

The intersection of the anterograde and retrograde airflows near the center of the miner caused airflow there to be relatively stagnant. This, combined with the higher temperature of the retrograde flow, would have caused very high temperatures about 3 to 10 cm away from the miner's exhaust port.

The Antminer S5 has a bug (linked elsewhere on this page) in which the miner's fans will stop immediately when cgminer dies (or when the network is disconnected), but the hashboards will continue to generate heat for several minutes. Given that ASICSPACE was having frequent and persistent networking problems during this time, I expect that to be a contributing factor in this case. However, I think that is at most a contributing factor, since many people have reported that bug in the absence of heat damage, and I haven't seen any other reports attributing actual heat damage to that bug.

Based on what I saw when I visited and what I know had been changed recently at ASICSPACE, I estimate cold aisle temps were likely 20°C hotter during early/mid April at ASICSPACE, and likely reached about 55°C. I have also heard people allege to have seen 57°C intake temps on their miners at ASICSPACE at one point (I think that was March), although I can't mention the source for that right now so you should treat it as a rumor. I have also seen screenshot evidence from a KNCMiner device indicating temperatures in the same ballpark. With mostly S4s and a few SP35s, plus with the fans working against positive pressure and consequently having reduced forward airflow, their delta-T might have been around 10°C, or possibly a little higher. I thus estimate their hot aisle temps at around 65°C. The air passing from the hot aisle through the S5 heatsinks would have gotten even hotter before hitting the side panels; perhaps 15°C hotter. Many plastics have glass transition temperatures around 80 to 120°C. ABS, for example, is 105°C. I thus think it is plausible that the plastic side panels were heated close enough to their glass transition temperature that air pressure differences caused them to permanently and plastically deform. Having a 55°C cold aisle with strong positive pressure in the hot aisle would also explain why the ASICs and capacitors in the S5s would have burned out in such large numbers, especially if the 80°C protection was bypassed either due to a Bitmain bug or due to an attempt to get the S5s to hash despite the poor working conditions.

Note: As the head of Toomim Bros, I am a competitor of ASICSPACE, and clearly have a conflict of interest in this case. Apply salt liberally before hashing.

[JaredR26]

Extremely thorough explanation Toomim.  I'm impressed, hats off to you.

Did Toomim Brothers already give a timeline when first, or the complete, results will be available?

We essentially completed our investigation of the damaged units a week ago, but now we're busy trying to repair them (and the PSUs that ASICSPACE included with them) and to install another 100 S5s, so we haven't had much time to comment here. We kept CryptoCoin2015 updated about the progress of our investigation as we performed it. Here is some information:

Electrical problems

About 40% of the hashboards are damaged and defective, and do not hash at all. My initial count was 89 damaged hashboards out of 200, but I think that was a slight overestimate because I made that count on a weekend and misinterpreted a pile that one of my employees had made. I'll have to make another count soon, but the total is probably closer to 70 damaged hashboards. There may be another half-dozen hashboards that still hash with some dead ASICs. We did not observe any otherwise dead hashboards working if we reduced the clockspeed.

About 30% of the damaged hashboards had visibly burned or popped SMT G337 2V capacitors. This means about 12% of all of the hashboards. Many had several capacitors damaged on a single board. We didn't count all of these; this is just an estimate based on a small random sample.

The resistance across the 12V and Gnd pins for a disconnected hashboard should be around 56 to 58 ohms. This translates to about 4 ohms per stage. (I think it's actually around 4.5 ohms per stage with a parallel resistance of ~330 ohms from 12V straight to Gnd, but that doesn't matter much for this.) All of the working hashboards were in this 56 to 58 ohm range. All of the broken hashboards had lower resistances than this. About 30% had resistances close to 53 ohms, suggesting that they had one bad stage. The rest had resistances below 50 ohms, and about half of them were 30 or below. This suggests several bad stages is typical for the failed hashboards.

We measured the voltage across each stage for a few different hashboards. Here is one such:

15-14:   0.990
14-13:   0.953
13-12:   0.055
12-11:   1.023
11-10:   0.999
10-09:   0.780
09-08:   0.360
08-07:   0.994
07-06:   0.948
06-05:   1.054
05-04:   0.477
04-03:   1.017
03-02:   0.995
02-01:   1.061
01-gn:   0.478

All of the working stages have nearly equal voltages. On this hashboard, those voltages are around 0.980 V +/- 0.070 V. On a good hashboard, they are around 0.800 V, +/- 0.070 V. The voltage abnormalities indicate failed stages. I don't have the resistance measurements recorded, but they were similar, with powered-on voltage per stage corresponding closely with powered-off measured resistance. Several stages were measured at 0.8 ohms, which is the same resistance I measure when I short the leads of the multimeter together. (Multimeters typically are not good at measuring very small resistances, due to the additional resistance of the test probes and leads.)

All of the visibly damaged capacitors were on stages with abnormal stage voltage and stage resistance. Not all stages with abnormal voltage or resistance had visibly damaged capacitors. Removing and replacing damaged capacitors did not fix the abnormal voltage or resistance. (I did not test to see if they hashed after replacing the caps; I just assumed they didn't.) I also tested three severely damaged capacitors after removing them, and despite the crazy physical damage, they still showed the correct electrical characteristics (320 to 370 µF capacitance, DC resistance > 10 Mohm). This suggests that the functionally important damage was to a component other than the capacitors. The most likely candidate is the ASICs themselves.

The G337 2V capacitors are probably aluminum polymer capacitors. Those are typically rated for 105°C operating temperatures. This is lower than the 125°C silicon operating temperature for most mining ASICs and most other discrete power components. I think it's possible that a cooling failure could have caused the capacitors to explode before or at the same time as damage to the ASICs occurred, even though the capacitors were not themselves generating any heat.

We overnighted four bad hashboards to Bitmain Warranty in Denver, CO. I haven't heard back from them about those specifically, but my guess is that they discarded them as unrepairable.

Plastic shield deformations

Most of the S5s we received from ASICSPACE have some deformation of the plastic air guide shield panels. The severity varies widely. There seems to be a correlation between the deformation of the plastic shields and whether the hashboard is broken. There may also be a correlation between deformation and presence of damaged capacitors. I haven't made a close study of that, though. Mostly, all I know is that the shields we took off when investigating broken hashboards tend to be worse than the shields that we left on.

It appears that the worst point on the plastic shields is on the "tail" end (exhaust side) of the miners. The deformation is typically greatest (i.e. shortest radius of curvature) in between the screws. The top edge (where the PCIE connectors and control board are) is also heavily deformed on many machines. In all cases, the deformation causes the shields to bend away from the case of the Ant. In a few cases, the plastic has a wavy appearance in between the screw holes up to approximately 1 cm in from the edge, suggesting that the plastic there had been stretched or elongated and had caused the surface to take a non-linear path between the two fixed screw points. None of the S5s that have been in the Toom.im facility since January show any hints of deformation like this. The only machines like this that we've seen were in ASICSPACE during early Aprli.

Most of the shields have no scuff marks or indications of physical trauma. The deformations are smooth, with no creases. I do not think these deformations were caused by contact with a solid object. Deformation due to air pressure would explain these deformations in every case I've looked at, as long as the plastic were soft enough. I'll describe how I think that happened after I mention some of what I've seen about the environment in which they were operating.

First-person observations

When I visited to pick up the S5s, ASICSPACE's cold aisle was negatively pressurized relative to the outside air. The hot aisle was positively pressurized relative both to outside and to the cold aisle. As a result, the cold aisle was also by no means "cold". Near the cold air supply ducts, the air felt like it was about 15°C. In most of the rest of the cold aisle, it felt like it was 35°C. When I first arrived, I walked past a gap in the cold aisle containment where an Antminer S4 had previously been. The velocity and volume of airflow through that hole was comparable to the airflow coming out of their cold air supply duct outlet, except a lot hotter. I think it felt like about 500 to 1000 CFM through that hole. As we removed S5s, and more holes appeared in their cold air containment, the velocity through each hole decreased substantially. ASICSPACE had noticed this pressure difference, and as a way of mitigating it had set up 9 air ducts (approx. 0.5 m in diameter each) going from their garage door to their cold aisles to serve as supplementary air intake. Note that in proper cold aisle containment design, the cold aisle should be positively pressurized, so ducts like this would normally let air out (and thus normally would not be used). I also spoke with Robert about this, and he was (fortunately) aware of this problem. The ducts looked like they had been hastily added, likely within the previous week. While I was there, their network was experiencing severe problems, causing a large proportion of their machines to stop hashing (and thus not produce heat). It was also not a very hot day, and it was evening when I arrived, making it about 11°C cooler than the daytime high the day before. I can only imagine what their facility was like during the weeks prior.

Fits; Temp around your time appears to have been around ~61F, high temp was ~80F.  A few assumptions went into that(like which day).

http://mesowest.utah.edu/cgi-bin/droman/mesomap.cgi?state=WA&rawsflag=3

I did not see any significant exhaust fans installed at ASICSPACE to remove hot air. There were some small fans mounted above the hot aisle, but they were unducted, small, and not very numerous, so I guess they were not significant. There may or may not have been some exhaust fans on the roof of ASICSPACE. From what I've heard through other channels, they chose not to install exhaust fans, and were instead relying on the stack effect to move heat up through their tall (30m?) building and exhaust through natural convection at the top. Unfortunately, the stack effect relies on the interior of the building being hotter than the outside temperature, so when the outside temperatures increase to 30°C, their interior temps would have to rise too. Since they're pre-cooling with evaporative coolers, this means that the air inside their building would have to recirculate several times until it was enough hotter than the outside air for the stack effect to blow it out. However, if they have 300,000 to 800,000 cfm of exhaust fans that I didn't see, this paragraph is irrelevant.

The containment sealing system (which looked pretty tight, to be honest --  kudos to ASICSPACE for that) sealed off the exhaust end of each miner, with a gap between the case and the edge of the sealing panel on the order of 1 or 2 cm.

The S5s were installed next to pairs of S4s along both sides of the cold aisle on each of the shelves.

This piece I did not know until after I had posted my previous posts, and singly explains the failure.  In a worst case scenario, this pressure/counterflow would have caused the S5 fans to basically stop working, as you described.

This indicates that the S4s and S5s were competing with each other for airflow. The S4 is a sealed tunnel with 4 fans arranged in a push-pull configuration -- i.e., 2 in parallel by 2 in series. Placing fans in series multiplies the amount of static pressure they can produce, and also allows them to maintain their airflow quite well when working against a significant positive pressure gradient, but does very little for the airflow when working in a neutral-pressure regime. This means that the S4 fans were able to to pressurize the hot aisle quite effectively. The S5, on the other hand, is an open semi-tunnel configuration with a single fan in push configuration. The semi-tunnel has large gaps near the exhaust end through which air can escape out the top, as well as small gaps on the bottom.

What I think happened

As the single S5 fans were unable to compete with the doubled S4 fans in terms of pressure output, the airflow from the S5 fans would instead curve out through the gaps in the S5 pseudo-tunnel. Meanwhile, the positive pressure from the hot aisle may have been strong enough to cause airflow to go in reverse from the hot aisle into the "exhaust" port of the S5, and then out the top of the miner. This retrograde flow was likely strongest in the side areas, in between the hashboards and the plastic shields.  These two effects caused the interior and side spaces of the S5s to get very hot and positively pressurized. The heat caused the plastic sides to become soft and "plastic" (in the non-elastic sense of the word). The positive pressure then stretched and deformed the heat shields away from the mounting screws. The deformation caused a small gap to appear at the tail end of the shields in between the two screws. This gap allowed for a very short path -- approximately 3 cm -- for the air to go from the hot aisle to the cold aisle, by entering the tail of the miner through the side slots, making a roughly 150° turn to curve around the steel frame (and partially bouncing off the plastic shield), and then exiting backwards/sideways through the tail/side gaps in between the screws.

Fits with what I know except the plastic shields part.  I legally can't state why, but it doesn't make much difference.

This air was the closest to the hot aisle, and thus the hottest and highest pressure, and consequently caused the greatest deformation, enlarging these gaps considerably over time. Additional airflow that came in through the side slots would have passed out the top of the miner. Though this airflow was greater, the size of the gaps on the top were much larger, causing lower pressure differences, which made the radius of curvature of the deformation in that area smaller.

The intersection of the anterograde and retrograde airflows near the center of the miner caused airflow there to be relatively stagnant. This, combined with the higher temperature of the retrograde flow, would have caused very high temperatures about 3 to 10 cm away from the miner's exhaust port.

The Antminer S5 has a bug (linked elsewhere on this page) in which the miner's fans will stop immediately when cgminer dies (or when the network is disconnected), but the hashboards will continue to generate heat for several minutes. Given that ASICSPACE was having frequent and persistent networking problems during this time, I expect that to be a contributing factor in this case. However, I think that is at most a contributing factor, since many people have reported that bug in the absence of heat damage, and I haven't seen any other reports attributing actual heat damage to that bug.

Based on what I saw when I visited and what I know had been changed recently at ASICSPACE, I estimate cold aisle temps were likely 20°C hotter during early/mid April at ASICSPACE, and likely reached about 55°C.

Temperature rise is relatively linear even if it doesn't feel like it and the damage caused is not linear.  You noticed around ~95F in the cold aisle @ ~61F outside.  On a 81F day, that would be ~115F(~46C).  Maybe a bit more, but the damage to these miners(and why other miners were not similarly damaged) was primarily because of the airflow around them in relation to the S4's you mentioned, not primarily because of the intake temperatures.

I have also heard people allege to have seen 57°C intake temps on their miners at ASICSPACE at one point (I think that was March), although I can't mention the source for that right now so you should treat it as a rumor. I have also seen screenshot evidence from a KNCMiner device indicating temperatures in the same ballpark.  With mostly S4s and a few SP35s, plus with the fans working against positive pressure and consequently having reduced forward airflow, their delta-T might have been around 10°C, or possibly a little higher. I thus estimate their hot aisle temps at around 65°C. The air passing from the hot aisle through the S5 heatsinks would have gotten even hotter before hitting the side panels; perhaps 15°C hotter. Many plastics have glass transition temperatures around 80 to 120°C. ABS, for example, is 105°C.  I thus think it is plausible that the plastic side panels were heated close enough to their glass transition temperature that air pressure differences caused them to permanently and plastically deform.

Polyethylene has a glass transition temperature of -80 to -120C.  Polyethylene is malleable at room temperature and cheap to mass produce, just like the S5 plastic, which makes it a reasonable assumption unless Bitmain can confirm.  Regardless, the temperatures that caused the deformation & damage would have largely been due to the "stuck" stagnant air that you described, not a direct result of the hot aisle temperatures. (i.e., primary cause = pressure)  Other minor information I now know that I can't share agrees with that.

Having a 55°C cold aisle with strong positive pressure in the hot aisle would also explain why the ASICs and capacitors in the S5s would have burned out in such large numbers, especially if the 80°C protection was bypassed either due to a Bitmain bug or due to an attempt to get the S5s to hash despite the poor working conditions.

[jtoomim]

Temperature rise is relatively linear even if it doesn't feel like it and the damage caused is not linear.  You noticed around ~95F in the cold aisle @ ~61F outside.  On a 81F day, that would be ~115F(~46C).  Maybe a bit more, but the damage to these miners(and why other miners were not similarly damaged) was primarily because of the airflow around them in relation to the S4's you mentioned, not primarily because of the intake temperatures.

I observed cold aisle temps around 35°C. My estimate of cold aisle temps around 55°C in the weeks before I visited was based on several factors:

1. There were network problems when I visited, resulting in less heat generated. It looked like about 50% of ASICSPACE's miners were off while I was there due to the network problems. If the 50% estimate is correct, then the delta T versus outside might be sqrt(2) to 2 times higher, depending on how much higher inside temperatures assist the stack effect. As the cold aisle temps were already 20°C hotter than outside, this alone might be enough to produce cold aisle temps of 55°C even on a somewhat cool day. However, my 50% estimate could be off. Also, many of the machines were S4s, which may also produce heat while the network is disconnected. For my estimate, I think I was generous and presumed that the network problems reduced temps by 8°C.

2. The nine flexible ducts supplementing the cold aisle with outside air at ambient pressure looked to be hastily added. I presumed they were added within the last week before my visit, and were not present when the damage occurred. I estimate they added about 20% to the airflow into the cold aisles. Roughly speaking, that should have decreased the cold-aisle/outdoor delta T by 20%, which would be about 4°C on the day I visited, or possibly as much as 8°C on other days. There may have been other similar quick fixes that I did not notice. I used 2°C in my quick estimate.

3. When I visited (6pm to 11pm), the outside temp was 16°C or lower. The high the day before was 26°C. That's 10°C. I think there was a day or two in the week or two prior that was even hotter.

8°C + 2°C + 10°C = 20°C
35°C + 20°C = 55°C

Other effects: as the temperatures rose, many machines may have turned off. On the other hand, reliance on the stack effect may have worsened the effects of a high outside temperature.

My math might not be right, of course. It's just an educated guess.

Temperature rise is relatively linear even if it doesn't feel like it and the damage caused is not linear.

The damage we observed was not linear. In terms of deformation, pretty much all of the plastic panels were deformed. Most of the panels were only slightly deformed, maybe 1 or 2 mm per panel. Other panels were very heavily deformed, with about 20 mm of deformation at several different points on each panel. If the maximum temperature seen by each miner was normally distributed, and the amount of deformation as a function of temperature is something like D(T) = e^T, that would fit our observed distribution of deformation pretty closely.

We saw something similar with the amount of electrical damage on each hashboard. Most of the hashboards had either zero bad stages or more than two bad stages. This could also be caused by a domino effect, though.

It's also worth mentioning that the damage to the two hashboards in a single miner were correlated. Most of the miners we received either had two bad hashboards or zero bad hashboards. The number of machines with exactly one bad hashboard was less than I had expected.

Polyethylene has a glass transition temperature of -80 to -120C.  Polyethylene is malleable at room temperature and cheap to mass produce, just like the S5 plastic, which makes it a reasonable assumption unless Bitmain can confirm.  Regardless, the temperatures that caused the deformation & damage would have largely been due to the "stuck" stagnant air that you described, not a direct result of the hot aisle temperatures. (i.e., primary cause = pressure)  Other minor information I now know that I can't share agrees with that.

I disagree. I do not see how air pressure alone could have caused this deformation. I just went over and tested, and using my finger, it takes about 20 pounds of force to permanently deform a S5 side panel at room temperature by about 3 mm. That would mean you'd need close to 1 atmosphere of pressure difference across the panel to deform it from pressure alone. Typical HVAC system pressure differences are on the order of 100 Pa (0.015 PSI). Axial fans typically have static pressure capabilities around 300 Pa. If we are very pessimistic and say that the hot aisle was 600 Pa higher pressure than the cold aisle due to the two S4 fans (i.e. the S4s had zero net airflow, 100% static pressure), then that would mean about 0.06 newtons per square centimeter, or 0.08 PSI. I think temperature must have been a large factor.

Polylactic acid has a glass transition temperature of 60°C to 65°C. What are the panels actually made of? I don't know. Also, even below or above the glass transition temperature, plastics will become softer and more plastic as temperature increases. The Vicat softening temperature for HDPE is about 70°C, for example. The 0.08 PSI might not be enough to deform it at all at 20°C, but it might be enough to deform it by 1 mm at 70°C, and enough to deform by 10 mm at 80°C. I do not feel inclined to test this right now, but if this goes to court, it would be simple enough to verify.

[JaredR26]

Temperature rise is relatively linear even if it doesn't feel like it and the damage caused is not linear.  You noticed around ~95F in the cold aisle @ ~61F outside.  On a 81F day, that would be ~115F(~46C).  Maybe a bit more, but the damage to these miners(and why other miners were not similarly damaged) was primarily because of the airflow around them in relation to the S4's you mentioned, not primarily because of the intake temperatures.

My math might not be right, of course. It's just an educated guess.

I don't mean to offend you on this point; most of your work is very thorough.  But your temperature estimations are pure guesswork.  Even approximating temperature in the midst of high airflow/turbulence/heat generation situations is extremely difficult.  Temperature/thermal airflow software is extremely expensive exactly because it is so hard to approximate.  Most of those formulas and estimations regarding that aren't going to hold up to any critical analysis at all.  Delta T estimations for a point in the past with multiple unknowns, without actual measurements from a velometer or thermometer are obviously going to be terrible at best.  And since this guesswork isn't needed to draw the conclusions... why?

It was hot.  Too hot.  Definitely less than 100C.  Less than 70C.  More than 40C.  We can agree there for sure.  Even if we could calculate them, we don't need more accurate numbers based on the conclusions you covered or the ones I reached...

Temperature rise is relatively linear even if it doesn't feel like it and the damage caused is not linear.

The damage we observed was not linear.

Which would be why I said it is not linear...

Polyethylene has a glass transition temperature of -80 to -120C.  Polyethylene is malleable at room temperature and cheap to mass produce, just like the S5 plastic, which makes it a reasonable assumption unless Bitmain can confirm.  Regardless, the temperatures that caused the deformation & damage would have largely been due to the "stuck" stagnant air that you described, not a direct result of the hot aisle temperatures. (i.e., primary cause = pressure)  Other minor information I now know that I can't share agrees with that.

I disagree. I do not see how air pressure alone could have caused this deformation.

I wasn't saying that air pressure did the bend, air pressure only influenced the direction of bend at best.

I posit that air pressure, primarily, caused the runaway temperatures localized to a certain point on the boards.

Think of it this way; take two S5 miners in a chill, 15C room.  Point the exhausts at eachother directly and start moving them towards eachother.  Eventually at a certain point, maybe around an inch apart or less, the restricted airflow between them will cause the boards to overheat and fry(add time); The air temperature of the room can exacerbate the problem and make it happen faster, but it is otherwise irrelevant to the mechanism.  That is what I think happened from your description, but rather than an S5 inches away it was dozens of S4's blasting towards S5's from a few feet.

The increased local temperature that resulted from this, likely within 3-10cm from the exhaust(as you mentioned) is sufficient to explain the plastic warping and board failure without any other input.  To explain the mechanism I'm envisioning I'll have to take a big leap- Ever seen perler bead art?  Looks like 8-bit graphics.  Anyway, the beads are put on a plastic pegboard and then ironed.  They are a flat board with flat beads on them that are ironed on a flat surface.  The result, if the user isn't careful, is a warped and useless pegboard.  The heated plastic undergoes thermal expansion, but it is not uniform, and the plastic deforms.  I couldn't find any pictures to show this, unfortunately.  But the concept is exactly the same reason you can't mix aluminum and copper wiring connections; over time and through heat cycling, the aluminum will disconnect no matter how much torque you apply.  Same mechanism.

Specifically to this case... I looked up the coefficient of thermal expansion for PE and found from 80F to 135F you get ~0.5% expansion.  Putting that in a right triangle gives you a bowing outwards of about 2.5cm from a ~17.5cm plastic strip.  You said you measured 2cm at worst, so that's about right.

None of that even requires excessive intake or exhaust temperatures(whether present or not).  That, combined with other things I know and the difficulty of calculating or approximating exhaust/intake temperatures, leave me stating 'The cause was localized cooling system failure on the miner caused by high incoming exhaust pressure, exacerbated by the fact that it was "too hot".'

The air pressure differences typically seen at "reasonable" air velocities for an HVAC system (i.e. < 20 m/s) are much lower than you can apply with a finger. Typical HVAC system pressure differences are on the order of 100 Pa. Axial fans typically have static pressure capabilities around 300 Pa. If we are very pessimistic and say that the hot aisle was 600 Pa higher pressure than the cold aisle due to the two S4 fans (i.e. the S4s had zero airflow, 100% static pressure), then that would mean about 0.06 newtons per square centimeter, or 0.08 PSI. I just went over and tested, and using my finger, it takes about 20 pounds of force to permanently deform a S5 side panel at room temperature by about 3 mm. That would mean you'd need nearly 1 atmosphere of pressure difference across the panel to deform it from pressure alone. I think temperature must have been a large factor.

Polylactic acid has a glass transition temperature of 60°C to 65°C. What are the panels actually made of? I don't know. Also, even below the glass transition temperature, plastics will become softer and more plastic as temperature increases.

[jtoomim]

And since this guesswork isn't needed to draw the conclusions... why?

It helps with forensics to have multiple independent sources of information in order to cross-check facts. I first had an estimate of 55°C peak cold aisle temps based on my in-person observations. I later heard a report of ~57°C intake temps on an SP10 from another party who may have an axe to grind with ASICSPACE and might be lying. I then saw a screenshot of a KNCMiner device in their facility showing temps that would indicate intake temps around 50°C. Finally, I observed plastic deformation patterns that could be explained by hot aisle temps in the vicinity of 60°C-90°C.

It was hot.  Too hot.  Definitely less than 100C.  Less than 70C.  More than 40C.  We can agree there for sure.  Even if we could calculate them, we don't need more accurate numbers based on the conclusions you covered or the ones I reached...

Someone else might read this wondering how hot things need to be to damage an S5. I don't want people to think that they magically start to burn up as soon as the intake temperature exceeds 40°C. Accuracy is useful even when it's not necessary for the immediate task.

I wasn't saying that air pressure did the bend, air pressure only influenced the direction of bend at best.

I posit that air pressure, primarily, caused the runaway temperatures localized to a certain point on the boards.

I agree, mostly. I think that runaway temperature was mostly restricted to the hashboards and the intra-heatsink space. It's worth mentioning that all of these S5s were of the older variety that do not have any heatsinks between the hashboards and the side panels. With minimal surface area, very little heat is transferred from the hashboards to the air that passes through that space. Typically, I see a delta-T of about 1 or 2°C for that air when airflow is not obstructed. In the ASICSPACE case, that may have been 4x higher, but that would still only have been around 8°C. Furthermore, the pattern of deformation suggests that the deformation was greatest where airflow was highest, not where it was most stagnant. This implicates ambient temperatures more strongly as the dominant factor in the plastic deformation.

Specifically to this case... I looked up the coefficient of thermal expansion for PE and found from 80F to 135F you get ~0.5% expansion.  Putting that in a right triangle gives you a bowing outwards of about 2.5cm from a ~17.5cm plastic strip.  You said you measured 2cm at worst, so that's about right.

(Minus the expansion of steel and aluminum, which is much smaller.)

Most of the plastic deformation indicates a shear stress, not compressive stress. The only exception to this is the buckling that was observed in a small number of the most severely deformed plastic. The buckling of the plastic occurred entirely along the short axis of the plastic shields. If thermal expansion were the culprit, I would expect to see buckling along the long axis as well, especially along the line in between the front and back screws. That line is the least deformed part of the side panels. The bottom edge is more securely attached than the top edge as well, so if the main cause of deformation was thermal expansion, one would expect the bottom edge to show more buckling and deformation. The opposite is true. The greatest deformation occurred on edges which I expect had the greatest airflow (evenly along the top edge, plus the back and front edge near the holes for the tail exhaust and fan).

Much of the deformation occurred on the top edge, above the highest screw attachment point. That edge typically is relatively straight, but it sags outward and downward. Since this edge was not being squeezed, and if it were it wouldn't cause the edge to sag like that, I don't think thermal expansion is a satisfactory explanation. Thermal softening combined with airflow is.

The apparent buckling I observed on a few panels might actually have been fluid dynamic effects similar to ocean waves or sand dunes being created by wind rather than actual buckling.

Something just occurred to me. If the airflow was anterograde but slow through the whole intra-heatsink space, the air coming out the exhaust port would be very hot. The side spaces would not maintain their pressure as well, so airflow there would be more likely to turn retrograde. This would pull the hot exhaust air around 180° back in the side ports before flowing either out the top of the miner or through the gap in the panel between the screws. This could explain the pattern of deformation pretty well.

None of that even requires excessive intake or exhaust temperatures(whether present or not).  That, combined with other things I know and the difficulty of calculating or approximating exhaust/intake temperatures, leave me stating 'The cause was localized cooling system failure on the miner caused by high incoming exhaust pressure, exacerbated by the fact that it was "too hot".'

I think high pressures alone might have been enough to cause the ASIC damage observed, but I doubt it. I think high temperatures alone might have been enough to cause both the ASIC damage and the plastic deformation, but I doubt it. I think the evidence is clear that both existed. Indeed, it's difficult to have a positively pressurized hot aisle without getting hot air recirculation and overheating everything. I don't think there's any reason to try to pin the blame on one factor.

[SebastianJu]

jtoomim... im impressed from this investigation. Looks like you do this properly. And i like that you really seem to be independent since you dont follow the words of one side in this easily. If i would be in mining business i would definitely use your service in case i need it.

[JaredR26]

Someone else might read this wondering how hot things need to be to damage an S5. I don't want people to think that they magically start to burn up as soon as the intake temperature exceeds 40°C. Accuracy is useful even when it's not necessary for the immediate task.

True but equally frightening.  A miner burning up is a factor of a lot more things than intake temperature as you agree with the two S5's example.  Normally we talk about "intake" temperatures because it simplifies the discussion down to a workable set, but doing so makes some basic assumptions, such as that exhaust airflow isn't constrained.  As soon as someone starts violating those assumptions, everything changes.

It is possible(though not practical) to get a S5 to run safely with a 65 degree intake as well as to fail with a 15 degree intake.  We do the readers no favors by glossing over this reality in my opinion.

I wasn't saying that air pressure did the bend, air pressure only influenced the direction of bend at best.

I posit that air pressure, primarily, caused the runaway temperatures localized to a certain point on the boards.

Specifically to this case... I looked up the coefficient of thermal expansion for PE and found from 80F to 135F you get ~0.5% expansion.  Putting that in a right triangle gives you a bowing outwards of about 2.5cm from a ~17.5cm plastic strip.  You said you measured 2cm at worst, so that's about right.

(Minus the expansion of steel and aluminum, which is much smaller.)

Most of the plastic deformation indicates a shear stress, not compressive stress. The only exception to this is the buckling that was observed in a small number of the most severely deformed plastic. The buckling of the plastic occurred entirely along the short axis of the plastic shields. If thermal expansion were the culprit, I would expect to see buckling along the long axis as well, especially along the line in between the front and back screws. That line is the least deformed part of the side panels.  The bottom edge is more securely attached than the top edge as well, so if the main cause of deformation was thermal expansion, one would expect the bottom edge to show more buckling and deformation. The opposite is true. The greatest deformation occurred on edges which I expect had the greatest airflow (evenly along the top edge, plus the back and front edge near the holes for the tail exhaust and fan).

Remember when talking about temperature under cases where normal assumptions are violated, it is useless without also talking about specifically where the temperature is.  The middle of the plastic sheets didn't warp because the middle didn't get so hot.  The edges did because the edges touched the metal, a better heat conductor, and the exhaust edge specifically had higher heat. In the pictures you have provided, one perhaps 2 of the plastic have deformations on both sides.  6 or 7 have deformations on one side with far smaller or no deformations on the opposite side.  Of all of the sheets in the picture, one shows minor top-edge deformations, one shows a lot of top edge deformation, and the only other one with top edge deformations is very close to the screws/my proposed hot spot.  6 show no top edge deformation at all.

Much of the deformation occurred on the top edge, above the highest screw attachment point. That edge typically is relatively straight, but it sags outward and downward. Since this edge was not being squeezed, and if it were it wouldn't cause the edge to sag like that, I don't think thermal expansion is a satisfactory explanation. Thermal softening combined with airflow is.  The apparent buckling I observed on a few panels might actually have been fluid dynamic effects similar to ocean waves or sand dunes being created by wind rather than actual buckling.

Something just occurred to me. If the airflow was anterograde but slow through the whole intra-heatsink space, the air coming out the exhaust port would be very hot. The side spaces would not maintain their pressure as well, so airflow there would be more likely to turn retrograde. This would pull the hot exhaust air around 180° back in the side ports before flowing either out the top of the miner or through the gap in the panel between the screws. This could explain the pattern of deformation pretty well.

To be clear, I don't believe airflow played no part in what we see, that would be silly of me.

[CryptoCoin2015]

jtoomim... im impressed from this investigation. Looks like you do this properly. And i like that you really seem to be independent since you dont follow the words of one side in this easily. If i would be in mining business i would definitely use your service in case i need it.

... and thats why we host 200 S5 with Jonathan and hope, that ASICCRAP bites the dust...

Thanks Jonathan for the hard and dedicated work you did put into this.

[SebastianJu]

jtoomim... im impressed from this investigation. Looks like you do this properly. And i like that you really seem to be independent since you dont follow the words of one side in this easily. If i would be in mining business i would definitely use your service in case i need it.

... and thats why we host 200 S5 with Jonathan and hope, that ASICCRAP bites the dust...

Thanks Jonathan for the hard and dedicated work you did put into this.

Hm... i only now, when you mentioned this, realized that toomim brothers are a competitor of AsicSpace. Even though the investigation sounds trustworthy... this can hardly be seen as an independent investigation then. I thought they are hardware nerds who can check those things out. But letting the competitor decide if the competition made an error is not really something that comes to mind when one wants an independent investigation about the source of the problems. I dont say anything, i only point out a conflict of interests.

[jtoomim]

Hm... i only now, when you mentioned this, realized that toomim brothers are a competitor of AsicSpace. Even though the investigation sounds trustworthy... this can hardly be seen as an independent investigation then. I thought they are hardware nerds who can check those things out. But letting the competitor decide if the competition made an error is not really something that comes to mind when one wants an independent investigation about the source of the problems. I dont say anything, i only point out a conflict of interests.

Yes, I have a conflict of interest. Sorry if that wasn't very clear. I'll edit a note of that into my initial analysis.

Edit: It's also worth mentioning that I used to be business partners with Robert and Damir, and my brother and I originally planned to design and fund most of their datacenter for them in exchange for free hosting for our SP30s and partial ownership of the business. My brother and I decided that we did not want to be involved with them (mostly because of Damir), so we broke off and forged out on our own. They claim we acted inappropriately and dishonestly, and that our original intent was just to deceive them and use them to find a location to build our own facility. I claim that they were not competent enough to continue collaboration with, and that we didn't realize this until after we were heavily involved with them.
You can read about that history here.

[jtoomim]

In other news, ASICSPACE fired their CEO, Damir Kalinkin, last week. I believe the exact process was that the shareholders held a vote of no confidence, then cleared out the Board of Directors (of which Damir was a member), and then the new Board fired Damir. In my (biased?) opinion, Damir was the worst part of ASICSPACE, and responsible for the most egregious misdeeds I have heard attributed to them. With Damir gone, I expect ASICSPACE to improve considerably. Robert is a nice guy, and I like him. From what I have seen, Robert wants to treat his customers properly, unlike Damir, who treats customers as a resource to be exploited.

However, I do not see how Damir can be responsible for the heat, airflow, and networking problems that ASICSPACE has had. He is not a technical person at all. Those issues were due to the decisions made by the engineer and contractors who built the facility and the person who oversaw them, Robert.

When I was at ASICSPACE earlier today, the conditions were considerably improved. The nine flexible ducts that were previously just neutral pressure outside-air intakes have now been connected to a 125 ton portable AC unit on a trailer, powered by what appeared to be a 400 amp 480V 3-phase connection. I didn't go into the cold aisles, but they appeared to be only slightly negatively pressurized relative to the hot aisles, maybe around 10 or 20% of what they had been at before. The general building interior (contiguous with the hot aisles) was strongly positively pressurized relative to the outside air, and the airflow out the open front door was very strong, maybe 50,000 to 100,000 CFM. The exhaust air from S4s felt like about 40 to 45°C, suggesting cold aisle temps around 30 to 35°C. Still high, but no longer unsafe. Most of the miners in their facility appeared to be on and hashing, although I did hear S4 beeps coming from somewhere.

ASICSPACE appears to be making a good faith effort to maintain proper operating conditions in their facility. They are putting a lot of money behind that effort. Unfortunately, they are spending it on the wrong things, like the air conditioning unit instead of a bunch of high-throughput fans. They don't need colder air, they just need more fresh air. However, the AC unit they are renting did come with a large fan inside, so there's that.

[CryptoCoin2015]

Unfortunately, they are spending it on the wrong things,

right, like e.g. paying back the damage they did to our equipment and the fees for services they never rendered to us

[CryptoCoin2015]

Hm... i only now, when you mentioned this, realized that toomim brothers are a competitor of AsicSpace. Even though the investigation sounds trustworthy... this can hardly be seen as an independent investigation then. I thought they are hardware nerds who can check those things out. But letting the competitor decide if the competition made an error is not really something that comes to mind when one wants an independent investigation about the source of the problems. I dont say anything, i only point out a conflict of interests.

And why the fuck would there have to be an independent investigation anyways? Because you would like it?
If ASICSPACE wants this investigation, then they should do so and PAY FOR IT.

We kindly asked Jonathan to assess the damage to our equipment and the cause, because we need to decide whether it is ASICCRAP to sue or BITMAIN. What right do you have to claim an independent investigation? If you want a different consultant doing an "independent" investigation, go ahead. Hire some consultant and get it done. For now, you got to live with what is on the table and you either take that or you don't.

But please don't come up with the call for an "independent" investigation, as this is needed. It does not alter the tuth in Jonathans statement by an inch, if he is now rsponsible for our equipment or not. The damage was done beforehand.

I dont say anything, ...

truer words have never been spoken. You always come up with the most ridiculous claims and the request for a observatory stance like this is Kindergarten.

Fuck this. Damage has been done, ASICCRAP is not paying up and customers of theirs are not getting reimbursed for the damage.

It is a good thing, that Damir is gone. Now, only 2 more are left to be dealt with.

Robert van Kirk and Geoff Smith, two fucking cowards who cannot deal with taking their share of responsibility like a man should.

[Bicknellski]

Hm... i only now, when you mentioned this, realized that toomim brothers are a competitor of AsicSpace. Even though the investigation sounds trustworthy... this can hardly be seen as an independent investigation then. I thought they are hardware nerds who can check those things out. But letting the competitor decide if the competition made an error is not really something that comes to mind when one wants an independent investigation about the source of the problems. I dont say anything, i only point out a conflict of interests.

And why the fuck would there have to be an independent investigation anyways? Because you would like it?
If ASICSPACE wants this investigation, then they should do so and PAY FOR IT.

We kindly asked Jonathan to assess the damage to our equipment and the cause, because we need to decide whether it is ASICCRAP to sue or BITMAIN. What right do you have to claim an independent investigation? If you want a different consultant doing an "independent" investigation, go ahead. Hire some consultant and get it done. For now, you got to live with what is on the table and you either take that or you don't.

But please don't come up with the call for an "independent" investigation, as this is needed. It does not alter the tuth in Jonathans statement by an inch, if he is now rsponsible for our equipment or not. The damage was done beforehand.

I dont say anything, ...

truer words have never been spoken. You always come up with the most ridiculous claims and the request for a observatory stance like this is Kindergarten.

Fuck this. Damage has been done, ASICCRAP is not paying up and customers of theirs are not getting reimbursed for the damage.

It is a good thing, that Damir is gone. Now, only 2 more are left to be dealt with.

Robert van Kirk and Geoff Smith, two fucking cowards who cannot deal with taking their share of responsibility like a man should.

I think it is clear yet again that Seb has overstepped.

[JaredR26]

Hm... i only now, when you mentioned this, realized that toomim brothers are a competitor of AsicSpace. Even though the investigation sounds trustworthy... this can hardly be seen as an independent investigation then. I thought they are hardware nerds who can check those things out. But letting the competitor decide if the competition made an error is not really something that comes to mind when one wants an independent investigation about the source of the problems. I dont say anything, i only point out a conflict of interests.

Yes, I have a conflict of interest. Sorry if that wasn't very clear. I'll edit a note of that into my initial analysis.

Edit: It's also worth mentioning that I used to be business partners with Robert and Damir, and my brother and I originally planned to design and fund most of their datacenter for them in exchange for free hosting for our SP30s and partial ownership of the business. My brother and I decided that we did not want to be involved with them (mostly because of Damir), so we broke off and forged on our own. They claim we acted inappropriately and dishonestly, and that our original intent was just to deceive them and use them to find a location to build our own facility. I claim that they were not competent enough to continue collaboration with, and that we didn't realize this until after we were heavily involved with them.
You can read about that history here.

Interesting, it is good to finally know the rest of the story.  Edit- Hmm, found more info I didn't know.  Removed negative comment.

truer words have never been spoken. You always come up with the most ridiculous claims and the request for a observatory stance like this is Kindergarten.

Yes, being an asshole to longtime highly trusted members of the community is going to go REALLY well for your war on Asicspace.

Fuck this. Damage has been done, ASICCRAP is not paying up and customers of theirs are not getting reimbursed for the damage.

You might want to review your hosting agreement with Toomim.  No hosting company anywhere is going to reimburse customers for miners(according to the contract).  They can't.  The margins are too slim, and there's too many things that can possibly go wrong with miners, many of which(not all, and not necessarily in this case) aren't under the control of the hoster.  Some may reimburse in some situations, but it won't be a standard thing except perhaps with the highest cost providers(at a net loss to their customer base).

For all your kicking and screaming, if you take this to court, you will almost certainly lose.

So your intent as you have made clear is to make Asicspace suffer economically.  Might I suggest that if you really want to tarnish Asicspace's reputation, being an asshole about it will only make people doubt your statements?  Or did you not notice Toomim's statements, being a stand-up guy(respect): "When I was at ASICSPACE earlier today, the conditions were considerably improved."  Are you going to rant and rage at toomim now for saying something positive like you did to SebastianJU for saying something neutral?

For now, you got to live with what is on the table and you either take that or you don't.

Hmm....

[SebastianJu]

Hm... i only now, when you mentioned this, realized that toomim brothers are a competitor of AsicSpace. Even though the investigation sounds trustworthy... this can hardly be seen as an independent investigation then. I thought they are hardware nerds who can check those things out. But letting the competitor decide if the competition made an error is not really something that comes to mind when one wants an independent investigation about the source of the problems. I dont say anything, i only point out a conflict of interests.

And why the fuck would there have to be an independent investigation anyways? Because you would like it?
If ASICSPACE wants this investigation, then they should do so and PAY FOR IT.

We kindly asked Jonathan to assess the damage to our equipment and the cause, because we need to decide whether it is ASICCRAP to sue or BITMAIN. What right do you have to claim an independent investigation? If you want a different consultant doing an "independent" investigation, go ahead. Hire some consultant and get it done. For now, you got to live with what is on the table and you either take that or you don't.

But please don't come up with the call for an "independent" investigation, as this is needed. It does not alter the tuth in Jonathans statement by an inch, if he is now rsponsible for our equipment or not. The damage was done beforehand.

I dont say anything, ...

truer words have never been spoken. You always come up with the most ridiculous claims and the request for a observatory stance like this is Kindergarten.

Fuck this. Damage has been done, ASICCRAP is not paying up and customers of theirs are not getting reimbursed for the damage.

It is a good thing, that Damir is gone. Now, only 2 more are left to be dealt with.

Robert van Kirk and Geoff Smith, two fucking cowards who cannot deal with taking their share of responsibility like a man should.

I really dont know why you are so aggressive and attack me. I got the wrong impression that you wanted to do an independent investigation. Im not so much into mining that i would have known who they really are.

Its really a pity that the behaviour on this forum lately turns to people being the most aggressive they can. Maybe its because of AirWolf who was able with extortion, threatening relatives and so on to get a refund from a scamming company.

I would prefer you speak normal with me. I admitted my error and theres no reason that you feel instantly attacked the way you seem to feel. Only because you feel i demanded an independent investigation doesnt that make your statement valid. It says more about you than me.

I think it is clear yet again that Seb has overstepped.

Because i dont have the time to investigate every username i come up on here? I admit my errors. It would be great if others would be able to do so too.

[Bicknellski]

Hopefully you can nego a settlement it is pretty clear ASICSPACE is at fault in this regard given all the retrofitting going on there to REDUCE heat. ASICSPACE should simply compensate you immediately and end this without anymore delays. That is what good businesses do.