Bitcoin Forum

Anyone considering doing business with ASICSPACE, stay away from them.

Anyone doing business with ASICSPACE, get away from them.

What we experienced with them was and still is nothing short of a nightmare.

We hosted around 100 S5 with them of which the majority has suffered severe damages from heat whilst in their data"hut" (I cannot honestly call this a center).

They never ever at any point in time delivered the contractually agreed upon services, due to a total desaster and incapability of staff and technology to deliver at all, what they claim to be able to deliver.

They had and have network, heat and power issues.

We still wait for the reimbursement of the prepaid hosting services in the height of around 5.000 USD which they failed to deliver even in the slightest.

Robert van Kirk is a nice guy and working his ass off to get anything going at all, but he is not a technician, that could get problems of this magnitude as they are facing them off the table. He should be in a better company than this.

Geoff Smith is probably just asking himself what he has gotten himself into.

Damir Kalinkin is a crook and madman, that should not be trusted at all.

This will only be the beginning of an intense legal shitstorm raining down on them and afterwards, there will probably nothing left of their company anyway, so better get all of your stuff out there before they sell it off in order to pay for all the damages they did.

Feel free to contact me.

That's a real shame.

From your posting I surmise that this a legitimate project that ran aground for various reasons, as opposed to an outright scam or fraud. Is that right? Sorry to hear about your damaged S5's.

Hut? Judging from these images it looks like they are really good prepared. https://bitcointalk.org/index.php?topic=1036376.msg11178401#msg11178401

Hi! I'm from ASICSPACE and let me add some information:

This customer's miners did receive damage, the cause is still being investigated.

Temperatures in the data center have been elevated due to record setting high temperatures over the past week, but it appears that the temperature control limits on this customers miners were set to "off".

ASICSPACE staff did not manually do any such thing, and we are working the the developer of our management software to see if is at all possible that it could have changed that setting.

We'll update this thread after our investigation.

Robert

I wanted to give my reply at least a couple days to see, what ASICSPACE would come up with. As expected nothing, as they can and do no not contribute in any way with resolving the situation or the detailed technical analysis. They even could not as they have no longer access to any machine at all.


This is correct along the line, that the investigation is not done by ASICSPACE, as they lack the competency to do so. This is done by Toomim Bros. to whom we have relocated the equipment. They have all the means and skills required to do so and work in close relationship with Bitmain to do the detail root cause analysis. One thing is at this stage however undoubtful: The reason is excessive heat.


So, lets conclude: If ASICSPACE did not do this, then it must have been someone else from outside ASICSPACE. This means, that ASICSPACE does not rule out that other people than ASICSPACE staff have access to their customers equipment. Interesting.

Well, as it could not have been us, the customers, as we did not have any remote access to the machines at all we are talking about a Third Party here, that is probably living under the floorboards of the datahut and comes out at night to fiddle with Antminers. One possibility.

Or, it could have been due to the lacking technological skills of a yet unknown software developer who is probably called by the name Chuck the Chimp, that the software tools are faulty in a way that they distribute wrong settings or cfgminer profiles.

However, even after two days of investigation, ASICCRAP has not been able to come up with an explanation.

Other than that however, ASICSRAP has meanwhile come up with a new twist:

- they pretended to have found a buyer, who would buy all our machines in their current state for the money we paid for them. Of course, 24 hours later that buyer vanished.
- they now claim, that we did not pay for energy in advance ("if you closely look at the contract") but for maintenance for our machines. As if it would make a difference, which service you did not deliver and get paid for.

Thanks to the contributions of the community though, we are getting a more and more complete picture of their mode of operations, that we are putting together piece by piece to prepare for the big take down.

Stay tuned to learn about interesting topics such as setting up miners to mine into ASICCRAPs pools instead of their customer ones.


Well, lets see who has the more interesting updates ;)

Maybe we should start a countdown until ASICRAPS ultimate downfall.

I've got some miners there.  I was very happy with them until a couple weeks ago, then I lost remote to my miners and during that the pool rate dropped to almost half. 

But the last few days my miners have been hashing away.  One thing about these guys, when they screw up they are quick to respond with a rebate of the hosting fee.  So I am tolerably happy, at least for now.  I'm certainly watching my pool results closer, though.

They may not be the sharpest guys, but they do seem honest.

When you think about that topic then there is a third explaination. What if a competitor sends some miners there and knows they will break because they are changed that way?

The same might go for a scammer who wants to dump useless miners that way in the hope to get a refund.

Of course then a new account would be used for the accusation. Of course there are other explainations for a new account too.

So in fact there is one possibility more than you claimed. Its no accusation, this came only up when reading that topic.

So the miners arent there already and you expect them to find out the problem themselfes? I guess they would wait for Toomim Bros right?

And you claim the rooms are not vented. If the temperature sensor isnt working and the chips overheat then all the fresh air around it shouldnt be able to change anything. So if the temperature sensor was deactivated then this wouldnt have to do anything with proper cooling of the hosting room, am i right? I dont know the type of miners though.

Might be your anger that you write that way. I expect that some of it will flow my way now though i dont accuse, i only say there is another possibility. So maybe you should wait about what the investigation shows.

What kind of miners was it? Did you run them by yourself before or did you buy them from someone and let them send there?

You are not providing an explanation. You are coming up with guesses based on nothing but assumptions. Not helpful.

How could you provide an explanation? In no way, as you are involved party at all with no actual factual knowledge of the situation.


Have you ready anything I have written at all? They are brand new S5, delivered straight from Bitmain to ASICCRAP.

They have damaged them, they already admitted doing so.

Right, im sorry that i didnt read carefully.

So these were delivered to ASICSPACE directly from Bitmain and are brand new.

I guess there are no other complaints about the temp sensor from S5-Customers?

I would like to read the results of the investigation from Toomim Bros.

How in fuck could they overheat them? You would have to try to do that.

Look at these pictures. What the hell have they done to them? Doused them with gasoline and set them on fire? There must have been beyond 100 degrees C.

https://i.imgur.com/dsKxjPv.jpg
https://i.imgur.com/6vZjPbq.jpg?1

Jesus that's bad. I assume there is no warranty because it is human error.

EDIT: Did they not form an LLC and obtain Liability Insurance? People need to ask to see that.

Do you have more pictures than that? PM me if you don't want to post them.

Because so far those pictures and your conclusions don't indicate much except that you have never held an actual S5 in your hands.  That isn't sheet metal, that is flimsy soft plastic, most likely polyethylene.  They practically deform in my hands when I hold them, and definitely would deform if grabbed roughly.  If it WAS evidence of temperature damage, LDPE has a deformation temperature of 32C to 41C and HDPE has a deformation temperature of 43C to 49C, so there's no "100C" anywhere near this.  Source: http://www.pvc.org/en/p/heat-distortion-temperature-softening-temperature

That second pictures indicates very little about temperature either.  I've seen PCB capacitors catch fire and burn out like that at a variety of temperatures; most of them weren't heat related.  Jonathan should know this stuff too, he's a smart guy, so I am surprised he sent you those pictures as evidence of heat damage.

I'm not saying there wasn't or was heat damage, I'm saying those pictures show jack shit to someone who actually has done this stuff before.  Jonathan is smart; if Toomim is truly as awesome as you say, they should be held to a higher standard than pictures like these for an "investigation."

** Full disclosure - I used to consult for Asicspace, but haven't had any real impact there in months and discontinued my affiliation with them weeks before this incident or even before the DC was filled.  **

One thing that I would investigate is that the internet connection had been lost. If your scour the Antminer S5 OP I seem to remember someone saying that their S5 fan slowed down after losing internet connection but the machine kept on running at full throttle to the point that if they had not been there watching it happen the machine world have burned up.

I do not know how you can prove that this happened in this case but if that did occur then you will have a warranty claim.
Also, scouring that OP, others have claimed that the 80°C fail-safe does not work either.

This user does this on a lot of threads. Seemingly defending the indefensible actions and doesn't READ THE FUCKING THREAD. Careful he might call you an EXTORTIONIST next if you are looking for fair compensation for the damage. Never seen warped parts on a miner like that before WTF? That is some sort of extreme and sustained heating to cause that to warp.

Admission of liability?

No where have I stated, that this is the conclusion of Jonathan. So leave him out of this, unless he himself makes any statement that you can comment on. The blame for misinterpreting or misunderstanding the damage is full on me.

Jonathan is still working on getting to the root of this and especially to find a way, if any, to get the damage repaired. Give him a call if you want, tell him he can release the info to you if he wishes, including more pics.

This is my statement and I have all rights to be fucking angry with ASICCRAP and accusing them of mishandling my stuff in any way I want ;)

Chances are close to zero to get a warranty for this, as even Bitmain seems to know meanwhile whats up with ASICCRAP.

It was a necessary decision to move quick and void warranty or to leave things there and go the lengthy and probably unsuccessful route of claims and lies and more lies. ASICCRAP would have never handled the return process properly.

No insurance is gonna pay for mishandling :( This would require ASICCRAP to admit to be in the wrong.

Asicspace admitted to elevated temperatures.  Asicspace admitted to downtime of your miners.  Great, both facts, not really in dispute.

YOU accused them of dangerous temperatures upwards of 100C based on photos that you didn't even realize was flimsy plastic.  As coinits points out, there may even be a bug at fault that neither you nor asicspace knew about that would have Bitmain share the blame here.

Since you clearly don't know how to evaluate the damage or the responsible party, your accusations are really just going to make you look like an angry asshole instead of a poor mistreated customer.

That is what liability insurance is for. Mine keeps my clients protected if I fuck up. I have never been sued so over time my rates go down but all that it takes is one major fuck up of incompetence and they would either drop my coverage or jack my rates exponentially but at least my client would be covered.

Them not having this is sad.

Thank you for clarifiying this. I understand the nature of the insurance policy now. Well, they at least never offered to use it so I guess, they do not have it :(

Check this out OP

https://bitcointalk.org/index.php?topic=902305.msg11188709;topicseen#msg11188709

I have friggin S5 here at my place, so again: Do not jump to conclusions if you want to appear as someone applying rational thinking. Because you are not, you are assuming again.


Again: assumptions. You neither know my background nor anything else, yet you seem to know it all.

For all I know, you have no idea at all about anything.

And besides that, you are wrong: A hoster is responsible for the hosted equipment, hence ASICSPACE should have taken action once they recognized the damage. The truth is, ASICCRAP was at no point in time aware of the damage, as they simply had not taken a single look at the machines but rather said something about network issues, slight heat problems etc.

If this was not about hiding, then it was about incompetence. And what knowledge do we have about systematic S5 failures? Right, none. Bitmain has a faultrate, which is by far below 1% They disclosed this and much more when working on the systematic evaluation of the situation and are doing so right now in Denver.

Does this look like "slight heat problems" to you, if 50% of the boards look like this?

Ah, you do not need to answer, it does not contribute in any way to the solution of the problem.

Thanks, I will bring this to the knowledge of Bitmain to have this investigated in our current case. Very helpful, thank you.

Welcome. I hope that it works out for you. It is hard enough to make a buck in crypto without shit like this happening. Keep us posted.

Ok.  So why did you make claims that it might have been 100C to deform it?  You can practically deform it with your hand.  By your own admission you must have known this.  Why make such claims?


I didn't claim to know your background.  You literally said "There must have been beyond 100 degrees C."  I showed that that claim is completely bogus.  People can draw their own conclusions about our respective backgrounds.


I know this to be at least partially untrue, but it is all secondhand information.  However you and I can definitely agree that things weren't handled right, and that this was not how a hoster should treat their customers.


Right, and it will be good to have their assistance; keeping in mind that they also have a very strong impetus to avoid both the liability for the hardware and the reputation impact that it would have if they admitted any fault such as automatic shut-off temperatures not working.  Bitmain are good guys, I know them, but they are also very clever.  If Bitmain can avoid reputation impact by pinning 100% of the blame on another party, they will likely do so even if a failure on their side contributed to 20% of the problem or more.

I also know that they run large quantities of their hardware in "datacenters" in China, often under high temperatures and with a build construction that makes Asicspace look like a tier-II datacenter.  We are not likely to get the operational data(temperatures) from those places nor the failure rates in them because the numbers won't be good.  (The numbers aren't supposed to be good; it is designed to fail and be cheap to replace).


50% of what boards?  50% of the pieces of plastic?  I'd say that's rough handling or the result of running in a BTC mine.  50% of the circuit boards have a capacitor blown?  Now this would be some useful data.  Yes, I would agree that if you have 50% of S5 boards with visible capacitors blown like that I'd agree that you either have a major electrical problem(more likely, although I know the electrical is pretty solid there) or a major heat problem(heat affects chips more than capacitors).


What you mean to say is, it isn't helping your angry arguments.  Lets just try sticking to the facts, shall we?

There is a huge difference in a material being easily able to be bent by physical force and material sitting in a shelf being deformed by heat. I will put a 1.200 watt blowdryer on the weekend against one of my S5s sides (gotta pick one) and we will see the outcome. I will measure the heat at impact, ok?


Yes, and no hoster should have the customer get aid from the police to retrieve his equipment.


Agreed. I am suspicious of Bitmain as well, but to have such an error quote in such a short time frame under the same conditions, with only me claiming these errors seems to statistically less relevant and only an option to pursue after all other reason have been ruled out.


50% of S5 boards with visible capacitors blown or similar damages.


Agreed.

Fair enough, but that doesn't rule out physical deformation.

In addition, having recently done this with a different type of plastic, heat deformation of plastic generally leads to very different results than your pictures show.  Since the deformation itself is caused by the expansion and contraction of different sections of the plastic at different rates, what you end up getting is generally edges aren't straight anymore and/or that the material will bulge, bow out, or sag at certain points.

The pictures you've shown show perfectly straight edges on all of the plastic, with the possible exceptions of the middle one(rear, left side) and the far left rear one, but both of that could be camera angle too.  I don't see any bulges or sagging, nor any deformation of the antminer/bitmain logos.  All of the units have bends near the screwholes, sometimes much sharper than what heat alone will achieve, implying that they were bent either on installation(at bitmain), during transportation on one of the several trips, during setup, or during the removal of the covers.  The largest bend is the far rear right one, but A. the bend is sharper than what most heat damage would cause, and B. both the edges and the lettering on the logo are still straight.

I honestly think the plastic sheets are a red herring completely.


In this particular case I agree this could and should have been handled better.

That said, if a customer hasn't paid their bills, they can't retrieve their equipment.  Hosters have big electricity bills and other bills to pay, none of whom give a damn if your customers haven't paid their bills.

And clearly your hardware did run up its portion of an electricity bill, otherwise how could it have damaged capacitors?  There is no temperature in an operational bitcoin mine, no matter how poorly cooled, that can damage a device that isn't powered on(Short of fire, not relevant here).


Do you happen to have more pictures you could post and/or send me?  That is actually very very suspicious.  I know of bitcoin mines that had spots of up to 135F(57C, some with GPU, some with bitfury hw).  The bitfuries had few failures, and the GPU farm had mostly PSU failures.  I've personally run AM hardware at 45C stably, and know of other mines that ran (for a time) at upwards of 76C with bitmain hardware and afterwards ran at 49C for a long time, and had a lower failure rate than that.

All that said, I've been in that DC a bunch and know their cooling pretty well.  I'm sure it got hot, but I really have a hard time believing that it got above 57C without a major cooling failure(Which, conspiracy theories aside, doesn't seem to be the case).  And that's where my background is important, as I've designed and built the cooling for 4 or 5 bitcoin mines now.  The record day was 27C at peak, so that would be a 30C temperature rise, which is crazy.

Even if you doubt me or my background, I'd be really careful about making the central point of your investigation/case a > 57C ambient temperature, because they may have/gather other data and if they prove you wrong, your entire case falls apart.

If the failure rate is truly that high, my best guess from what I know is:
A. It was definitely too hot in the mine, though lower than the highs I have seen in other mines(57C)
B. S5's are particularly sensitive to heat damage, moreso than most other hardware brands and even previous generations of antminers.
C. The S5's did not apply any temperature protection, if they have such.

On another thought, what power supplies were used with these S5's?  Link or image?  To me those are more likely to blow capacitors(likely in combination with the heat), if they don't deliver clean voltages.

Nope. The deformation perfectly shows, that the internal heat has been causing this, as the hot air was blown out of the casing and hence causing this type of deformation. The logos are not in an area where any air circulation happens but the sides between the screws are. Your theory for external impact due to move and such surely would have caused such random effects and on logos as well. The type of damage you see here, being consistent on all machines, shows that the hot air was taking its way all from inside to outside, going through the slits between the screws and hence deforming the shielding.


If you are implying, i did not pay my bill, you are in the wrong again. I payed them 4661 USD for service and energy upfront for a month.

They managed to get all this damage done in a couple hours, so I guess, not much energy was needed for that. Since then they refuse to either assist in repaying the damage nor paying back the bill they never provided anything for but trouble.


As said. If you want to know more, you need to contact Jonathan. Pictures do not help here any longer for any serious investigation.

Listen to other customers here. I have been contacted by a dozen, whom machines do not answer any longer and who only can see and get a couple of them working again.

All I can tell you is, that ASICSPACE is being run by a bunch of criminals. Maybe you know them, maybe they are not longer the people you once knew.

Today, all that they are is a bunch of crooks with highly criminal intent.

I expect, that I will never see any dime from them. But I will make sure, that they pay the bill.

The airflow in the S5 is crossways from the fan, internally.  The plastic shielding does not touch or enclose the S5 circuit boards at any point.  Are you certain you aren't confusing an S5 and with an S3?  Because an S3 was metal and fully enclosed.

Here, let me help.
https://www.avito.ru/moskva/oborudovanie_dlya_biznesa/mayner_bitcoin_antminer_s5_1.3th_blok_pitaniya_537197819

Oh shit, temperature deformation!

http://i00.i.aliimg.com/photo/v1/60145139694_1/New_model_ANTMINER_Ant_Miner_S5_1155.jpg

No hot air would have been trying to go "from inside to outside, going through the slits between the screws" when it can just go up or down through large openings.  More likely the thing you are pointing at affecting 5 of the 8(and we likely are only seeing the worst) plastic ones combined picked up conductive heat from the metal frame, which would be much closer to the temperatures of the circuit boards/chips, and definitely hotter than ambient air.

Since the deformation temperature of plastic is less than 49C, and many miners I have seen operate without such failure rates at up to 57C, deformed plastic that was adjacent to the hot metal frame means absolutely nothing of relevance except that the metal itself, but not the ambient air, must have been higher than 41-49C.


You made a blanket statement; my blanket response wasn't intended to be about you personally.  Regardless of that, Asicspace did get screwed over a few weeks beforehand by a customer who did not pay their bill.


Not saying they don't have other complaints or other issues.  I know they do.  But that doesn't indicate heat damage directly, as there are many ways a miner can fail.  And yes, I'm aware that they have other heat complaints, but so far as I know no other heat-damage complaints.


As far as I can tell, nothing even you have accused them of is against the law.  This is a civil and customer service matter, so stop acting like they are some kind of gangbangers.

Asicspace isn't perfect, no one is.  I know they do not have any intentions to scam anyone.


So basically, you aren't interested in the truth of what happened, you are just angry and vindictive and want to hurt Asicspace in either reputation or with vague legal threats?

Who has the worse intent, Asicspace who may have blundered(both cooling, network, and customer service) but may not be 100% to blame, or you who wants to see them burn regardless of the facts?

Understood. They finally got someone out here from their crew to calm things down and get their reputation back.

Well, thats one way to earn a Dollar :)


You have a funny understanding of the law.

Asicspace has not blundered. They have stolen my money, damaged my property and bent the truth in each and every conversation.


Oh, I am interested in the truth, it just so happens you do not have it.

If that is your kind of company to keep, please be welcome to do so.

Well, maybe they just fucked with the wrong guy this time and yes, I want to see them burn for this. And I will.

Also note, hair dryers are supposed to flow at up to 140F/71C at least in the U.S., so that raw temperature is higher than the ambient temperatures we are discussing/expecting here.  If you have access to a thermal camera that would really let you see what temperature it happens at.  However, miner chips are often safe to run at up to 80C safely(though not ideal), so for certain boards it is possible that the metal frame, directly connected to solid objects that conduct heat better than air, could be hotter than ~60-70C without indicating a failure situation.  I don't know about the S5, too many things to possibly calculate that.  But when talking about temperatures, it is really important to specify what/where you are measuring to know what it indicates.

In my experience, ambient intake temperatures are safe up to 40C for most miners, and can be workable for some devices up to ~55C.  Ambient exhaust temperatures can safely be at 49C and for some devices are workable up to ~62C.  Chips are happy at 60C, safe to around 80C, and start to get dangerous/damaged somewhere over 95-100C.  I haven't measured circuit boards temps enough to know their failure ranges.


I make nothing from Asicspace, and honestly lost money on the deal I had with them, spending more money talking to my lawyer about the situation than I ended up being paid.  I am the one that ended my association with them before I was paid what I expected to make.  That doesn't make them criminals, and neither does this.

This is the business world.  You signed a contract* and paid money.  Asicspace failed to deliver as promised, thus defaulting on their side of the contract.  That is not theft, that is default.

They attempted to operate your equipment per the contract and the heat it produced damaged it.  They are not criminally liable for this by any law.  This is a civil and contractual dispute.  I know you aren't in the U.S., but do you really not have this concept of civil court versus criminal court?  Theft versus contractual dispute?

You demanded your equipment be removed long before the contract had ended, and before any investigation that any court would expect could have been completed.  You defaulted on the contract as well.

Both sides are in default.  In my opinion you should get part of your money back, probably most, but I am in no position to judge how much and don't want to be.  That's why we have courts.

* or at least, you should have if they did things right.  If you didn't sign, it was still a verbal/written agreement regardless which carries some weight.


I'm not sure what you call a long network outage followed by high temperatures on a hot day, but I'm pretty sure someone screwed up somewhere.  If they had criminal intent they would have taken everything from everyone and fled.  But they are still there, still mining, and they still have paying customers, some happy, some unhappy.  What part of that doesn't sound like a blunder?

Sorry, but you clearly have no idea what you are talking about when it comes to legal issues.
You do neither know the content of the contract, the details that have been agreed upon nor the exact chain of events.
Yet, you jump from one assumption to the next assumption, where your clearly lack the required facts upon which your statements should be based in the first place.
That is just not how it works buddy.


I give you a couple hints here, legal lessons for free: the contract is void, due to their inability to perform in the first place. It has been virtually impossible for them to fulfill.
Your lawyer will explain the rest to you. Or wikipedia. Where ever you get your knowledge from these days.


Again, you mix up things where you have no idea how the legal system works: I have at no time given up my ownership of anything. This is like you claiming, that a car parked in a parking garage is during that time the property of the parking garage owner. This is pure nonsense.


Jeez, you are contradicting yourself in two sentences: First, you claim both sides are at fault, then you state, that you are in no position to judge at all. Stick to the latter, please!


Again: You are the customer any criminal wishes for. I have seen plenty of your type during the boiler room days.

The intent, the sole intent of the whole operation is not to deliver services. It is to screw their customers over and over and over again.

Be it that the miners are set to mine into their own pockets. Be it, that they deliver less power than paid for. Be it, that they manage and administrate the property of other parties recklessly with no regards to ownership.

This is all the big picture and the driver behind this is criminal intent.

A blunder is an accident. This "blunder" however is an event that happened due to a gamble: the gamble from their sides, whether they could despite their lack of knowledge and technical capabilities deliver a contract that they had nearly no means for to deliver. And which they knew.

At the moment they accepted that gamble, they were hiding the truth, lying and cheating. They did this with intent and hence they did this with criminal energy. Fraud.

They were desperate to get the deal, desperate enough to not care what would have been best in both interests and reckless, when the time would have been right to take responsibility and get things straight again.

Judge: "So you are claiming that these individuals are not actual hosters for these servers?"
You: "Yes your honor, as we have shown, these individuals clearly made an agreement with the full knowledge that they were and are incapable of fulfilling it, and -"
Judge: "Wait, hold on.  They were hosting for others before you, right?"
You: "Well yes, but they-"
Judge: "And today, months later, they are still hosting for others, right?"
You: "Well, yes but they-"
Judge: "Case dismissed.  Next plaintiff."


Next two words after judge: "how much"


And yet every day that others get said services as agreed your criminal accusations look like nonsense.


Oh my, you read the future from bent plastic?


https://i.imgur.com/lI37y37.jpg

Now I'm not an esteemed criminal justice major like yourself, sir, but that to me looks like $200,000 worth of means and intent to deliver.  Which would make this a civil/contractual dispute.


Reckless enough that they didn't have a licensed electrical engineer design the electrical.  Oh wait, they did.  Oh well, reckless enough that they didn't have a licensed HVAC contractor design and build the cooling.

Oh wait.  They did.  What part of this is not a civil/contractual dispute, exactly?

This is not ASICSPACEs property. They are managing this, but it is not theirs. They own jack shit and their net value is close to 0.

You know far less than you should, if your claims are true and you know them so well.

Lets agree to disagree and be done with it.

I never said it was theirs, I only said it was an intent and means to deliver.  Asicspace is contractually bound to that space and it to them, which definitely passes the bar for an intent and means to deliver on contracts.  If you think being pedantic about it matters, I don't think a single big Bitcoin mine operator owns their land, at least none in central WA.  Amazon.com doesn't even own the buildings they are headquartered in, none of the entire lake union campus, and they don't manufacture any of the Amazon products they sell.  Microsoft didn't even have an operating system when they sold one to IBM.  None of that has ANYTHING to do with whether someone intends to or is capable of delivering.

I know far more than you have been told.  Others you esteem highly are not the angels you think they are.  But I'm sure they didn't tell you that part of the story.

** Going back to clarify for someone reading this later, I'm not saying Asicspace did right or good here or that I support them or don't.  All I'm saying is I know they don't have criminal intent, and I know they have not broken any laws.  I'm almost certain that the temperature didn't get hotter (or equal to) other functional mines I've seen(big and small).  I know they have cooling improvements coming.  I also know they have had network outages.  I have no opinion on whether they did or didn't actually damage or contribute to the damage of the hardware. **

Quite the thread...

In case it wasn't already said... Criminal and civil law are not mutually exclusive..

Is it a civil/contract dispute, without a doubt and the limited facts I have seen from the OP, and another customer I have crossed paths with show there may be a case..

Is it criminal, perhaps if a pattern of willful misconduct can be shown but that is a much higher bar to cross and it will be tough to claim without hard evidence, which in itself might be hard to come by..

For any criminal prosecution you need to prove intent, and no matter how incompetent they may have proven to be, you will never ever get any D.A to file criminal charges in this type of dispute. To think otherwise is criminal in and of itself.. IMHO

Yep, and this is virtually impossible, at least with the given evidence at hand and the limited availability of other customers willing to sue (they have intends, but fear the costs which would add on to their losses).

So since i did not want to remove the NEUTRAL rating for you i now get you going after me?

Its clear why i rated you since i linked the problem with the rating. And you still werent able to explain how someone would have to be refunded for a miner that he rated as perfectly fine, ran it for months until it was worthless, and then got the idea that this miner was so broken that it could have not ever been sold. Then starting a campaign and tries everything to move dogie for a refund.

Whatever, i dont want to go about this in this thread. Maybe you should consider how much worth is in going after others you blame for your own errors. You look like you have enough time to create something constructive instead.

You're also better off sticking with what you know and not arguing with Jared. I wouldn't put his working knowledge of creating mines too far above the people he left behind who cooked your units, and I think even you can see he's here more to argue than to solve anything. Circular logic and speculation, are exactly that. Good for you for having the balls to stand up to these folks. Of course the twits here are going to poke all kinds of holes in what you're doing. Fair point though, you won't see a criminal prosecution out of this, so go for civil. Good luck.

Thanks for the advice and the heads up !

Did Toomim Brothers already give a timeline when first, or the complete, results will be available?

We essentially completed our investigation of the damaged units a week ago, but now we're busy trying to repair them (and the PSUs that ASICSPACE included with them) and to install another 100 S5s, so we haven't had much time to comment here. We kept CryptoCoin2015 updated about the progress of our investigation as we performed it. Here is some information:

Electrical problems

About 40% of the hashboards are damaged and defective, and do not hash at all. My initial count was 89 damaged hashboards out of 200, but I think that was a slight overestimate because I made that count on a weekend and misinterpreted a pile that one of my employees had made. I'll have to make another count soon, but the total is probably closer to 70 damaged hashboards. There may be another half-dozen hashboards that still hash with some dead ASICs. We did not observe any otherwise dead hashboards working if we reduced the clockspeed.

About 30% of the damaged hashboards had visibly burned or popped SMT G337 2V capacitors. This means about 12% of all of the hashboards. Many had several capacitors damaged on a single board. We didn't count all of these; this is just an estimate based on a small random sample.

The resistance across the 12V and Gnd pins for a disconnected hashboard should be around 56 to 58 ohms. This translates to about 4 ohms per stage. (I think it's actually around 4.5 ohms per stage with a parallel resistance of ~330 ohms from 12V straight to Gnd, but that doesn't matter much for this.) All of the working hashboards were in this 56 to 58 ohm range. All of the broken hashboards had lower resistances than this. About 30% had resistances close to 53 ohms, suggesting that they had one bad stage. The rest had resistances below 50 ohms, and about half of them were 30 or below. This suggests several bad stages is typical for the failed hashboards.

We measured the voltage across each stage for a few different hashboards. Here is one such:

15-14:   0.990
14-13:   0.953
13-12:   0.055
12-11:   1.023
11-10:   0.999
10-09:   0.780
09-08:   0.360
08-07:   0.994
07-06:   0.948
06-05:   1.054
05-04:   0.477
04-03:   1.017
03-02:   0.995
02-01:   1.061
01-gn:   0.478

All of the working stages have nearly equal voltages. On this hashboard, those voltages are around 0.980 V +/- 0.070 V. On a good hashboard, they are around 0.800 V, +/- 0.070 V. The voltage abnormalities indicate failed stages. I don't have the resistance measurements recorded, but they were similar, with powered-on voltage per stage corresponding closely with powered-off measured resistance. Several stages were measured at 0.8 ohms, which is the same resistance I measure when I short the leads of the multimeter together. (Multimeters typically are not good at measuring very small resistances, due to the additional resistance of the test probes and leads.)

All of the visibly damaged capacitors were on stages with abnormal stage voltage and stage resistance. Not all stages with abnormal voltage or resistance had visibly damaged capacitors. Removing and replacing damaged capacitors did not fix the abnormal voltage or resistance. (I did not test to see if they hashed after replacing the caps; I just assumed they didn't.) I also tested three severely damaged capacitors after removing them, and despite the crazy physical damage, they still showed the correct electrical characteristics (320 to 370 µF capacitance, DC resistance > 10 Mohm). This suggests that the functionally important damage was to a component other than the capacitors. The most likely candidate is the ASICs themselves.

The G337 2V capacitors are probably aluminum polymer capacitors. Those are typically rated for 105°C operating temperatures. This is lower than the 125°C silicon operating temperature for most mining ASICs and most other discrete power components. I think it's possible that a cooling failure could have caused the capacitors to explode before or at the same time as damage to the ASICs occurred, even though the capacitors were not themselves generating any heat.

We overnighted four bad hashboards to Bitmain Warranty in Denver, CO. I haven't heard back from them about those specifically, but my guess is that they discarded them as unrepairable.

Plastic shield deformations

Most of the S5s we received from ASICSPACE have some deformation of the plastic air guide shield panels. The severity varies widely. There seems to be a correlation between the deformation of the plastic shields and whether the hashboard is broken. There may also be a correlation between deformation and presence of damaged capacitors. I haven't made a close study of that, though. Mostly, all I know is that the shields we took off when investigating broken hashboards tend to be worse than the shields that we left on.

It appears that the worst point on the plastic shields is on the "tail" end (exhaust side) of the miners. The deformation is typically greatest (i.e. shortest radius of curvature) in between the screws. The top edge (where the PCIE connectors and control board are) is also heavily deformed on many machines. In all cases, the deformation causes the shields to bend away from the case of the Ant. In a few cases, the plastic has a wavy appearance in between the screw holes up to approximately 1 cm in from the edge, suggesting that the plastic there had been stretched or elongated and had caused the surface to take a non-linear path between the two fixed screw points. None of the S5s that have been in the Toom.im facility since January show any hints of deformation like this. The only machines like this that we've seen were in ASICSPACE during early Aprli.

Most of the shields have no scuff marks or indications of physical trauma. The deformations are smooth, with no creases. I do not think these deformations were caused by contact with a solid object. Deformation due to air pressure would explain these deformations in every case I've looked at, as long as the plastic were soft enough. I'll describe how I think that happened after I mention some of what I've seen about the environment in which they were operating.

First-person observations

When I visited to pick up the S5s, ASICSPACE's cold aisle was negatively pressurized relative to the outside air. The hot aisle was positively pressurized relative both to outside and to the cold aisle. As a result, the cold aisle was also by no means "cold". Near the cold air supply ducts, the air felt like it was about 15°C. In most of the rest of the cold aisle, it felt like it was 35°C. When I first arrived, I walked past a gap in the cold aisle containment where an Antminer S4 had previously been. The velocity and volume of airflow through that hole was comparable to the airflow coming out of their cold air supply duct outlet, except a lot hotter. I think it felt like about 500 to 1000 CFM through that hole. As we removed S5s, and more holes appeared in their cold air containment, the velocity through each hole decreased substantially. ASICSPACE had noticed this pressure difference, and as a way of mitigating it had set up 9 air ducts (approx. 0.5 m in diameter each) going from their garage door to their cold aisles to serve as supplementary air intake. Note that in proper cold aisle containment design, the cold aisle should be positively pressurized, so ducts like this would normally let air out (and thus normally would not be used). I also spoke with Robert about this, and he was (fortunately) aware of this problem. The ducts looked like they had been hastily added, likely within the previous week. While I was there, their network was experiencing severe problems, causing a large proportion of their machines to stop hashing (and thus not produce heat). It was also not a very hot day, and it was evening when I arrived, making it about 11°C cooler than the daytime high the day before. I can only imagine what their facility was like during the weeks prior.

I did not see any significant exhaust fans installed at ASICSPACE to remove hot air. There were some small fans mounted above the hot aisle, but they were unducted, small, and not very numerous, so I guess they were not significant. There may or may not have been some exhaust fans on the roof of ASICSPACE. From what I've heard through other channels, they chose not to install exhaust fans, and were instead relying on the stack effect to move heat up through their tall (30m?) building and exhaust through natural convection at the top. Unfortunately, the stack effect relies on the interior of the building being hotter than the outside temperature, so when the outside temperatures increase to 30°C, their interior temps would have to rise too. Since they're pre-cooling with evaporative coolers, this means that the air inside their building would have to recirculate several times until it was enough hotter than the outside air for the stack effect to blow it out. However, if they have 300,000 to 800,000 cfm of exhaust fans that I didn't see, this paragraph is irrelevant. (Edit 5/6/2015: We visited ASICSPACE again today, and Robert showed me their exhaust system. They do not have exhaust fans, and are relying on the stack effect plus (in principle) positive pressure from their intake fans.)

The containment sealing system (which looked pretty tight, to be honest --  kudos to ASICSPACE for that) sealed off the exhaust end of each miner, with a gap between the case and the edge of the sealing panel on the order of 1 or 2 cm.

The S5s were installed next to pairs of S4s along both sides of the cold aisle on each of the shelves. This indicates that the S4s and S5s were competing with each other for airflow. The S4 is a sealed tunnel with 4 fans arranged in a push-pull configuration -- i.e., 2 in parallel by 2 in series. Placing fans in series multiplies the amount of static pressure they can produce, and also allows them to maintain their airflow quite well when working against a significant positive pressure gradient, but does very little for the airflow when working in a neutral-pressure regime. This means that the S4 fans were able to to pressurize the hot aisle quite effectively. The S5, on the other hand, is an open semi-tunnel configuration with a single fan in push configuration. The semi-tunnel has large gaps near the exhaust end through which air can escape out the top, as well as small gaps on the bottom.

The power supplies which ASICSPACE had obtained for these S5s were the DPS-800GB using the Gigampz breakout boards. These boards are miswired (http://www.rcgroups.com/forums/showthread.php?t=1301679&page=4) to connect pin 30 (voltage adjust pin) to the 12V rail, which increases their output voltage to around 12.80 V (no load). Power consumption for the S5s' BM1384 (as with most ASICs) is proportional to frequency times voltage squared, so the 6.6% higher voltage should result in 13.8% more power consumed and heat generated.

What I think happened

As the single S5 fans were unable to compete with the doubled S4 fans in terms of pressure output, the airflow from the S5 fans would instead curve out through the gaps in the S5 pseudo-tunnel. Meanwhile, the positive pressure from the hot aisle may have been strong enough to cause airflow to go in reverse from the hot aisle into the "exhaust" port of the S5, and then out the top of the miner. This retrograde flow was likely strongest in the side areas, in between the hashboards and the plastic shields. These two effects caused the interior and side spaces of the S5s to get very hot and positively pressurized. The heat caused the plastic sides to become soft and "plastic" (in the non-elastic sense of the word). The positive pressure then stretched and deformed the heat shields away from the mounting screws. The deformation caused a small gap to appear at the tail end of the shields in between the two screws. This gap allowed for a very short path -- approximately 3 cm -- for the air to go from the hot aisle to the cold aisle, by entering the tail of the miner through the side slots, making a roughly 150° turn to curve around the steel frame (and partially bouncing off the plastic shield), and then exiting backwards/sideways through the tail/side gaps in between the screws. This air was the closest to the hot aisle, and thus the hottest and highest pressure, and consequently caused the greatest deformation, enlarging these gaps considerably over time. Additional airflow that came in through the side slots would have passed out the top of the miner. Though this airflow was greater, the size of the gaps on the top were much larger, causing lower pressure differences, which made the radius of curvature of the deformation in that area smaller.

The intersection of the anterograde and retrograde airflows near the center of the miner caused airflow there to be relatively stagnant. This, combined with the higher temperature of the retrograde flow, would have caused very high temperatures about 3 to 10 cm away from the miner's exhaust port.

The Antminer S5 has a bug (linked elsewhere on this page) in which the miner's fans will stop immediately when cgminer dies (or when the network is disconnected), but the hashboards will continue to generate heat for several minutes. Given that ASICSPACE was having frequent and persistent networking problems during this time, I expect that to be a contributing factor in this case. However, I think that is at most a contributing factor, since many people have reported that bug in the absence of heat damage, and I haven't seen any other reports attributing actual heat damage to that bug.

Based on what I saw when I visited and what I know had been changed recently at ASICSPACE, I estimate cold aisle temps were likely 20°C hotter during early/mid April at ASICSPACE, and likely reached about 55°C. I have also heard people allege to have seen 57°C intake temps on their miners at ASICSPACE at one point (I think that was March), although I can't mention the source for that right now so you should treat it as a rumor. I have also seen screenshot evidence from a KNCMiner device indicating temperatures in the same ballpark. With mostly S4s and a few SP35s, plus with the fans working against positive pressure and consequently having reduced forward airflow, their delta-T might have been around 10°C, or possibly a little higher. I thus estimate their hot aisle temps at around 65°C. The air passing from the hot aisle through the S5 heatsinks would have gotten even hotter before hitting the side panels; perhaps 15°C hotter. Many plastics have glass transition temperatures around 80 to 120°C. ABS, for example, is 105°C. I thus think it is plausible that the plastic side panels were heated close enough to their glass transition temperature that air pressure differences caused them to permanently and plastically deform. Having a 55°C cold aisle with strong positive pressure in the hot aisle would also explain why the ASICs and capacitors in the S5s would have burned out in such large numbers, especially if the 80°C protection was bypassed either due to a Bitmain bug or due to an attempt to get the S5s to hash despite the poor working conditions.

Note: As the head of Toomim Bros, I am a competitor of ASICSPACE, and clearly have a conflict of interest in this case. Apply salt liberally before hashing.

Extremely thorough explanation Toomim.  I'm impressed, hats off to you.


Fits; Temp around your time appears to have been around ~61F, high temp was ~80F.  A few assumptions went into that(like which day).

http://mesowest.utah.edu/cgi-bin/droman/mesomap.cgi?state=WA&rawsflag=3


This piece I did not know until after I had posted my previous posts, and singly explains the failure.  In a worst case scenario, this pressure/counterflow would have caused the S5 fans to basically stop working, as you described.


Fits with what I know except the plastic shields part.  I legally can't state why, but it doesn't make much difference.


Temperature rise is relatively linear even if it doesn't feel like it and the damage caused is not linear.  You noticed around ~95F in the cold aisle @ ~61F outside.  On a 81F day, that would be ~115F(~46C).  Maybe a bit more, but the damage to these miners(and why other miners were not similarly damaged) was primarily because of the airflow around them in relation to the S4's you mentioned, not primarily because of the intake temperatures.


Polyethylene has a glass transition temperature of -80 to -120C.  Polyethylene is malleable at room temperature and cheap to mass produce, just like the S5 plastic, which makes it a reasonable assumption unless Bitmain can confirm.  Regardless, the temperatures that caused the deformation & damage would have largely been due to the "stuck" stagnant air that you described, not a direct result of the hot aisle temperatures. (i.e., primary cause = pressure)  Other minor information I now know that I can't share agrees with that.

I observed cold aisle temps around 35°C. My estimate of cold aisle temps around 55°C in the weeks before I visited was based on several factors:

1. There were network problems when I visited, resulting in less heat generated. It looked like about 50% of ASICSPACE's miners were off while I was there due to the network problems. If the 50% estimate is correct, then the delta T versus outside might be sqrt(2) to 2 times higher, depending on how much higher inside temperatures assist the stack effect. As the cold aisle temps were already 20°C hotter than outside, this alone might be enough to produce cold aisle temps of 55°C even on a somewhat cool day. However, my 50% estimate could be off. Also, many of the machines were S4s, which may also produce heat while the network is disconnected. For my estimate, I think I was generous and presumed that the network problems reduced temps by 8°C.

2. The nine flexible ducts supplementing the cold aisle with outside air at ambient pressure looked to be hastily added. I presumed they were added within the last week before my visit, and were not present when the damage occurred. I estimate they added about 20% to the airflow into the cold aisles. Roughly speaking, that should have decreased the cold-aisle/outdoor delta T by 20%, which would be about 4°C on the day I visited, or possibly as much as 8°C on other days. There may have been other similar quick fixes that I did not notice. I used 2°C in my quick estimate.

3. When I visited (6pm to 11pm), the outside temp was 16°C or lower. The high the day before was 26°C. That's 10°C. I think there was a day or two in the week or two prior that was even hotter.

8°C + 2°C + 10°C = 20°C
35°C + 20°C = 55°C

Other effects: as the temperatures rose, many machines may have turned off. On the other hand, reliance on the stack effect may have worsened the effects of a high outside temperature.

My math might not be right, of course. It's just an educated guess.


The damage we observed was not linear. In terms of deformation, pretty much all of the plastic panels were deformed. Most of the panels were only slightly deformed, maybe 1 or 2 mm per panel. Other panels were very heavily deformed, with about 20 mm of deformation at several different points on each panel. If the maximum temperature seen by each miner was normally distributed, and the amount of deformation as a function of temperature is something like D(T) = e^T, that would fit our observed distribution of deformation pretty closely.

We saw something similar with the amount of electrical damage on each hashboard. Most of the hashboards had either zero bad stages or more than two bad stages. This could also be caused by a domino effect, though.

It's also worth mentioning that the damage to the two hashboards in a single miner were correlated. Most of the miners we received either had two bad hashboards or zero bad hashboards. The number of machines with exactly one bad hashboard was less than I had expected.


I disagree. I do not see how air pressure alone could have caused this deformation. I just went over and tested, and using my finger, it takes about 20 pounds of force to permanently deform a S5 side panel at room temperature by about 3 mm. That would mean you'd need close to 1 atmosphere of pressure difference across the panel to deform it from pressure alone. Typical HVAC system pressure differences are on the order of 100 Pa (0.015 PSI). Axial fans typically have static pressure capabilities around 300 Pa. If we are very pessimistic and say that the hot aisle was 600 Pa higher pressure than the cold aisle due to the two S4 fans (i.e. the S4s had zero net airflow, 100% static pressure), then that would mean about 0.06 newtons per square centimeter, or 0.08 PSI. I think temperature must have been a large factor.

Polylactic acid has a glass transition temperature of 60°C to 65°C. What are the panels actually made of? I don't know. Also, even below or above the glass transition temperature, plastics will become softer and more plastic as temperature increases. The Vicat softening temperature for HDPE is about 70°C, for example. The 0.08 PSI might not be enough to deform it at all at 20°C, but it might be enough to deform it by 1 mm at 70°C, and enough to deform by 10 mm at 80°C. I do not feel inclined to test this right now, but if this goes to court, it would be simple enough to verify.

I don't mean to offend you on this point; most of your work is very thorough.  But your temperature estimations are pure guesswork.  Even approximating temperature in the midst of high airflow/turbulence/heat generation situations is extremely difficult.  Temperature/thermal airflow software is extremely expensive exactly because it is so hard to approximate.  Most of those formulas and estimations regarding that aren't going to hold up to any critical analysis at all.  Delta T estimations for a point in the past with multiple unknowns, without actual measurements from a velometer or thermometer are obviously going to be terrible at best.  And since this guesswork isn't needed to draw the conclusions... why?

It was hot.  Too hot.  Definitely less than 100C.  Less than 70C.  More than 40C.  We can agree there for sure.  Even if we could calculate them, we don't need more accurate numbers based on the conclusions you covered or the ones I reached...


Which would be why I said it is not linear...


I wasn't saying that air pressure did the bend, air pressure only influenced the direction of bend at best.

I posit that air pressure, primarily, caused the runaway temperatures localized to a certain point on the boards.

Think of it this way; take two S5 miners in a chill, 15C room.  Point the exhausts at eachother directly and start moving them towards eachother.  Eventually at a certain point, maybe around an inch apart or less, the restricted airflow between them will cause the boards to overheat and fry(add time); The air temperature of the room can exacerbate the problem and make it happen faster, but it is otherwise irrelevant to the mechanism.  That is what I think happened from your description, but rather than an S5 inches away it was dozens of S4's blasting towards S5's from a few feet.

The increased local temperature that resulted from this, likely within 3-10cm from the exhaust(as you mentioned) is sufficient to explain the plastic warping and board failure without any other input.  To explain the mechanism I'm envisioning I'll have to take a big leap- Ever seen perler bead art?  Looks like 8-bit graphics.  Anyway, the beads are put on a plastic pegboard and then ironed.  They are a flat board with flat beads on them that are ironed on a flat surface.  The result, if the user isn't careful, is a warped and useless pegboard.  The heated plastic undergoes thermal expansion, but it is not uniform, and the plastic deforms.  I couldn't find any pictures to show this, unfortunately.  But the concept is exactly the same reason you can't mix aluminum and copper wiring connections; over time and through heat cycling, the aluminum will disconnect no matter how much torque you apply.  Same mechanism.

Specifically to this case... I looked up the coefficient of thermal expansion for PE and found from 80F to 135F you get ~0.5% expansion.  Putting that in a right triangle gives you a bowing outwards of about 2.5cm from a ~17.5cm plastic strip.  You said you measured 2cm at worst, so that's about right.

None of that even requires excessive intake or exhaust temperatures(whether present or not).  That, combined with other things I know and the difficulty of calculating or approximating exhaust/intake temperatures, leave me stating 'The cause was localized cooling system failure on the miner caused by high incoming exhaust pressure, exacerbated by the fact that it was "too hot".'

It helps with forensics to have multiple independent sources of information in order to cross-check facts. I first had an estimate of 55°C peak cold aisle temps based on my in-person observations. I later heard a report of ~57°C intake temps on an SP10 from another party who may have an axe to grind with ASICSPACE and might be lying. I then saw a screenshot of a KNCMiner device in their facility showing temps that would indicate intake temps around 50°C. Finally, I observed plastic deformation patterns that could be explained by hot aisle temps in the vicinity of 60°C-90°C.


Someone else might read this wondering how hot things need to be to damage an S5. I don't want people to think that they magically start to burn up as soon as the intake temperature exceeds 40°C. Accuracy is useful even when it's not necessary for the immediate task.


I agree, mostly. I think that runaway temperature was mostly restricted to the hashboards and the intra-heatsink space. It's worth mentioning that all of these S5s were of the older variety that do not have any heatsinks between the hashboards and the side panels. With minimal surface area, very little heat is transferred from the hashboards to the air that passes through that space. Typically, I see a delta-T of about 1 or 2°C for that air when airflow is not obstructed. In the ASICSPACE case, that may have been 4x higher, but that would still only have been around 8°C. Furthermore, the pattern of deformation suggests that the deformation was greatest where airflow was highest, not where it was most stagnant. This implicates ambient temperatures more strongly as the dominant factor in the plastic deformation.

(Minus the expansion of steel and aluminum, which is much smaller.)

Most of the plastic deformation indicates a shear stress, not compressive stress. The only exception to this is the buckling that was observed in a small number of the most severely deformed plastic. The buckling of the plastic occurred entirely along the short axis of the plastic shields. If thermal expansion were the culprit, I would expect to see buckling along the long axis as well, especially along the line in between the front and back screws. That line is the least deformed part of the side panels. The bottom edge is more securely attached than the top edge as well, so if the main cause of deformation was thermal expansion, one would expect the bottom edge to show more buckling and deformation. The opposite is true. The greatest deformation occurred on edges which I expect had the greatest airflow (evenly along the top edge, plus the back and front edge near the holes for the tail exhaust and fan).

Much of the deformation occurred on the top edge, above the highest screw attachment point. That edge typically is relatively straight, but it sags outward and downward. Since this edge was not being squeezed, and if it were it wouldn't cause the edge to sag like that, I don't think thermal expansion is a satisfactory explanation. Thermal softening combined with airflow is.

The apparent buckling I observed on a few panels might actually have been fluid dynamic effects similar to ocean waves or sand dunes being created by wind rather than actual buckling.

Something just occurred to me. If the airflow was anterograde but slow through the whole intra-heatsink space, the air coming out the exhaust port would be very hot. The side spaces would not maintain their pressure as well, so airflow there would be more likely to turn retrograde. This would pull the hot exhaust air around 180° back in the side ports before flowing either out the top of the miner or through the gap in the panel between the screws. This could explain the pattern of deformation pretty well.


I think high pressures alone might have been enough to cause the ASIC damage observed, but I doubt it. I think high temperatures alone might have been enough to cause both the ASIC damage and the plastic deformation, but I doubt it. I think the evidence is clear that both existed. Indeed, it's difficult to have a positively pressurized hot aisle without getting hot air recirculation and overheating everything. I don't think there's any reason to try to pin the blame on one factor.

jtoomim... im impressed from this investigation. Looks like you do this properly. And i like that you really seem to be independent since you dont follow the words of one side in this easily. :) If i would be in mining business i would definitely use your service in case i need it.

True but equally frightening.  A miner burning up is a factor of a lot more things than intake temperature as you agree with the two S5's example.  Normally we talk about "intake" temperatures because it simplifies the discussion down to a workable set, but doing so makes some basic assumptions, such as that exhaust airflow isn't constrained.  As soon as someone starts violating those assumptions, everything changes.

It is possible(though not practical) to get a S5 to run safely with a 65 degree intake as well as to fail with a 15 degree intake.  We do the readers no favors by glossing over this reality in my opinion.


Remember when talking about temperature under cases where normal assumptions are violated, it is useless without also talking about specifically where the temperature is.  The middle of the plastic sheets didn't warp because the middle didn't get so hot.  The edges did because the edges touched the metal, a better heat conductor, and the exhaust edge specifically had higher heat. In the pictures you have provided, one perhaps 2 of the plastic have deformations on both sides.  6 or 7 have deformations on one side with far smaller or no deformations on the opposite side.  Of all of the sheets in the picture, one shows minor top-edge deformations, one shows a lot of top edge deformation, and the only other one with top edge deformations is very close to the screws/my proposed hot spot.  6 show no top edge deformation at all.


To be clear, I don't believe airflow played no part in what we see, that would be silly of me.

... and thats why we host 200 S5 with Jonathan and hope, that ASICCRAP bites the dust...

Thanks Jonathan for the hard and dedicated work you did put into this.

Hm... i only now, when you mentioned this, realized that toomim brothers are a competitor of AsicSpace. Even though the investigation sounds trustworthy... this can hardly be seen as an independent investigation then. I thought they are hardware nerds who can check those things out. But letting the competitor decide if the competition made an error is not really something that comes to mind when one wants an independent investigation about the source of the problems. I dont say anything, i only point out a conflict of interests.

Yes, I have a conflict of interest. Sorry if that wasn't very clear. I'll edit a note of that into my initial analysis.

Edit: It's also worth mentioning that I used to be business partners with Robert and Damir, and my brother and I originally planned to design and fund most of their datacenter for them in exchange for free hosting for our SP30s and partial ownership of the business. My brother and I decided that we did not want to be involved with them (mostly because of Damir), so we broke off and forged out on our own. They claim we acted inappropriately and dishonestly, and that our original intent was just to deceive them and use them to find a location to build our own facility. I claim that they were not competent enough to continue collaboration with, and that we didn't realize this until after we were heavily involved with them.
You can read about that history here (https://bitcointalk.org/index.php?topic=622998.msg8565456#msg8565456).

In other news, ASICSPACE fired their CEO, Damir Kalinkin, last week. I believe the exact process was that the shareholders held a vote of no confidence, then cleared out the Board of Directors (of which Damir was a member), and then the new Board fired Damir. In my (biased?) opinion, Damir was the worst part of ASICSPACE, and responsible for the most egregious misdeeds I have heard attributed to them. With Damir gone, I expect ASICSPACE to improve considerably. Robert is a nice guy, and I like him. From what I have seen, Robert wants to treat his customers properly, unlike Damir, who treats customers as a resource to be exploited.

However, I do not see how Damir can be responsible for the heat, airflow, and networking problems that ASICSPACE has had. He is not a technical person at all. Those issues were due to the decisions made by the engineer and contractors who built the facility and the person who oversaw them, Robert.

When I was at ASICSPACE earlier today, the conditions were considerably improved. The nine flexible ducts that were previously just neutral pressure outside-air intakes have now been connected to a 125 ton portable AC unit on a trailer, powered by what appeared to be a 400 amp 480V 3-phase connection. I didn't go into the cold aisles, but they appeared to be only slightly negatively pressurized relative to the hot aisles, maybe around 10 or 20% of what they had been at before. The general building interior (contiguous with the hot aisles) was strongly positively pressurized relative to the outside air, and the airflow out the open front door was very strong, maybe 50,000 to 100,000 CFM. The exhaust air from S4s felt like about 40 to 45°C, suggesting cold aisle temps around 30 to 35°C. Still high, but no longer unsafe. Most of the miners in their facility appeared to be on and hashing, although I did hear S4 beeps coming from somewhere.

ASICSPACE appears to be making a good faith effort to maintain proper operating conditions in their facility. They are putting a lot of money behind that effort. Unfortunately, they are spending it on the wrong things, like the air conditioning unit instead of a bunch of high-throughput fans. They don't need colder air, they just need more fresh air. However, the AC unit they are renting did come with a large fan inside, so there's that.

right, like e.g. paying back the damage they did to our equipment and the fees for services they never rendered to us ;)

And why the fuck would there have to be an independent investigation anyways? Because you would like it?
If ASICSPACE wants this investigation, then they should do so and PAY FOR IT.

We kindly asked Jonathan to assess the damage to our equipment and the cause, because we need to decide whether it is ASICCRAP to sue or BITMAIN. What right do you have to claim an independent investigation? If you want a different consultant doing an "independent" investigation, go ahead. Hire some consultant and get it done. For now, you got to live with what is on the table and you either take that or you don't.

But please don't come up with the call for an "independent" investigation, as this is needed. It does not alter the tuth in Jonathans statement by an inch, if he is now rsponsible for our equipment or not. The damage was done beforehand.


truer words have never been spoken. You always come up with the most ridiculous claims and the request for a observatory stance like this is Kindergarten.

Fuck this. Damage has been done, ASICCRAP is not paying up and customers of theirs are not getting reimbursed for the damage.

It is a good thing, that Damir is gone. Now, only 2 more are left to be dealt with.

Robert van Kirk and Geoff Smith, two fucking cowards who cannot deal with taking their share of responsibility like a man should.

I think it is clear yet again that Seb has overstepped.

http://www.troll.me/images/monocle-guy/well-said.jpg

Interesting, it is good to finally know the rest of the story.  Edit- Hmm, found more info I didn't know.  Removed negative comment.


Yes, being an asshole to longtime highly trusted members of the community is going to go REALLY well for your war on Asicspace.


You might want to review your hosting agreement with Toomim.  No hosting company anywhere is going to reimburse customers for miners(according to the contract).  They can't.  The margins are too slim, and there's too many things that can possibly go wrong with miners, many of which(not all, and not necessarily in this case) aren't under the control of the hoster.  Some may reimburse in some situations, but it won't be a standard thing except perhaps with the highest cost providers(at a net loss to their customer base).

For all your kicking and screaming, if you take this to court, you will almost certainly lose.

So your intent as you have made clear is to make Asicspace suffer economically.  Might I suggest that if you really want to tarnish Asicspace's reputation, being an asshole about it will only make people doubt your statements?  Or did you not notice Toomim's statements, being a stand-up guy(respect): "When I was at ASICSPACE earlier today, the conditions were considerably improved."  Are you going to rant and rage at toomim now for saying something positive like you did to SebastianJU for saying something neutral?


Hmm....

I really dont know why you are so aggressive and attack me. I got the wrong impression that you wanted to do an independent investigation. Im not so much into mining that i would have known who they really are.

Its really a pity that the behaviour on this forum lately turns to people being the most aggressive they can. Maybe its because of AirWolf who was able with extortion, threatening relatives and so on to get a refund from a scamming company.

I would prefer you speak normal with me. I admitted my error and theres no reason that you feel instantly attacked the way you seem to feel. Only because you feel i demanded an independent investigation doesnt that make your statement valid. It says more about you than me.


Because i dont have the time to investigate every username i come up on here? I admit my errors. It would be great if others would be able to do so too.

Hopefully you can nego a settlement it is pretty clear ASICSPACE is at fault in this regard given all the retrofitting going on there to REDUCE heat. ASICSPACE should simply compensate you immediately and end this without anymore delays. That is what good businesses do.

It looks like you want to preserve your little bit of green on your main account. I warn you only once now. Stop the hate!

If you only could see the irony in your statement on so many levels then you would laugh. But i think i wont waste more time with you. If you still want to keep me as part of your crusade then you might want to do it elsewhere since youre offtopic in here. And dont forget my previous post since i wont watch that too long of course.

Hello everyone,
my experience with asicspace isn't great either.
in january i sent in an mat miner, which mined there for 1 month.
i did a payment for 6 months and after 1 month my machine stopt working ( in my opinion heat).

i had 2 mail contacts after that and haven't heared from them again.

they are investigating????

in my opinion not a good company.

the hole mining industry is screwed up because of this dishonest companys.

I visited ASICSPACE again on Friday, June 26th. Overall cold aisle temperature was similar to what I had seen on May 17th, probably in the upper 30s. Their cooling system had been improved a bit since then, mostly with an evaporative cooling pad in their garage door. However, their changes were only about enough to keep pace with the increasing summer temperatures. Their hot aisle was still pretty strongly positively pressurized, and any holes in their containment (or miners with their fans turned off) resulted in a significant backflow of hot air. It was about 36°C outside when I visited, but the forecast for the next day was for a high of 41-42°C (possibly the hottest it will get all summer), so I recommended that they impose rolling blackouts to keep things under control. I think Robert actually did this.

I said on May 17th that I thought their facility was no longer unsafe. When I said that, I thought that they would have made more progress by now. Temperatures in their facility is close to the border between safe and unsafe. I don't know which side of the line they are on.

I've heard they're getting 50k to 100k cfm more airflow in the next two weeks. If that materializes, that should help. Whether or not it will be enough I cannot say.

Can we get some images of their data "hut"? :D

I have several years experience in building these types of area for customers- just like asicspace.
i also know the lack of effort and responsibility for these types services, and the overall general well being of the customers equipment, (it is what insurances are for and that is their mentality), So... I see this whole thread as a stupid is as stupid does, when related to a business decision.
We all have probably been there, and some learned and some never learn
I just heard of a nightmare with another mining host over in that area, and they had gave people about a week to respond to their being booted from a facility, locked up their customers miners, ect.
I believe asicspace quickly jumped on their cusotmers on this mining hosting situation

Are any ASICSPACE customers here?  My hosted machines stopped hashing Thursday night.  When I entered a support ticket, I got a call back from Robert on Friday indicating that they were moving my machines to a new facility and they should be hashing again within 48 hours.  The machines are still offline now.  Just wondering if any other customers are experiencing the same issue?

Does this place still exist? Was anyone ever able to reach them? Their website is still online but their last post was 5 months ago. Looks like this is definitely a hosting service to avoid?

are there other miners who got scammed or have burned miners at asicspace, i am be trying to get (as they say) my broken miner.
I am asking them sent it back for 5 months now.
Every time a diffrent story why they don't sent.
Trying to set up a lawsuit, looking for more people that have experience with this.

I've got some Sp20's, S5's and a S4 running there and I'm happy so far. They are very responsive to questions and will do a hard restart of a miner if needed.

The dashboard feature is good and has only had one small/short outage that did not affect the hashing (just remote access). Once I notified them it was quickly fixed.

FYI I live in Washington and drive there to deliver and pick up miners so they don't have to waste time and cost with shipping. That works out well for me.

But I'll make sure during summer to have tempature settings on and may under clock them if needed to keep them from overheating. That's not a problem now with external (outside) temps in the teens (16F).

I'll keep posting as I slowly build out my mining empire there. 😀

Thanks much

Stay out from ASICSPACE.

I was working with them very well from feb 2016 to september 2016, and they gave me a very good service. After september 16, my S7's started to NOT work, and after write incidence and try to contact them, I couldn't.

I payed them in order to send one of the machines, and the machines didn't get out from there.

In september, my machines started to work badly and they stopped the rest of the machines at the end of the month of september.

I try to talk to them, and nobody from ASICSPACE reply me.

Today is november 2016, and i can not contact them!

DON NOT TRUST THEM! BECAREFUL IF YOU WANT TO START TO WORK WITH THEM! 

What is your insurer - they are offering policies covering bitcoin miners?

I think everyone who has had these problems with AsicSpace should get together.

I have 15 miners there. 1 S9 got 'toasty' and failed, per one of Robert's scarce text messages. I've tried to arrange to get it back so I can get it fixed or at least tinker with it myself, but no luck. It doesn't seem anyone is ever at the facility.

Four days ago all but one of my miners went strange. They mined intermittently, failing over to the last choice pool, with ridiculous rejection rates (As high as 80%). Now only 4 show up at all, with hash rates in the 200 GH range. Numerous texts have been ignored. Yesterday he texted me 'someone will be there Wednesday'. Well, it's Wednesday at 5 pm and there is no change. Here's BTC at $840 and the majority of my hashpower is out for the count. 70 TH/s is down to about 6 TH/s.

========================================
Update: It turns out that a switch failed, and it took 5 days to get a new one due to winter storms delaying shipment.
Seems to be a strong argument that they should keep spares in stock...

Also it doesn't explain why the customer service is so crappy. If they just said on Day 1 that they had a bad switch and needed to send for a new one it would have been a lot better.
I shouldn't have to badger them to find out what's going on. A proactive message to those affected would help a lot in building confidence in this business. Honestly I don't think they care.

Does anyone who had miners there have the address they shipped their miners too?  Is so please contact me, I might be able to help.