Pictures of your mining rigs!

[itod]

pre fire

This cabling looks really amateurish. Sorry for their loss, but if power lines were done like this Ethernet cabling no wonder problems occurred.

[1713533216]

1713533216

[1713533216]

1713533216

[1713533216]

1713533216

[1713533216]

1713533216

[king_pin]

What did they expect, wood and electricity don't go together very well

I bet this will give some scammers ideas:
Before someone wanted to steal peoples coins - "Oh,.. I got hacked"
Now - "Oh God,... everything burned I escaped with my life sorry no coins!"

[klondike_bar]

This cabling looks really amateurish. Sorry for their loss, but if power lines were done like this Ethernet cabling no wonder problems occurred.

The post-fire pictures are taken from the back, and the power cabling does seem to be better organised (though bunching them together neatly may be worse, since it puts all the warm conductive cables tight together so if one melts they all burn together)

I have no clue why the pre-fire picture couldn't have the ethernet cabling nicely clustered and numbered, running in straight lines with the shelving rack - it would be tidier, safer, and easier to service.

[Coopster8888]

A little time on the cable would be nice for service on units, but they don't have to worry about that anymore right now

[Coopster8888]

Mucho ouch.  Is there a link to the story anywhere?

http://bravenewcoin.com/news/bitcoin-hashrate-jumps-then-drops-how/

and worst part is he had NO Insurance
The good thing is that the difficulty might go down

[kingcolex]

Mucho ouch.  Is there a link to the story anywhere?

http://bravenewcoin.com/news/bitcoin-hashrate-jumps-then-drops-how/

and worst part is he had NO Insurance

Not going to be able to get insurance for that kind of hardware and price.

[Coopster8888]

Mucho ouch.  Is there a link to the story anywhere?

http://bravenewcoin.com/news/bitcoin-hashrate-jumps-then-drops-how/

and worst part is he had NO Insurance

Not going to be able to get insurance for that kind of hardware and price.

Ouch, but it would pay for the building at least, but no insurance at all hurts

[itod]

I have no clue why the pre-fire picture couldn't have the ethernet cabling nicely clustered and numbered, running in straight lines with the shelving rack - it would be tidier, safer, and easier to service.

The answer is simple: amateurs don't know how it should be done, they think the way they did is just OK. If they knew better, they would do it properly.

[skizzneena]

After looking at those Thai pictures............. I cant even............Why would they.....Jesus.

[yochdog]

Mucho ouch.  Is there a link to the story anywhere?

http://bravenewcoin.com/news/bitcoin-hashrate-jumps-then-drops-how/

and worst part is he had NO Insurance

Not going to be able to get insurance for that kind of hardware and price.

Sure you can.

Many of us do it all the time.  Just takes the right amount of premium to justify the risk. 

I have a very large business policy on all my equipment.  It's just standard practice. 

[klondike_bar]

I have no clue why the pre-fire picture couldn't have the ethernet cabling nicely clustered and numbered, running in straight lines with the shelving rack - it would be tidier, safer, and easier to service.

The answer is simple: amateurs don't know how it should be done, they think the way they did is just OK. If they knew better, they would do it properly.

Amateur hour ended when they set up and operated (safely and at good temperatures) 450+ units for over two months.
-yes, the web of wiring is ugly, but the obstruction to airflow is pretty minimal, especially as its hard to tell in the pictures but there appears to be a 6" or greater gap between the wires and units.
-power wiring on the back side was cleaner it seems.
-area is clean, freshly painted, and had at least a few fire extinguishers and a shutoff switch.

This was not a novice operation, and IMO/IME there is no problem with using crappy basic shelving - it does the job and doesnt obstruct airflow. I dont think the setup was in any way the problem. the problem was:
1) whatever component started the fire (a unit, a fan, a power bar?)
2) The fact that 2 IT workers were sleeping. If one was awake as they should have been the shutoff switch at the transformer would have been hit and the fire stopped dead in its tracks (possibly without even needing an extinguisher). Apparently it took over an hour for the fire to spread according to the dropping hashrate of the farm.
2b) smoke alarms or other automated systems must have been absent or insufficient.

[darkaire]

I have no clue why the pre-fire picture couldn't have the ethernet cabling nicely clustered and numbered, running in straight lines with the shelving rack - it would be tidier, safer, and easier to service.

The answer is simple: amateurs don't know how it should be done, they think the way they did is just OK. If they knew better, they would do it properly.

Amateur hour ended when they set up and operated (safely and at good temperatures) 450+ units for over two months.
-yes, the web of wiring is ugly, but the obstruction to airflow is pretty minimal, especially as its hard to tell in the pictures but there appears to be a 6" or greater gap between the wires and units.
-power wiring on the back side was cleaner it seems.
-area is clean, freshly painted, and had at least a few fire extinguishers and a shutoff switch.

This was not a novice operation, and IMO/IME there is no problem with using crappy basic shelving - it does the job and doesnt obstruct airflow. I dont think the setup was in any way the problem. the problem was:
1) whatever component started the fire (a unit, a fan, a power bar?)
2) The fact that 2 IT workers were sleeping. If one was awake as they should have been the shutoff switch at the transformer would have been hit and the fire stopped dead in its tracks (possibly without even needing an extinguisher). Apparently it took over an hour for the fire to spread according to the dropping hashrate of the farm.
2b) smoke alarms or other automated systems must have been absent or insufficient.

This is absolutely an operation put together by amateurs, not real datacenter techs or designers.

I see:

No proper fire suppression equipment anywhere in that building, must less one that is appropriate for a data center fire (large electrical fire). This alone would be totally out of code in the US and un-insurable. If a fire marshal saw this they would rage. I got flack for this one at a place I managed IT for because my racks were too close to the wall (which were like that when I got there).

No real power distribution units you would find in a real datacenter that may have saved this mine, as any power surges or failures would be instantly shut down.

Stack after stack of hot SP30's right on top of each other is the dumbest thing they could have done. Do you have any idea how hot these machines are? They are above and beyond what a typical server puts out. They needed space above and below for this reason, as well as fire safety as one machine in this configuration burning up could quickly spread.

I don't feel sorry for these idiots. No trained engineer would do this. They just packed a few warehouses full of hot, power hungry gear with no safety precautions and paid the price for their ignorance to basic data center fire safety and building code. Hopefully this is a nice lesson for the rest of you to be careful and stay safe.

[Hippie Tech]

This must be how/why so many Blackarrow/minersource deliveries were TOA. Toasted On Arrival.

Could SP be BA/MS ?

[Group Bitcoin]

An Inert gas fire suppression system would ruin ROI for any Bitcoin mine but surely if you spent a few million on SP30's you'd buy a few $5 fire alarms and have a security guard?

I think it's an insurance scam but they forgot to buy any first.

You could fry an egg on top of an SP10 I certainty wouldn't put 2 on top of each other.

[mazedk]

An Inert gas fire suppression system would ruin ROI for any Bitcoin mine but surely if you spent a few million on SP30's you'd buy a few $5 fire alarms and have a security guard?

I think it's an insurance scam but they forgot to buy any first.

You could fry an egg on top of an SP10 I certainty wouldn't put 2 on top of each other.

They have/had people on site.. But agreed, a fire alarm setup with linked firealarms would have been nice.. might have saved a lot of their gear.

[klondike_bar]

An Inert gas fire suppression system would ruin ROI for any Bitcoin mine but surely if you spent a few million on SP30's you'd buy a few $5 fire alarms and have a security guard?

I think it's an insurance scam but they forgot to buy any first.

You could fry an egg on top of an SP10 I certainty wouldn't put 2 on top of each other.

SP10 had the heatsinks bolted to the bottom of the frame, heance the case being so hot. the SP30 'hovers' so its case doesnt get remotely as hot to the touch.

and most fire suppression systems would either damage the miners, or are designed for enclosed data centers (you cant just sray inert gas into a windtunnel - it would be pushed out of the building in a heartbeat by all those fans

[itod]

I have no clue why the pre-fire picture couldn't have the ethernet cabling nicely clustered and numbered, running in straight lines with the shelving rack - it would be tidier, safer, and easier to service.

The answer is simple: amateurs don't know how it should be done, they think the way they did is just OK. If they knew better, they would do it properly.

Amateur hour ended when they set up and operated (safely and at good temperatures) 450+ units for over two months.
-yes, the web of wiring is ugly, but the obstruction to airflow is pretty minimal, especially as its hard to tell in the pictures but there appears to be a 6" or greater gap between the wires and units.
-power wiring on the back side was cleaner it seems.
-area is clean, freshly painted, and had at least a few fire extinguishers and a shutoff switch.

This was not a novice operation, and IMO/IME there is no problem with using crappy basic shelving - it does the job and doesnt obstruct airflow. I dont think the setup was in any way the problem. the problem was:
1) whatever component started the fire (a unit, a fan, a power bar?)
2) The fact that 2 IT workers were sleeping. If one was awake as they should have been the shutoff switch at the transformer would have been hit and the fire stopped dead in its tracks (possibly without even needing an extinguisher). Apparently it took over an hour for the fire to spread according to the dropping hashrate of the farm.
2b) smoke alarms or other automated systems must have been absent or insufficient.

It's not just the obstruction to airflow issue with that ethernet wiring, it shows the general sloppiness together with absolute absence of attention to detail. Anyone who wired 450+ ethernet devices like that most probably never wired anything more complex than a home router, it was bound to cause at least some network glitches. Network cables should not carry it's own weight, and certainly not the weight of hundred or more cables meshed into a spider net, with the whole combined weight of all of them carried by whichever is the shortest one of them, certainly giving some tension on the motherboard it was connected to. It sure was not the cause of the fire, but is obviously wrong to anyone who wired a single rack of equipment. We can only assume how they did the power distribution, but looking at the front side it would be surprising if there were not some very elementary errors which directly contributed to what happened.

[molecular]

An Inert gas fire suppression system would ruin ROI for any Bitcoin mine but surely if you spent a few million on SP30's you'd buy a few $5 fire alarms and have a security guard?

I think it's an insurance scam but they forgot to buy any first.

You could fry an egg on top of an SP10 I certainty wouldn't put 2 on top of each other.

They have/had people on site.. But agreed, a fire alarm setup with linked firealarms would have been nice.. might have saved a lot of their gear.

You'd think you'd invest in cable ties before investing in fire alarms, no?

[mazedk]

An Inert gas fire suppression system would ruin ROI for any Bitcoin mine but surely if you spent a few million on SP30's you'd buy a few $5 fire alarms and have a security guard?

I think it's an insurance scam but they forgot to buy any first.

You could fry an egg on top of an SP10 I certainty wouldn't put 2 on top of each other.

They have/had people on site.. But agreed, a fire alarm setup with linked firealarms would have been nice.. might have saved a lot of their gear.

You'd think you'd invest in cable ties before investing in fire alarms, no?

Yup.. but again.. you snooze, you loose.. and so they did.

[klondike_bar]

I have no clue why the pre-fire picture couldn't have the ethernet cabling nicely clustered and numbered, running in straight lines with the shelving rack - it would be tidier, safer, and easier to service.

The answer is simple: amateurs don't know how it should be done, they think the way they did is just OK. If they knew better, they would do it properly.

Amateur hour ended when they set up and operated (safely and at good temperatures) 450+ units for over two months.
-yes, the web of wiring is ugly, but the obstruction to airflow is pretty minimal, especially as its hard to tell in the pictures but there appears to be a 6" or greater gap between the wires and units.
-power wiring on the back side was cleaner it seems.
-area is clean, freshly painted, and had at least a few fire extinguishers and a shutoff switch.

This was not a novice operation, and IMO/IME there is no problem with using crappy basic shelving - it does the job and doesnt obstruct airflow. I dont think the setup was in any way the problem. the problem was:
1) whatever component started the fire (a unit, a fan, a power bar?)
2) The fact that 2 IT workers were sleeping. If one was awake as they should have been the shutoff switch at the transformer would have been hit and the fire stopped dead in its tracks (possibly without even needing an extinguisher). Apparently it took over an hour for the fire to spread according to the dropping hashrate of the farm.
2b) smoke alarms or other automated systems must have been absent or insufficient.

It's not just the obstruction to airflow issue with that ethernet wiring, it shows the general sloppiness together with absolute absence of attention to detail. Anyone who wired 450+ ethernet devices like that most probably never wired anything more complex than a home router, it was bound to cause at least some network glitches. Network cables should not carry it's own weight, and certainly not the weight of hundred or more cables meshed into a spider net, with the whole combined weight of all of them carried by whichever is the shortest one of them, certainly giving some tension on the motherboard it was connected to. It sure was not the cause of the fire, but is obviously wrong to anyone who wired a single rack of equipment. We can only assume how they did the power distribution, but looking at the front side it would be surprising if there were not some very elementary errors which directly contributed to what happened.

the post-fire images indicate that all the power cabling on the backside was bundled far more cleanly. The ethernet was a mess but honestly i dont think its the issue being made out to be. They appear overlapping, but not tangled, so its not like 10lbs of cable is pulling on the shortest link. furthermore, to run wires all the way to the floor, then to the networking rack and back up again would probably double or triple the length of some cables used, which really just leads to more mess of its own, and being unable to follow individual cables when 50 white cords are ziptied together.

A lot could be improved, but until the actual cause of fire is revealed its just a lot of guessing - with most being accusations of the wiring or hardware, when the fire could have been initiated by anything.