|
Title: 200 GPU farm got rained on, had to move it. Post by: kotarius on August 19, 2015, 06:41:49 AM There was a freak thunderstorm that shut down 1-phase power, exhaust fan stopped spinning and stopped producing outward pressure on the rain.
Rain got in and affected PSU, MOBO, and GPU, resulting in cascading failure. PSUs fried, motherboards shorted, graphics cards VRM burst. It was impossible to troubleshoot, hard to know which hardware component cause failure. Luckily, empty GPU rigs (just barebones, no GPU) existed elsewhere in the building (downstairs). A decision was made to move the GPUs from upstairs to downstairs, into known working hardware. Here is a picture of all the GPUs extracted: https://i.imgur.com/7QjpmIK.jpg Title: Re: 200 GPU farm got rained on, had to move it. Post by: aa on August 19, 2015, 11:41:54 AM Mmmm deep fried GPU.
Title: Re: 200 GPU farm got rained on, had to move it. Post by: madmartyk on August 19, 2015, 01:25:43 PM You need a truckload of rice to suck all the wet out of them!!! Man, that SUCKS!!
Title: Re: 200 GPU farm got rained on, had to move it. Post by: jjjordan on August 19, 2015, 02:48:15 PM You need a truckload of rice to suck all the wet out of them!!! Man, that SUCKS!! You only need a pound of rice. Just scatter it around and wait for the Chinese to come and repair...Title: Re: 200 GPU farm got rained on, had to move it. Post by: djm34 on August 19, 2015, 05:11:12 PM You need a truckload of rice to suck all the wet out of them!!! Man, that SUCKS!! You only need a pound of rice. Just scatter it around and wait for the Chinese to come and repair...was about to tell to use silica gel instead... but yeah rice is better ;D Title: Re: 200 GPU farm got rained on, had to move it. Post by: Mikestang on August 19, 2015, 10:47:00 PM There was a freak thunderstorm that shut down 1-phase power, exhaust fan stopped spinning and stopped producing outward pressure on the rain. Ouch, that looks like a lot of hardware! I'm surprised you didn't have backup power on-site. I'm also surprised your setup only relies on positive air pressure to keep it weather-tight. I guess hind sight is 20-20, any plans to upgrade the facility so this doesn't happen again? Title: Re: 200 GPU farm got rained on, had to move it. Post by: noideacoin on August 19, 2015, 11:02:30 PM How much did you lose from it? How many were salvageable?
Title: Re: 200 GPU farm got rained on, had to move it. Post by: VirosaGITS on August 19, 2015, 11:10:45 PM Well thats catastrophic, my condolences... xD
I recommend you invest in at least a 50$ Back-UPS. Or maybe a good 150$ one. Seem pretty cheap to protect 20k$ of hardware. Or/and just a tarp. You know you put a tarp outside on the exhaust and if the fan stop it fall down and will prevent crap from enter. During outward pressure making it in, the tarp will kind of get sucked in and seal the hole. 2$ fix. Ugly but cheap and effective. Title: Re: 200 GPU farm got rained on, had to move it. Post by: kotarius on August 20, 2015, 06:30:42 AM Most of the GPUs are downstairs now, looks like about 20 or 30 are broken.
If this upstairs GPU farm is launched again, the following will be done: 1. Installation of awning and drainage on window to prevent rain from entering. 2. Exhaust fan will be connected to the same 3-phase power supplying farm. Upon power loss and subsequent restore, APC power strip will be instructed to turn fan on immediately, and the rest of the machines only after 30 minutes. This way, if any rain gets in, it will be dried out long before hardware is damaged. Title: Re: 200 GPU farm got rained on, had to move it. Post by: MaxDZ8 on August 20, 2015, 06:49:20 AM Owch!
Seriously, just ask a mason to install a huge ventilation pipe 90deg turn on the outside. It is ridiculous you relied on pressure to keep the rain out and then pushed directly on the hardware. As long as rain isn't being pushed up from the ground, no chance it's going to enter the building. No, an awning does not offer the same level of protection. In particular, installing drainage is pretty much nonsense as it doesn't catch airborn water droplets unless you install a filter which will reduce fan efficiency. Plus drainage requires maintenance (albeit little, and that's probably something you're not interested in). I've never heard of such issues when the 90deg turn is used (it depends on how fast you suck in BTW). Idea (2) is actually fairly good. Not sure 30 min is sufficient in case of catastrophic failure of the first layer of protections but I like the idea. Title: Re: 200 GPU farm got rained on, had to move it. Post by: Cryptozillah on August 20, 2015, 12:17:21 PM Damn. I hope some hardware is still ok !
Title: Re: 200 GPU farm got rained on, had to move it. Post by: Gladimor on August 20, 2015, 05:29:11 PM Most of the GPUs are downstairs now, looks like about 20 or 30 are broken. If this upstairs GPU farm is launched again, the following will be done: 1. Installation of awning and drainage on window to prevent rain from entering. 2. Exhaust fan will be connected to the same 3-phase power supplying farm. Upon power loss and subsequent restore, APC power strip will be instructed to turn fan on immediately, and the rest of the machines only after 30 minutes. This way, if any rain gets in, it will be dried out long before hardware is damaged. Do you still have RMA and warranty on those cards? Title: Re: 200 GPU farm got rained on, had to move it. Post by: VirosaGITS on August 20, 2015, 08:13:49 PM Most of the GPUs are downstairs now, looks like about 20 or 30 are broken. If this upstairs GPU farm is launched again, the following will be done: 1. Installation of awning and drainage on window to prevent rain from entering. 2. Exhaust fan will be connected to the same 3-phase power supplying farm. Upon power loss and subsequent restore, APC power strip will be instructed to turn fan on immediately, and the rest of the machines only after 30 minutes. This way, if any rain gets in, it will be dried out long before hardware is damaged. Do you still have RMA and warranty on those cards? This gave me a good laugh. Yeah i'm sure warranty cover damage of hardware if left under the rain. ;) I don't think Ugt will let it happen again. Title: Re: 200 GPU farm got rained on, had to move it. Post by: kotarius on August 21, 2015, 04:05:26 AM It is ridiculous you relied on pressure to keep the rain out and then pushed directly on the hardware. As long as rain isn't being pushed up from the ground, no chance it's going to enter the building. No, an awning does not offer the same level of protection. In particular, installing drainage is pretty much nonsense as it doesn't catch airborn water droplets unless you install a filter which will reduce fan efficiency. I've never heard of such issues when the 90deg turn is used (it depends on how fast you suck in BTW). The exhaust fan was blowing out, not sucking in. Even with the fan off, regular rain would not get in unless the storm was severe and catastrophic. |
