[How To] Puwaha's Poor Man's Networked PDU: Using Smart Plugs and Awesome Miner

[puwaha]

I thought I'd share this with everyone since I think this is a problem we can all relate to.  While it would be nice to have a networked PDU, they are quite expensive... even on the used market.  So I thought about it and came up with this solution.


I'm sure we all have a rig or two where it may work fine, but every now and then it needs to be rebooted.  It always seems to happen at the worst of times... like when you are at work... or out of town.  Trying to get your spouse to reset a rig over the phone can sometimes be just as frustrating as when you are away and can't do anything about it.

I also use Awesome Miner, which is fantastic, and can do a lot of automation to keep your rigs running.  But if your rigs need a hard reboot, the only real solution is to "pull the plug" and reboot.  Awesome Miner has "rules" that you can define to watch for triggers like an offline miner, hashrate thresholds, dead GPUs, and a lot more... and then perform an action.  It's kind of like an IFTTT (IF This Then That) solution for mining.

IFTTT... hey wait a minute... I could use the online service IFTTT to control a smart plug.  I could have Awesome Miner watch for some condition, then send a signal to IFTTT, where in turn IFTTT could turn off and then on a smart plug.

I started out with a Wemo Insight smart plug that I had laying around, and after a bit of tweaking, I got this working well.  The problem is that the Wemo Insight smart plug is only rated for 120V, and all of my miners run off of my 24 amp 220V PDU that I got for real cheap off Ebay.  I then trolled around to find some cheaper parts to do a 220V equivalent to the Wemo Insight smart plug.

Puwaha's Poor Man's Networked Power Distribution Unit
(or PPMNPDU... nevermind, that's a bad acronym)

Parts you'll need:

• Cheap 220V PDU from Ebay (I'll let you find these to suit your liking. You could even do this with 120V PDUs)
• Cheapo 240V capable Smart Plugs like these at Amazon (https://www.amazon.com/gp/product/B078CPYQYS/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1
  )
• My PDU uses C14 connectors, and the Smart Plugs are 5-15P, so you'll need some good C14 to 5-15R adaptors like these at Amazon (https://www.amazon.com/gp/product/B004OC579E/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1
• Good quality 5-15P to C13 power cables.  These are your regular power cables that come with every power supply you've ever bought... you probably already have a few... so free!
• IFTTT account (it's free)
• Smart phone to setup the smart plugs (Smart Life app, and IFTTT app)... not free, but you probably already have one... so free
• Awesome Miner... it's free for up to two "miners" to give it a go, but easily worth the license cost in other feature you need to cover all your miners

Note:  if your setup requires the C14 to 5-15R adapters, do not get the crappy 18AWG 1 foot "monitor extensions".
  The wires are not meant to carry high amperage that your rigs usually require.


Now, to put it all together.

First thing, go setup an IFTTT account.  You'll need that before integrating with Smart Life.

Let's go ahead and setup your first Smart Plug.  If you didn't pick the ones I did above in the Amazon link, just make sure they are compatible with the Smart Life app.  This is pretty simple to setup.  I suggest you do them one at a time so you can mark them with a pen, and edit the entry in the Smart Life app... like "Plug 1", and "Plug 2".

After setting up the Smart Plug in Smart Life, give it a few whacks to make sure you can turn it off and on.  In the Smart Life app go to Profile settings and click Integration.  Select IFTTT, and it will give you further instructions on how to integrate.  You'll need to sign in using your Smart Life account to integrate.  You can either use the IFTTT app on your smartphone, or the IFTTT web page... same results.

It's time to build the first IFTTT applet.  Go to your IFTTT app or webpage and click on "My Applets", then "New Applet".  You are now going to setup a trigger (the "This" in IFTTT), and then an action (the "That" in IFTTT).  Click on the +this logo... you will need to choose a service.  Type email in the search area and then click on the email logo.  Choose the "Send IFTTT an email tagged" option.  We want the tagged option, you'll see why in just a second.  This trigger will kick off when you send an email to the trigger@applet.ifttt.com with the tag that you supply.  Our first trigger we will setup will be #plug1off so put that in the field where it says tag, then click create trigger.

Now, we need to add an action to the applet... so click on the word +that logo.  Now search for Smart Life, and then click the Smart Life icon.  You'll see a couple of options to turn a device or group on, and another to turn a device or group off.  If you think about that for a moment, you may realize that you can group smart plugs together in the Smart Life app on your phone, so you could control multiple plugs with some of these steps.  But for now, we will just finish setting up the applet for our first smart plug.  Since our trigger tag was #plug1off, then we need to click on the option to turn a device off.  IFTTT then will present you with your drop down list of Smart Plugs that you setup in the Smart Life app.  Since this is our first one, we will select Plug 1 and click the create action button.  Finally, click the finish button, and you've just created your first applet in IFTTT!  Make sure it's turned on.

Repeat setting up a new applet in IFTTT, use the same trigger of "email" and use a new tag called #plug1on, and an action from Smart Life to turn on a device or group.  Select your Plug 1 device and finish the applet steps.


If you've never used Awesome Miner before, the bare minimums you need here are a pool, a managed miner, and a rig.  Once your rig is being controlled by Awesome Miner, you can automate it with a new user-defined rule.  There are some built-in rules to give you some ideas of what you can do, but for our networked PDU concept, we will create a new one.  First, go into the Options, and under the Rules section go to the Email & SMS section.  You'd want to setup your email account here anyway, for notifications from Awesome Miner in general, but the most important part is that you use the same email account that you used to setup your IFTTT account.

You see, the IFTTT applets you created are expecting an email from your email account.  If you use the same email account in Awesome Miner, then when you setup the user-defined rule to send an email... it matches... and the trigger is set.

Now that you have Awesome Miner setup to use your email account, go to the Rules section in the Options in Awesome Miner.

You'll give it a fancy name like Plug 1 Reset.  If you have more than one rig, then click the "only apply to selected miners", and select your rig that will be connected to the Smart Plug.  Now, add a trigger for this Awesome Miner rule.  If your rig crashes in a way so that it doesn't respond to pings, then make the trigger to be a Ping (on timeout) function.  In my case, I chose to make my trigger "activate" if the rig isn't responding to pings after 9 seconds.  There are other triggers you could use, and you can even stack them for different conditions, either selecting the "Match all" (for an AND function), or the "Match at least on" option (for an OR function).  Sounds complex?  Nah, it's easy.

Finally, you need to set an Action in the Awesome Miner rule.  In our case we are going to stack a bunch of functions to be run in sequence.   The first action is to send an email to the IFTTT email address (trigger@applet.ifttt.com) with a #plug1off in the subject line.  Then I put a Wait action for 10 seconds.  Then put another email action to send an email to the IFTTT email address with a #plug1on in the subject line.  Finally, I put 6 individual wait actions for 60 seconds each.  It basically runs like this:

If no response from IP address for 9 seconds then
    Send and email with #plug1off in the subject line
    wait 10 seconds
    send another email with #plug1on in the subject line
    wait for 60 seconds
    wait for 60 seconds
    wait for 60 seconds
    wait for 60 seconds
    wait for 60 seconds
    wait for 60 seconds

You need the waits in there between the off and on action to give your rig a chance to fully power down.  You also need the waits in there after the turn on action so that your rig has a chance to boot up and start responding to pings again.  I put six waits of 60 seconds each which is plenty of time for an SSD rig to boot up and start responding to pings... but it also means that the Awesome Miner rule won't run again for 6 minutes in case something is wrong.

Here's a quick shot of the Awesome Miner rule setup:



If you click the option in the Awesome Miner rule at the bottom to "Support manual activation" then you can run the rule actions at will from the main Awesome Miner mining rig list under the Actions dropdown.  And if you have a license that allows a web page interface for Awesome Miner, then you can remotely trigger this rule at will while you are away.  That's fine and all... and could be good for getting something unstuck.  

The whole point of using Awesome Miner is to allow your rigs to keep mining, and it allows some pretty incredible automation.  Couple Awesome Miner's already rich feature set for automation with Smart Plugs and IFTTT and you've got Puwaha's Poor Man's Networked PDU!


There's probably a lot to absorb here, so I'm more than willing to answer any questions or help you out.

[boklos]

I wanted to do same thing till I found your post, thank you !
I have a 240v outlet and 9 GPUs drain 250w each, 2x1300 PSUs .
Can I plug each PSU in one of those smart plugs and both plugs would be plugged in the basic 240v PDU ?

[akhouston]

Nice and detailed write up, thanks for that.

I did something similar with SmartThings and smartplugs. Basically when AM notices any "service degradation", i.e. service is offline, device crapped out (i.e. # of devices less than expected) or "device is sick" (have no clue what that means - never saw that happened to a GPU) - then it triggers an action in webCore, which starts a 1 minute countdown timer. I get a notification on the phone with options to power cycle a rig right away or cancel. If no response is received within a minute then the rig is power cycled (if you are doing that to a GPU rig you need to ensure that "recover after power loss is turned on in BIOS).

One thing that didn't work for me is waits on the AM side - it kept sending reboot requests, so I basically created a latching switch, that allows a single reboot within a 5 minute interval - that allows plenty time for a rig to recover.

In all though, it's nice to see that we are thinking generally in the same direction

[puwaha]

I wanted to do same thing till I found your post, thank you !
I have a 240v outlet and 9 GPUs drain 250w each, 2x1300 PSUs .
Can I plug each PSU in one of those smart plugs and both plugs would be plugged in the basic 240v PDU ?

Yes, the smart plugs I picked were rated for 10 amps up to 240V... so they could handle 2400 watts each.  But I'd wouldn't go past 80% of that max, so 1920 watts max.  And really you probably have 220V... so they could handle 2200 watts each.  80% of that would be 1760 watts.

Since you will be using two PSUs, you'd want to put one smart switch on each PSU, and group them in the Smart Life app.  That way you can send the group the signals at the same time so they power off and on at the same time.

[puwaha]

Nice and detailed write up, thanks for that.

I did something similar with SmartThings and smartplugs. Basically when AM notices any "service degradation", i.e. service is offline, device crapped out (i.e. # of devices less than expected) or "device is sick" (have no clue what that means - never saw that happened to a GPU) - then it triggers an action in webCore, which starts a 1 minute countdown timer. I get a notification on the phone with options to power cycle a rig right away or cancel. If no response is received within a minute then the rig is power cycled (if you are doing that to a GPU rig you need to ensure that "recover after power loss is turned on in BIOS).

One thing that didn't work for me is waits on the AM side - it kept sending reboot requests, so I basically created a latching switch, that allows a single reboot within a 5 minute interval - that allows plenty time for a rig to recover.

That's why I added the 5-6 minutes of waits in the action script from AM... that gives the rig enough time to boot up and start responding to pings.  I've actually added another trigger condition in my AM script, as my cheapo Celeron processor sometimes gets overworked.  So I added a Detect Offline trigger of 60 seconds... using the Remote Agent offline tickbox.  I made both triggers a "Match All" which is an AND operator.  This way if the rig is too busy to respond to pings for 9 seconds (the max that the Ping AM trigger allows) it won't reboot the rig unless the rig is detected as "offline" for 60 seconds as well.

In all though, it's nice to see that we are thinking generally in the same direction

Agreed.  There are many opportunities to do some interesting automation using AM and IFTTT.

[akhouston]

Nice and detailed write up, thanks for that.

I did something similar with SmartThings and smartplugs. Basically when AM notices any "service degradation", i.e. service is offline, device crapped out (i.e. # of devices less than expected) or "device is sick" (have no clue what that means - never saw that happened to a GPU) - then it triggers an action in webCore, which starts a 1 minute countdown timer. I get a notification on the phone with options to power cycle a rig right away or cancel. If no response is received within a minute then the rig is power cycled (if you are doing that to a GPU rig you need to ensure that "recover after power loss is turned on in BIOS).

One thing that didn't work for me is waits on the AM side - it kept sending reboot requests, so I basically created a latching switch, that allows a single reboot within a 5 minute interval - that allows plenty time for a rig to recover.

That's why I added the 5-6 minutes of waits in the action script from AM... that gives the rig enough time to boot up and start responding to pings.  I've actually added another trigger condition in my AM script, as my cheapo Celeron processor sometimes gets overworked.  So I added a Detect Offline trigger of 60 seconds... using the Remote Agent offline tickbox.  I made both triggers a "Match All" which is an AND operator.  This way if the rig is too busy to respond to pings for 9 seconds (the max that the Ping AM trigger allows) it won't reboot the rig unless the rig is detected as "offline" for 60 seconds as well.

Right, but what I'm saying, initially I thought of adding same waiting (I was surprised we were limited to 99 sec wait ) but it didn't work for me for some reason - it just kept calling the url almost continuously. From the AM examples I understand that same "wait" can be achieved with using timer and setting "match all conditions", like "detect offline + timer every 5 minutes". This has its downside so that in the worst case scenario you'll detect your downtime 5 minutes after it occurred, BUT on the positive side it will not occur again for another 5 minutes, effectively doing same thing as those wait commands.

[puwaha]

I see what you are saying.  If you have multiple triggers with an OR condition, then I could see it triggering over and over because each trigger condition will rerun the rule despite any waits you have in the actions.

When I first started setting up the rule in AM, while testing I ran into the same problem with the 9 second ping trigger going off every 10 seconds.  Since I had the IP address that the ping trigger was using going to a non-existent device, it would trigger every 9-10 seconds.

[boklos]

I asked on Reddit and someone mentioned the sonoff pow for 220 v rigs, it requires wiring but he said it's sold and rated up to 3600w.
I like the plug idea .ore fir ease of use, but could you look into the sonoff and tell me what you think?
https://www.amazon.com/Sonoff-Wireless-Consumption-Measurement-Appliances/dp/B06XSD6PD6

[puwaha]

I asked on Reddit and someone mentioned the sonoff pow for 220 v rigs, it requires wiring but he said it's sold and rated up to 3600w.
I like the plug idea .ore fir ease of use, but could you look into the sonoff and tell me what you think?
https://www.amazon.com/Sonoff-Wireless-Consumption-Measurement-Appliances/dp/B06XSD6PD6

I've seen those mentioned before as well.  Two things...

One:  It uses the Ewelink software, and I don't have any devices that are compatible with it to know if IFTTT works the way you want.  There is an Ewelink service in IFTTT, but you have to link your account on Ewelink to IFTTT, and again, I don't have any devices that work on that service.  If you order some Sonoff plugs, let us know how they work.

Two:  Those Sonoff plugs have a review on Amazon that says that the relay may not be able to handle the rated 3600 watts.  The reviewer was running 220v x 16A through it and had to bypass the relay with solder.  He shouldn't have been pushing them up to the max 100% rating anyway.  If you run them at 80% which is around 2800 watts you might be fine.

[MarkAz]

Two:  Those Sonoff plugs have a review on Amazon that says that the relay may not be able to handle the rated 3600 watts.  The reviewer was running 220v x 16A through it and had to bypass the relay with solder.  He shouldn't have been pushing them up to the max 100% rating anyway.  If you run them at 80% which is around 2800 watts you might be fine.

Yeah, the Sonoff's definitely will have problems if you push them too hard, and I would consider 3600w to be the very optimistic max.  I really consider 1000w at 120v or 2000w at 240v is what I would consider the max...  But running at 1000w 240v (which is what I run all of my roughly at), they run nice and cold with zero issues.

[boklos]

I wouldn't run more than 1200w on each plug anyway, I really want to go your way but I'm scared from cheap/no name smart plugs with continous 24/7 1200w use, that they could overheat or worse .
How long have you been testing yours?
Also found a 15a smart plugs that run 220v but not ifttt support:
Zettaguard S31-Grey Wi-Fi Smart Plug Outlet, Compatible with Alexa, Timer Switch Socket, Energy Meter, Wireless Remote Control your Electronics from Smartphone or Tablet, Grey https://www.amazon.com/dp/B071RPFJ5S/ref=cm_sw_r_cp_apa_QxiEAbKXSVRBF

Also found a $50 one from a reputable company, but again no ifttt:
Aeotec Smart Switch 6, remote control smart plug, Z-Wave Plus, small size, side USB charging port, power metering https://www.amazon.com/dp/B00VQISOCG/ref=cm_sw_r_cp_apa_mziEAb5NDR3MT

What do you suggest?
Also for your plug or any smart plug, how do you connect it to the PDU?
It's nema 5-15p and pdu has c13.
I couldn't find any cable rated more than 10A that runs 5-15r to c14.
Am I ok with 10A even my pdu is 20A?

[puwaha]

I wouldn't run more than 1200w on each plug anyway, I really want to go your way but I'm scared from cheap/no name smart plugs with continous 24/7 1200w use, that they could overheat or worse .
How long have you been testing yours?

Just under a week at this point.  I wouldn't run them anywhere near the max output, but 1200W should be fine.  That's "only" 5.4 amps.  The only problem I could see would be the relay may not be rated for a full load.

I run mine on a few rigs with 6 1070s or 1070TIs.  I wouldn't run them on my 1080TI rig... but that rig uses a 2400W server PSU, and it runs cool and quiet and stable.  It's my best rig and not because of the 1080TIs.

Also found a 15a smart plugs that run 220v but not ifttt support:
Zettaguard S31-Grey Wi-Fi Smart Plug Outlet, Compatible with Alexa, Timer Switch Socket, Energy Meter, Wireless Remote Control your Electronics from Smartphone or Tablet, Grey https://www.amazon.com/dp/B071RPFJ5S/ref=cm_sw_r_cp_apa_QxiEAbKXSVRBF

Yeah, no IFTTT support for their Home Mate app.  There was a strange review for that app on the product review section of Amazon as well.  I'd stay away from these even if they are rated to run with higher load.

Also found a $50 one from a reputable company, but again no ifttt:
Aeotec Smart Switch 6, remote control smart plug, Z-Wave Plus, small size, side USB charging port, power metering https://www.amazon.com/dp/B00VQISOCG/ref=cm_sw_r_cp_apa_mziEAb5NDR3MT

These look ok, but you'd need a Zwave "hub" to integrate with them, like the Samsung Smartthings hub.  It will add to the expense of getting these up and running, but there's good support for Smartthings on IFTTT.  The good thing about Zwave is that there are tons of other smart appliances, switches and sensors you could use.

What do you suggest?

If you aren't running a smart switch anywhere near it's rated max, then I think you will be fine with the cheapo Smart Life compatible ones.  Those seem to be the most plentiful on the market, and Smart Life is really popular in China.

Also for your plug or any smart plug, how do you connect it to the PDU?
It's nema 5-15p and pdu has c13.
I couldn't find any cable rated more than 10A that runs 5-15r to c14.

Use the adaptors I suggest in the guide:  https://www.amazon.com/gp/product/B004OC579E/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1

I wouldn't suggest using a cable unless it has a low gauge wire.  16-14AWG should be ok for these smart plugs.  I prefer the adapters because they are more likely to be a solid piece of metal between the two plug ends, rather than a wire soldered between the two plug ends like a cable would be.  Don't get the cheap 18AWG "monitor extension" cables as 18AWG is only rated up to 10A... to close the edge for my tastes.

Am I ok with 10A even my pdu is 20A?

Your PDU will only deliver as much power as your load requires.  If you are running say 6 1070TIs (180W TDP) and run them at 75% power limit (135W), you are looking at 810W... 3.7A.  Adding an extra 100W for your motherboard, CPU, RAM, and SSD is not going to add that much to the load.  You'd be fine with a 10A rated smart switch.

[proteus7]

Any idea how to do this with an APC managed PDU? (has a webpage to turn plugs on/off)

[puwaha]

Any idea how to do this with an APC managed PDU? (has a webpage to turn plugs on/off)

The key integration here is by using IFTTT to link Awesome Miner with a product that is supported by IFTTT.  In this case, the smart plugs I'm using are supported by IFTTT through Smart Life.  I didn't see any IFTTT applets for APC, but that doesn't mean you can't cobble together some applet using Webhooks, external services like Google Docs, email, etc.

[trucobit]

Here is to complicate life when there are simpler options.

There are other interesting rules in AM, I refer to the documentation and also the use of a whatdogs, in addition to programming a restart every day, either from AM or from the Whatdogs that cost $ 5. Even there are some that control the graphics, if a reboot falls. if the light goes out and it comes back, it turns on, in short, it does not take more than an eBay whatdogs and to know better the options of AM

IFTT I have known it for a long time, I have used it more for self publishing in blogs and Spam, it is very useful, but it seems to me here that complicate what is easy, and read more carefully the rules of AM, and the API, can do this in a simple way.

[puwaha]

Here is to complicate life when there are simpler options.

There are other interesting rules in AM, I refer to the documentation and also the use of a whatdogs, in addition to programming a restart every day, either from AM or from the Whatdogs that cost $ 5. Even there are some that control the graphics, if a reboot falls. if the light goes out and it comes back, it turns on, in short, it does not take more than an eBay whatdogs and to know better the options of AM

IFTT I have known it for a long time, I have used it more for self publishing in blogs and Spam, it is very useful, but it seems to me here that complicate what is easy, and read more carefully the rules of AM, and the API, can do this in a simple way.

Well, thank you for that.  I'm glad you have your own methods... I shared my method.  Feel free to share yours.

By the way, watchdogs do not work if the rig is offline.