Saturday, September 14, 2013

Monitoring a UPS

The original reason for my interest in the Raspberry Pi was for a mini computer to monitor a UPS and alert us if there was a power loss.  The following is how I setup the Raspberry Pi to monitor the UPS and send an email to our cell phones in the event of a localized power loss.
CyberPower LCD "Smart" UPS

Short Story of Education the Hard Way


(note to reader: If you are just interested just in the UPS monitoring details, skip ahead...)

What are we monitoring with the UPS?  Well, we have a small chest style deep freezer in our garage.  We store steaks and other expensive meats in it.  Anytime there is an excellent sale, my wife stocks up on meats.  A few months ago, something caused the GFCI to trip, and we didn't notice until a few days latter when the rancid smell reached us.  We estimated over $500 worth of meat lost in one fell swoop.  So the wife tasked me with coming up with a solution to avoid a repeat in the future.

So my solution was to use a "smart" UPS, one that I could actively read the metrics from, and send a email notification in the event of a power failure.  The second benefit, or so I had thought, is the UPS could keep the fridge powered for short term power loss.  I already have a CyberPower UPS connected to my file server, and it works great for battery backed power outages, monitoring battery reserves, and shutting down the server if the reserves deplete.

A bit of painful education.  I wanted to make sure I purchased the "right" size of UPS, so I pulled out my Kill A Watt power meter and connected it to the refrigerator.  During the occasional freeze cycle it appeared to pull about 460-490 Watts, and lasted less than a minute.  At idle, it rated closer to 10 Watts.  I rationalized that the UPS should be able to handle a few of these cycles before failing out.

When I looked online for consumer UPS, a majority measured in VA.  So how does one get VA from Watts?  Well, I cheated.  The Kill A Watt also has a VA meter, so I selected that, and it showed around 560-580 VA.  I then found a 600 VA CyberPower UPS.  I admit that I realized 580 to 600 is not a lot of wiggle room, but the next battery size up jumped quite a bit in price.  I crossed my fingers and ordered it.  I should have done a bit more research.

The UPS arrived and I connected it to the freezer.  The freezer's motor kicked on and the UPS went offline.  Clearly I was wrong.

When I actually searched for VA to Watt conversion calculations online, I found that a good estimate is closer to 50% - 60%.  I would have needed almost 800 VA to be on the safe side.  Even this may not have been sufficient, as I learned that the initial motor spike can be quite high.

Lesson learned: You would probably need an industrial sized UPS to battery back a simple freezer.

Well, the battery protection was a bust, but at least I could still use the UPS to monitor for a power failure.  Unfortunately, had I known this, I could have saved a few bucks, and just purchased the cheaper smaller "smart" UPS.  The Raspberry Pi could run for days off even the smallest one.

So onto the technical stuff...


How to monitor a UPS with a Raspberry Pi


The setup.  The Raspberry Pi lives out in the garage, on top of the deep freezer.  I have a USB powered hub connected to the UPS, in one of the battery backed up slots.  The USB hub then powers the Raspberry Pi (to avoid having to waste two power adapters).  To talk to the USB hub I have a second USB cable connected to the data side of the Pi.  To connect to the network, I have a WiFi dongle connected to the USB powered hub.  To try and keep it as clean as possible, I have Raspberry Pi inside a plastic container, with holes for air flow.  Finally, to get as much use of the Raspberry Pi, it also serves double duty and controls/monitors the garage door.

There are numerous "UPS" monitoring packages available, but the one I have had the most success with, is the one from APC called "apcupsd - APC UPS Power Management (daemon)".  Although it was designed for APC branded UPS, it works quite well with numerous other UPS.

I tend to purchase the CyperPower UPS line with the LCD display.  I am a sucker for the LCD that provides quick visual metrics, of the UPS.  The CyperPower UPS connects to a PC through USB and works well with the apcupsd daemon.

Luckily the package is already included in the Raspbian repositories, so installation is this simple:

# apcupsd installation
apt-get update
apt-get install apcupsd

The next step is to modify the configuration scripts, so apcupsd can find the UPS.  Use your editor and modify /etc/apcupsd/apcupsd.conf with the following changes.

# vim /etc/apcupsd/apcupsd.conf

#UPSCABLE smart
UPSCABLE usb

#UPSTYPE apcsmart
UPSTYPE usb

#DEVICE /dev/ttyS0
DEVICE

These changes tell apcupsd that the UPS is connected via USB.

Next we want to test our connection to the UPS.  This can be achieved with the 'acptest' program.

# apctest

2013-09-14 14:49:10 apctest 3.14.10 (13 September 2011) debian
Checking configuration ...
Attached to driver: usb
sharenet.type = Network & ShareUPS Disabled
cable.type = USB Cable
mode.type = USB UPS Driver
Setting up the port ...
Doing prep_device() ...

You are using a USB cable type, so I'm entering USB test mode
Hello, this is the apcupsd Cable Test program.
This part of apctest is for testing USB UPSes.

Getting UPS capabilities...SUCCESS

If you see the "SUCCESS", everything is great.  There isn't much here that you will want to change, so let's quit this program and start the apcupsd daemon.

First, enable the service:

# vim /etc/default/apcupsd

ISCONFIGURED=yes

Next, restart the service:

# service apcupsd restart

Starting UPS power management: apcupsd.

To have apcupsd auto start at boot:

# update-rc.d apcupsd defaults

Next, we can query the UPS to see the various metrics available to us using the 'apcaccess' application.  The properties that apcupsd supports is quite large, and each UPS uses different properties, so your results may vary.

# apcaccess

APC      : 001,032,0781
DATE     : 2013-09-14 14:54:31 -0600
HOSTNAME : pi-ups
VERSION  : 3.14.10 (13 September 2011) debian
UPSNAME  : pi-ups
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2013-09-14 14:54:30 -0600
MODEL    : UPS CP600
STATUS   : ONLINE
LINEV    : 120.0 Volts
LOADPCT  :   0.0 Percent Load Capacity
BCHARGE  : 100.0 Percent
TIMELEFT :  82.0 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
OUTPUTV  : 120.0 Volts
DWAKE    : 000 Seconds
LOTRANS  : 100.0 Volts
HITRANS  : 140.0 Volts
ALARMDEL : 30 seconds
NUMXFERS : 0
TONBATT  : 0 seconds
CUMONBATT: 0 seconds
XOFFBATT : N/A
SELFTEST : OK
STATFLAG : 0x07000008 Status Flag
SERIALNO : AEC7101.251
NOMINV   : 120 Volts
NOMPOWER : 340 Watts
END APC  : 2013-09-14 14:55:02 -0600

Now that we can query the UPS, let's work with the apcupsd "event" scripts.  This is how we will get an email sent when a power failure occurs.  The event scripts can all be found in the /etc/apcupsd/ folder.  All events are controlled by the /etc/apcupsd/apccontrol master script, so take a quick peak there if you want to know what else can be modified.

The two scripts we are interested in are the "onbattery" and "offbattery" scripts.  The "onbattery" is triggered during a power failure, and "offbattery" is triggered when power returns.

By default these scripts are just simple bash scripts that send a message to the PC's console and send a local email.  Instead of using local mail, we will use Gmail to send our notification, and write the script in Python.  One other side benefit of having these scripts written in Python, is I can also integrate Xively and then update Xively with the various data points of my UPS.

Let's backup the current scripts, before modifying them:
cp /etc/apcupsd/onbattery /etc/apcupsd/onbattery.original
cp /etc/apcupsd/offbattery /etc/apcupsd/offbattery.original

This is the powery failure script I use:

onbattery:
#!/usr/bin/env python

import smtplib
import email.mime.text
import syslog

syslog.openlog('[UPS]')
def log(msg):
    syslog.syslog(str(msg))

GMAIL_ADDRESS = 'xxx@gmail.com'
GMAIL_PASSWORD = 'xxx'

from_email = GMAIL_ADDRESS
to_emails = ["xxxxxxxxxx@tmomail.net"]  # cell phone address

msg_subject = "ALERT: UPS Power Failure"
msg_text = "Auto Notification"

log(msg_subject)

msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = msg_subject
msg['From'] = from_email
msg['To'] = ", ".join(to_emails)
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login(GMAIL_ADDRESS, GMAIL_PASSWORD)
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()


And for when power returns:

offbattery:
#!/usr/bin/env python

import smtplib
import email.mime.text
import syslog

syslog.openlog('[UPS]')
def log(msg):
    syslog.syslog(str(msg))

GMAIL_ADDRESS = 'xxx@gmail.com'
GMAIL_PASSWORD = 'xxx'

from_email = GMAIL_ADDRESS
to_emails = ["xxxxxxxxxx@tmomail.net"]  # cell phone address

msg_subject = "OK: UPS Power Recovered"
msg_text = "Auto Notification"

log(msg_subject)

msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = msg_subject
msg['From'] = from_email
msg['To'] = ", ".join(to_emails)
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login(GMAIL_ADDRESS, GMAIL_PASSWORD)
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()

Finally, we test the scripts locally to make sure the email notification is sent.  If everything looks good, test a power failure by disconnecting the UPS from the wall.  Within a few seconds, the apcupsd daemon should trigger the onbattery event and our email should be sent.  Now we will immediately know the next time the UPS looses wall power.

So, what if the power loss is not local to just the UPS?  An external monitoring service could be used.  Something like Nagios running from the cloud works great for this, but I will save that for another time.

37 comments:

  1. Everything works dandy, did the install, tested the scripts with python, unplugged the UPS to test it and I'm good with everything except the email functionality. When I unplug the UPS from the wall, I get wall messages indicating the UPS is on battery & it will shutdown my Pi then reboot it when the power comes back, but it's not actually sending the email. Any thoughts on what I might be missing? I've checked the man pages, and it appears that APCUPSD is working fine (apaccess and apctest work) so I know the UPS its talking fine to the Pi, it just seems like somewhere in the apcontrol script. When I look at /var/log/messages I get:

    19-Sep 1:12:46 raspberrypi kernel: [ 2584.736661] usb 1-1.2: USB disconnect, device number 6
    19-Sep 1:12:46 raspberrypi kernel: [ 2584.736698] usb 1-1.2.1: USB disconnect, device number 7
    19-Sep 1:12:46 raspberrypi kernel: [ 2584.958567] usb 1-1.2.2: USB disconnect, device number 8
    19-Sep 1:15:54 raspberrypi shutdown[3604]: shutting down for system halt
    19-Sep 1:16:14 raspberrypi kernel: imklog 5.8.11, log source = /proc/kmsg started.
    19-Sep 1:16:14 raspberrypi rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="1856" x-info="http://www.rsyslog.com"] start
    19-Sep 1:16:14 raspberrypi kernel: [ 0.000000] Booting Linux on physical CPU 0x0



    This is the apcupsd.events file:
    2014-09-19 01:12:46 +0000 Power failure.
    2014-09-19 01:12:52 +0000 Running on UPS batteries.
    2014-09-19 01:15:53 +0000 Reached run time limit on batteries.
    2014-09-19 01:15:53 +0000 Initiating system shutdown!
    2014-09-19 01:15:53 +0000 User logins prohibited
    2014-09-19 01:15:55 +0000 apcupsd exiting, signal 15
    2014-09-19 01:15:55 +0000 apcupsd shutdown succeeded
    2014-09-19 01:16:17 +0000 apcupsd 3.14.10 (13 September 2011) debian startup succeeded



    Any suggestions on what I might be missing or where to try next?

    ReplyDelete
  2. Love the detailed description. But, remember that when the power fails you may not have a working network to send the mail out.

    ReplyDelete
  3. *UPDATE*

    I had a DOH! moment. I hit up a friend for some help and he noticed the APCCONTROL script by default does not actually have a call to either onbattery or offbattery, thus the reason my script didn't work. He suggested adding the following if statement (change offbattery to onbattery for that section):

    if [ -f /etc/apcupsd/offbattery ] ; then
    python /etc/apcupsd/offbattery
    fi


    It was one of those can't see the forest through the trees moments...

    As for as the power fail not sending email, in my case, I have the router and modem on the UPS as well so I should get a last gasp message, and if not, there are always the log files...

    ReplyDelete
  4. Thanks, this was very helpful. Exact what i was looking for.

    Greetings from Germany,
    Thomas

    ReplyDelete
  5. I' "borrowed" your coding and setup to monitor our power at work. It works great. Now i just need to add one more thing, when the script is activated for power failure, i need it to switch a relay high. This in turn will be connected to our security system and will send signal to the monitoring company and they will make a phone call. not exactly sure how to do this, can it be added into the existing scripts or should it run seperately?

    ReplyDelete
  6. Very nice setup....I did this on the cyberpower ups i have protecting my fileservers. Can I call a bash script from this onbattery script?

    ReplyDelete
  7. Or...Can I call more than just the onbattery script?

    ReplyDelete
  8. Ok...I figured it out. I have is wait 2 min then if it's still down, email me and shutdown my fileservers.

    ReplyDelete
    Replies
    1. How do you shutdown your servers? SNMP? I want to shutdown my NAS (Qnap TAS-168) using SNMP, how can I send the SNMP command?

      Delete
  9. Thank you for this posting. Worked fine for me.

    When configured, do not forget to edit the following file:
    /etc/default/apcupsd

    and set ISCONFIGURED=yes

    Greetiings vrom Germany
    Joachim

    ReplyDelete
  10. Thank you Joachim, updated the post.

    ReplyDelete
  11. Thank you for the great post!

    One thing that I missed as I was setting this up was that the `apctest` command must be run with super user privileges.

    I realize that a few of your other commands need to be too (editing the configs, etc.), but that one especially caught me because I was getting an error that didn't indicate that it was a privileges issue.

    In any case thanks for the writeup!

    ReplyDelete
  12. Works great even three years later, Thanks!

    ReplyDelete
  13. Can I use an ups for raspberry pi to provide power backup without having any other things to connect to ups

    ReplyDelete
  14. Don't know why not. I started this with the 1st Pi and it's now running on P3. I use it to power down my file servers and email me when power is lost.

    ReplyDelete
  15. Hi i am having a problem getting it to work with a cyber power cp600 and a pi with raspbx,
    0.000 apcupsd: drivers.c:281 Warning: no UPS driver found (ups->mode.type=0).
    apctest FATAL ERROR in apctest.c at line 268
    apctest cannot continue without a valid driver.
    can anyone help me?

    ReplyDelete
  16. Worked perfect with my Cyberpower UPS, just changed emails/password in notification scripts and voila! Kudos!

    ReplyDelete
  17. I found a python script for my Dlink DNS-323's on the forum for the NAS.

    ReplyDelete
  18. Thanks KennethB, after a lot of research, your approach with the RPi is really clean and efficient. Had mine up and running easily. I have a question. ON Battery, OFF Battery is all good. What about this scenario: UPS Goes ON BATTERY, Message gets sent. UPS reaches your magic % run-time remaining and shuts down devices. UPS is OFF, so is RPi. POWER gets restored, UPS is back online, RPi is restarted. How do / or can I get notified that the power is back ON after the previous outage message.

    -FrED

    ReplyDelete
  19. Please note if you’re intended to provide uninterruptible power for a refrigerator, you’ll need an UPS with SINE form of output signal. Any ordinary UPS used for PC-s produces meander form of signal in battery mode, which may cause damage to a fridge motor. The same is right for any electrical appliance unit with non-impulse mode of power consumption. So please don’t use UPS with meander or “approximated sine” forms of signal for fridges and similar household appliances. Those UPS are for PC-s only. Talk to a professional electrician, if necessary.
    Good luck!

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. Did anybody perfect the scripts to send more info, like the output from apcaccess?

    ReplyDelete
  22. Call your script from the Onbattery script.

    ReplyDelete
  23. Good article. You can also visit this website and read XNSPY review about tracking app.

    ReplyDelete
  24. It works! I had to make and install the latest version for my Linux Mint 18 system, see:

    https://sourceforge.net/projects/apcupsd/files/apcupsd%20-%20Stable/3.14.14/

    Please note that those who use an USB cable should use this command:

    ./configure --enable-usb

    instead of ./configure

    ReplyDelete
  25. I now have 3 pi's monitoring 3 of the 4 ups units in my house. They email me once a day with the status of their ups.

    ReplyDelete
  26. This comment has been removed by the author.

    ReplyDelete
  27. I just need to set it up so when the power goes off and there is 5 minutes of battery left it shuts down the Raspberry PI3, and when the power goes back online it starts the raspberry again.

    Anyway to do that?

    ReplyDelete
  28. It may difficult to set it up to power down when 5 min are left, but you could have a script in the On Battery script to issue a shutdown to the pi, and it it is in the UPS's battery outlet, it would power up with the power coming back. Although, I think when the UPS shuts down, you have to manually turn it back on. This is how my APC's do it.

    ReplyDelete
  29. you ll need to script that and place the script in '/etc/apcupsd/onbattery'

    you ll probably need to set up a loop to check the TIMELEFT value, and when it is less than 5, run the 'poweroff' command


    someth like
    While [On battery]
    if apcaccess | grep TIMELEFT< 5 minutes THEN run poweroff

    You ll have to play with the command to confirm the exact wording.

    ReplyDelete
  30. To include the output of apcaccess you can just replace the line:

    msg_text = "Auto Notification"

    with the following:

    import subprocess
    msg_text = subprocess.check_output("apcaccess", shell=True)

    ReplyDelete
  31. This comment has been removed by the author.

    ReplyDelete
  32. Hi I managed to do the script that checks every 30secs when TIMELEFT is <5 minutes, using the Python version of apcaccess (https://github.com/flyte/apcaccess). Here is the modified onbattery script:


    #!/usr/bin/env python

    import smtplib
    import email.mime.text
    import syslog

    syslog.openlog('[UPS]')
    def log(msg):
    syslog.syslog(str(msg))

    GMAIL_ADDRESS = ‘name@gmail.com’
    GMAIL_PASSWORD = ‘password’


    from_email = GMAIL_ADDRESS
    to_emails = [“recipient@gmail.com”]

    msg_subject = "ALERT: UPS Power Failure"
    msg_text = "Auto Notification"

    log(msg_subject)

    msg = email.mime.text.MIMEText(msg_text)
    msg['Subject'] = msg_subject
    msg['From'] = from_email
    msg['To'] = ", ".join(to_emails)
    s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
    s.login(GMAIL_ADDRESS, GMAIL_PASSWORD)
    s.sendmail(from_email, to_emails, msg.as_string())
    s.quit()

    msg2_subject = "ALERT: UPS triggered Shutdown"
    msg2_text = "Auto Notification"

    msg2 = email.mime.text.MIMEText(msg2_text)
    msg2['Subject'] = msg2_subject
    msg2['From'] = from_email
    msg2['To'] = ", ".join(to_emails)


    while True:
    >from apcaccess import status as apc

    >stato=apc.parse(apc.get(host="localhost"), strip_units=True)

    >value=stato['TIMELEFT']

    >newval=float(value)
    >if newval <= 5:
    >>s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
    >>s.login(GMAIL_ADDRESS, GMAIL_PASSWORD)
    >>s.sendmail(from_email, to_emails, msg2.as_string())
    >>s.quit()
    >>break
    >else:
    >>time.sleep(10)

    It's my first Python attempt so some of you may be able to perfect it a bit

    Thanks
    R

    ReplyDelete
  33. I just realised it's not necessary to reinvent the wheel - it's enough to create a script called doshutdown and insert there any actions you want to be performed when shutdown sequence is initiated...

    ReplyDelete
  34. Script works very good, love it! But I can't find the history (time power-outage and power-come-back).
    With those scirpts, what would be instructions to add to write in a file and give inof? My internet router is in a different place and when the power goes down I lost internet right away so I can't received the mail but having a file that gives information, will help when the power goes back on.

    ReplyDelete

Please be respectful.