Saturday, September 14, 2013

Monitoring a UPS

The original reason for my interest in the Raspberry Pi was for a mini computer to monitor a UPS and alert us if there was a power loss.  The following is how I setup the Raspberry Pi to monitor the UPS and send an email to our cell phones in the event of a localized power loss.
CyberPower LCD "Smart" UPS

Short Story of Education the Hard Way


(note to reader: If you are just interested just in the UPS monitoring details, skip ahead...)

What are we monitoring with the UPS?  Well, we have a small chest style deep freezer in our garage.  We store steaks and other expensive meats in it.  Anytime there is an excellent sale, my wife stocks up on meats.  A few months ago, something caused the GFCI to trip, and we didn't notice until a few days latter when the rancid smell reached us.  We estimated over $500 worth of meat lost in one fell swoop.  So the wife tasked me with coming up with a solution to avoid a repeat in the future.

So my solution was to use a "smart" UPS, one that I could actively read the metrics from, and send a email notification in the event of a power failure.  The second benefit, or so I had thought, is the UPS could keep the fridge powered for short term power loss.  I already have a CyberPower UPS connected to my file server, and it works great for battery backed power outages, monitoring battery reserves, and shutting down the server if the reserves deplete.

A bit of painful education.  I wanted to make sure I purchased the "right" size of UPS, so I pulled out my Kill A Watt power meter and connected it to the refrigerator.  During the occasional freeze cycle it appeared to pull about 460-490 Watts, and lasted less than a minute.  At idle, it rated closer to 10 Watts.  I rationalized that the UPS should be able to handle a few of these cycles before failing out.

When I looked online for consumer UPS, a majority measured in VA.  So how does one get VA from Watts?  Well, I cheated.  The Kill A Watt also has a VA meter, so I selected that, and it showed around 560-580 VA.  I then found a 600 VA CyberPower UPS.  I admit that I realized 580 to 600 is not a lot of wiggle room, but the next battery size up jumped quite a bit in price.  I crossed my fingers and ordered it.  I should have done a bit more research.

The UPS arrived and I connected it to the freezer.  The freezer's motor kicked on and the UPS went offline.  Clearly I was wrong.

When I actually searched for VA to Watt conversion calculations online, I found that a good estimate is closer to 50% - 60%.  I would have needed almost 800 VA to be on the safe side.  Even this may not have been sufficient, as I learned that the initial motor spike can be quite high.

Lesson learned: You would probably need an industrial sized UPS to battery back a simple freezer.

Well, the battery protection was a bust, but at least I could still use the UPS to monitor for a power failure.  Unfortunately, had I known this, I could have saved a few bucks, and just purchased the cheaper smaller "smart" UPS.  The Raspberry Pi could run for days off even the smallest one.

So onto the technical stuff...


How to monitor a UPS with a Raspberry Pi


The setup.  The Raspberry Pi lives out in the garage, on top of the deep freezer.  I have a USB powered hub connected to the UPS, in one of the battery backed up slots.  The USB hub then powers the Raspberry Pi (to avoid having to waste two power adapters).  To talk to the USB hub I have a second USB cable connected to the data side of the Pi.  To connect to the network, I have a WiFi dongle connected to the USB powered hub.  To try and keep it as clean as possible, I have Raspberry Pi inside a plastic container, with holes for air flow.  Finally, to get as much use of the Raspberry Pi, it also serves double duty and controls/monitors the garage door.

There are numerous "UPS" monitoring packages available, but the one I have had the most success with, is the one from APC called "apcupsd - APC UPS Power Management (daemon)".  Although it was designed for APC branded UPS, it works quite well with numerous other UPS.

I tend to purchase the CyperPower UPS line with the LCD display.  I am a sucker for the LCD that provides quick visual metrics, of the UPS.  The CyperPower UPS connects to a PC through USB and works well with the apcupsd daemon.

Luckily the package is already included in the Raspbian repositories, so installation is this simple:

# apcupsd installation
apt-get update
apt-get install apcupsd

The next step is to modify the configuration scripts, so apcupsd can find the UPS.  Use your editor and modify /etc/apcupsd/apcupsd.conf with the following changes.

# vim /etc/apcupsd/apcupsd.conf

#UPSCABLE smart
UPSCABLE usb

#UPSTYPE apcsmart
UPSTYPE usb

#DEVICE /dev/ttyS0
DEVICE

These changes tell apcupsd that the UPS is connected via USB.

Next we want to test our connection to the UPS.  This can be achieved with the 'acptest' program.

# apctest

2013-09-14 14:49:10 apctest 3.14.10 (13 September 2011) debian
Checking configuration ...
Attached to driver: usb
sharenet.type = Network & ShareUPS Disabled
cable.type = USB Cable
mode.type = USB UPS Driver
Setting up the port ...
Doing prep_device() ...

You are using a USB cable type, so I'm entering USB test mode
Hello, this is the apcupsd Cable Test program.
This part of apctest is for testing USB UPSes.

Getting UPS capabilities...SUCCESS

If you see the "SUCCESS", everything is great.  There isn't much here that you will want to change, so let's quit this program and start the apcupsd daemon.

First, enable the service:

# vim /etc/default/apcupsd

ISCONFIGURED=yes

Next, restart the service:

# service apcupsd restart

Starting UPS power management: apcupsd.

To have apcupsd auto start at boot:

# update-rc.d apcupsd defaults

Next, we can query the UPS to see the various metrics available to us using the 'apcaccess' application.  The properties that apcupsd supports is quite large, and each UPS uses different properties, so your results may vary.

# apcaccess

APC      : 001,032,0781
DATE     : 2013-09-14 14:54:31 -0600
HOSTNAME : pi-ups
VERSION  : 3.14.10 (13 September 2011) debian
UPSNAME  : pi-ups
CABLE    : USB Cable
DRIVER   : USB UPS Driver
UPSMODE  : Stand Alone
STARTTIME: 2013-09-14 14:54:30 -0600
MODEL    : UPS CP600
STATUS   : ONLINE
LINEV    : 120.0 Volts
LOADPCT  :   0.0 Percent Load Capacity
BCHARGE  : 100.0 Percent
TIMELEFT :  82.0 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME  : 0 Seconds
OUTPUTV  : 120.0 Volts
DWAKE    : 000 Seconds
LOTRANS  : 100.0 Volts
HITRANS  : 140.0 Volts
ALARMDEL : 30 seconds
NUMXFERS : 0
TONBATT  : 0 seconds
CUMONBATT: 0 seconds
XOFFBATT : N/A
SELFTEST : OK
STATFLAG : 0x07000008 Status Flag
SERIALNO : AEC7101.251
NOMINV   : 120 Volts
NOMPOWER : 340 Watts
END APC  : 2013-09-14 14:55:02 -0600

Now that we can query the UPS, let's work with the apcupsd "event" scripts.  This is how we will get an email sent when a power failure occurs.  The event scripts can all be found in the /etc/apcupsd/ folder.  All events are controlled by the /etc/apcupsd/apccontrol master script, so take a quick peak there if you want to know what else can be modified.

The two scripts we are interested in are the "onbattery" and "offbattery" scripts.  The "onbattery" is triggered during a power failure, and "offbattery" is triggered when power returns.

By default these scripts are just simple bash scripts that send a message to the PC's console and send a local email.  Instead of using local mail, we will use Gmail to send our notification, and write the script in Python.  One other side benefit of having these scripts written in Python, is I can also integrate Xively and then update Xively with the various data points of my UPS.

Let's backup the current scripts, before modifying them:
cp /etc/apcupsd/onbattery /etc/apcupsd/onbattery.original
cp /etc/apcupsd/offbattery /etc/apcupsd/offbattery.original

This is the powery failure script I use:

onbattery:
#!/usr/bin/env python

import smtplib
import email.mime.text
import syslog

syslog.openlog('[UPS]')
def log(msg):
    syslog.syslog(str(msg))

GMAIL_ADDRESS = 'xxx@gmail.com'
GMAIL_PASSWORD = 'xxx'

from_email = GMAIL_ADDRESS
to_emails = ["xxxxxxxxxx@tmomail.net"]  # cell phone address

msg_subject = "ALERT: UPS Power Failure"
msg_text = "Auto Notification"

log(msg_subject)

msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = msg_subject
msg['From'] = from_email
msg['To'] = ", ".join(to_emails)
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login(GMAIL_ADDRESS, GMAIL_PASSWORD)
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()


And for when power returns:

offbattery:
#!/usr/bin/env python

import smtplib
import email.mime.text
import syslog

syslog.openlog('[UPS]')
def log(msg):
    syslog.syslog(str(msg))

GMAIL_ADDRESS = 'xxx@gmail.com'
GMAIL_PASSWORD = 'xxx'

from_email = GMAIL_ADDRESS
to_emails = ["xxxxxxxxxx@tmomail.net"]  # cell phone address

msg_subject = "OK: UPS Power Recovered"
msg_text = "Auto Notification"

log(msg_subject)

msg = email.mime.text.MIMEText(msg_text)
msg['Subject'] = msg_subject
msg['From'] = from_email
msg['To'] = ", ".join(to_emails)
s = smtplib.SMTP_SSL('smtp.gmail.com', '465')
s.login(GMAIL_ADDRESS, GMAIL_PASSWORD)
s.sendmail(from_email, to_emails, msg.as_string())
s.quit()

Finally, we test the scripts locally to make sure the email notification is sent.  If everything looks good, test a power failure by disconnecting the UPS from the wall.  Within a few seconds, the apcupsd daemon should trigger the onbattery event and our email should be sent.  Now we will immediately know the next time the UPS looses wall power.

So, what if the power loss is not local to just the UPS?  An external monitoring service could be used.  Something like Nagios running from the cloud works great for this, but I will save that for another time.

25 comments:

AndyG said...

Everything works dandy, did the install, tested the scripts with python, unplugged the UPS to test it and I'm good with everything except the email functionality. When I unplug the UPS from the wall, I get wall messages indicating the UPS is on battery & it will shutdown my Pi then reboot it when the power comes back, but it's not actually sending the email. Any thoughts on what I might be missing? I've checked the man pages, and it appears that APCUPSD is working fine (apaccess and apctest work) so I know the UPS its talking fine to the Pi, it just seems like somewhere in the apcontrol script. When I look at /var/log/messages I get:

19-Sep 1:12:46 raspberrypi kernel: [ 2584.736661] usb 1-1.2: USB disconnect, device number 6
19-Sep 1:12:46 raspberrypi kernel: [ 2584.736698] usb 1-1.2.1: USB disconnect, device number 7
19-Sep 1:12:46 raspberrypi kernel: [ 2584.958567] usb 1-1.2.2: USB disconnect, device number 8
19-Sep 1:15:54 raspberrypi shutdown[3604]: shutting down for system halt
19-Sep 1:16:14 raspberrypi kernel: imklog 5.8.11, log source = /proc/kmsg started.
19-Sep 1:16:14 raspberrypi rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="1856" x-info="http://www.rsyslog.com"] start
19-Sep 1:16:14 raspberrypi kernel: [ 0.000000] Booting Linux on physical CPU 0x0



This is the apcupsd.events file:
2014-09-19 01:12:46 +0000 Power failure.
2014-09-19 01:12:52 +0000 Running on UPS batteries.
2014-09-19 01:15:53 +0000 Reached run time limit on batteries.
2014-09-19 01:15:53 +0000 Initiating system shutdown!
2014-09-19 01:15:53 +0000 User logins prohibited
2014-09-19 01:15:55 +0000 apcupsd exiting, signal 15
2014-09-19 01:15:55 +0000 apcupsd shutdown succeeded
2014-09-19 01:16:17 +0000 apcupsd 3.14.10 (13 September 2011) debian startup succeeded



Any suggestions on what I might be missing or where to try next?

Anonymousy said...

Love the detailed description. But, remember that when the power fails you may not have a working network to send the mail out.

AndyG said...

*UPDATE*

I had a DOH! moment. I hit up a friend for some help and he noticed the APCCONTROL script by default does not actually have a call to either onbattery or offbattery, thus the reason my script didn't work. He suggested adding the following if statement (change offbattery to onbattery for that section):

if [ -f /etc/apcupsd/offbattery ] ; then
python /etc/apcupsd/offbattery
fi


It was one of those can't see the forest through the trees moments...

As for as the power fail not sending email, in my case, I have the router and modem on the UPS as well so I should get a last gasp message, and if not, there are always the log files...

Thomas said...

Thanks, this was very helpful. Exact what i was looking for.

Greetings from Germany,
Thomas

Danny Swan said...

I' "borrowed" your coding and setup to monitor our power at work. It works great. Now i just need to add one more thing, when the script is activated for power failure, i need it to switch a relay high. This in turn will be connected to our security system and will send signal to the monitoring company and they will make a phone call. not exactly sure how to do this, can it be added into the existing scripts or should it run seperately?

Joe T. said...

Very nice setup....I did this on the cyberpower ups i have protecting my fileservers. Can I call a bash script from this onbattery script?

Joe T. said...

Or...Can I call more than just the onbattery script?

Joe T. said...

Ok...I figured it out. I have is wait 2 min then if it's still down, email me and shutdown my fileservers.

Joachim said...

Thank you for this posting. Worked fine for me.

When configured, do not forget to edit the following file:
/etc/default/apcupsd

and set ISCONFIGURED=yes

Greetiings vrom Germany
Joachim

Kenneth Burgener said...

Thank you Joachim, updated the post.

Jeff McFadden said...

Thank you for the great post!

One thing that I missed as I was setting this up was that the `apctest` command must be run with super user privileges.

I realize that a few of your other commands need to be too (editing the configs, etc.), but that one especially caught me because I was getting an error that didn't indicate that it was a privileges issue.

In any case thanks for the writeup!

Brendan Keyport said...

Works great even three years later, Thanks!

Unknown said...

Can I use an ups for raspberry pi to provide power backup without having any other things to connect to ups

Joe T. said...

Don't know why not. I started this with the 1st Pi and it's now running on P3. I use it to power down my file servers and email me when power is lost.

Andy L said...

very helpful Thanks!

Phil Smith said...

Hi i am having a problem getting it to work with a cyber power cp600 and a pi with raspbx,
0.000 apcupsd: drivers.c:281 Warning: no UPS driver found (ups->mode.type=0).
apctest FATAL ERROR in apctest.c at line 268
apctest cannot continue without a valid driver.
can anyone help me?

René Estanislao Gutiérrez Vega said...

Worked perfect with my Cyberpower UPS, just changed emails/password in notification scripts and voila! Kudos!

Ralphes Bushman said...

Check out this blog about flexispy phone monitoring application. I bet it will be useful.

Anonymous said...

How do you shutdown your servers? SNMP? I want to shutdown my NAS (Qnap TAS-168) using SNMP, how can I send the SNMP command?

Joe T. said...

I found a python script for my Dlink DNS-323's on the forum for the NAS.

Fred Fremaux said...

Thanks KennethB, after a lot of research, your approach with the RPi is really clean and efficient. Had mine up and running easily. I have a question. ON Battery, OFF Battery is all good. What about this scenario: UPS Goes ON BATTERY, Message gets sent. UPS reaches your magic % run-time remaining and shuts down devices. UPS is OFF, so is RPi. POWER gets restored, UPS is back online, RPi is restarted. How do / or can I get notified that the power is back ON after the previous outage message.

-FrED

Ivan Berezoff said...

Please note if you’re intended to provide uninterruptible power for a refrigerator, you’ll need an UPS with SINE form of output signal. Any ordinary UPS used for PC-s produces meander form of signal in battery mode, which may cause damage to a fridge motor. The same is right for any electrical appliance unit with non-impulse mode of power consumption. So please don’t use UPS with meander or “approximated sine” forms of signal for fridges and similar household appliances. Those UPS are for PC-s only. Talk to a professional electrician, if necessary.
Good luck!

v1ncen7 said...
This comment has been removed by the author.
v1ncen7 said...

Did anybody perfect the scripts to send more info, like the output from apcaccess?

Joe T. said...

Call your script from the Onbattery script.