The Zen of Python

The Zen of Python, by Tim Peters, is guiding principles for Python's design.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

This list can also be generated within python by executing:

>>> import this

Dilbert RSS

Dilbert.com used to have an RSS feed, but has since removed this feature. Luckily, dilbert-rss was created by fredley, which allows anyone to create their own Dilbert RSS feed.

"dilbert-rss - Scrapes dilbert.com and generates an RSS feed.

Since dilbert.com nerfed their own RSS feed, I created a tool to replicate the old functionality. This is designed to be used on your own server, using cron to update it."

Usage

To use the dilbert-rss (assuming you want to store the RSS feed in /www/):

git clone https://github.com/fredley/dilbert-rss.git
cd dilbert-rss
python dilbert.py /www/dilbert.xml

Dependencies

Before you can use dilbert-rss, you will need to install two Python dependencies: BeautifulSoup and PyRSS2Gen

pip install BeautifulSoup PyRSS2Gen

BeautifulSoup and PyRss2Gen

On this topic, learning about the BeautifulSoup and PyRSS2Gen Python libraries were a benefit of using the dilbert-rss project. Both libraries can be used for a number of other useful projects. BeautifulSoup will parse an HTML page turning it into a usable Python object and PyRSS2Gen can be used to easily generate RSS feeds.

Oeey.com Dilbert Feed

An example of the output can be found here: http://dilbert.oeey.com/

Open the URL within an RSS viewer. If you don't have one handy, use Firefox to see the feed as a "Live Bookmarks" view. The content will be rendered. Most other browsers will just show the XML content.

With the above I created the following cronjob:

0 18 * * *   python /usr/local/bin/dilbert.py /www/dilbert/dilbert.xml

And the matching Apache configuration, with a default DirectoryIndex pointing to the XML feed:

<virtualhost>
    ServerName dilbert.oeey.com
    DirectoryIndex dilbert.xml
    DocumentRoot /www/dilbert
    ErrorLog logs/dilbert-error_log
    CustomLog logs/dilbert-access_log common
</virtualhost>

Update: Recently the Dilbert HTML layout changed. I had forked dilbert-rss and posted a suggested fix, to which fredley accepted the pull request.

Agile Manifesto

Manifesto for Agile Software Development

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on

the right, we value the items on the left more.

Twitter Logger with Python

Want a more modern method for simple logging and notifications? Twitter is very popular, but it normally too "noisy" for me to follow for social messages. Add a bit of tech and now I have a reason to check Twitter at least once a day, or follow the events on my phone.

I had read articles in both Linux Journal and The MagPi Magazine about engineering minds using Twitter to receive notifications when certain events occur. I decided to take this a step further and create a tiny Twitter logger script for all of my Raspberry Pis. I now receive notifications for all kinds of events, such as:

• Daily "alive" notifications
• Reboots, power cycles
• Changes in dynamic IP address
• GPIO event triggers (think security alarm, garage door, sprinkler system, etc)
• And anything else that fancies me at the moment

Code

How complicated is the Python? Easy. Only 3 lines (ignoring the line wrapping and commented debug code):

import twitter

api = twitter.Api(consumer_key='XXX',
                  consumer_secret='XXX',
                  access_token_key='XXX',
                  access_token_secret='XXX')

status = api.PostUpdate('This is my status update')

# print status.text

To use this code you will need to first generate your Twitter App API Access Keys and also install the python-twitter Python package.

Twitter App API Access Keys

Step 1 - Twitter Account: So how does one get started?  Well first, you need a Twitter account.  I assume you already have one, or at least don't need help creating one.

Step 2 - Mobile Phone: Unfortunately, your code will only be able to read status updates, unless you tie a mobile phone number to your account. Once you have validated your mobile phone, you will then be able to write/post status updates as well. Go to your account Settings and select the Mobile section. Add your mobile phone number, and wait for the validation code to appear on your phone. (note: I did attempt to try a couple of free burner mobile phone numbers, none of which ever received the validation code)

"You must add your mobile phone to your Twitter profile before granting your application write capabilities. Please read https://support.twitter.com/articles/110250-adding-your-mobile-number-to-your-account-via-web for more information"

Step 3 - Twitter App: Next, we need to create a Twitter App. This is nothing more than a method to get access to your api keys. Browse to https://apps.twitter.com/ and click the "Create New App" button.

Fill in the Name, Description and Website fields, then accept the agreement and continue.

The important field is the Name field as it has to be unique among all Twitter Apps. I just used my twitter account name, as I assumed no one else would be using that for an app name.

The Website field has to be populated, but won't be used. Put some temporary website and continue.

The Callback URL can be ignored.

Step 4 - Access Level: After the Twitter App has been created, find the Application Settings section and modify the Access level option by clicking the Modify App Permission link.  Change the access level from Read Only to Read and Write.  If you receive a warning about your mobile phone number, go back to step 2.

Step 5 - Access Keys: Finally, find the Application Settings section again and click the Manage Keys and Access Tokens link. You will find the consumer keys at the top of the page. Under the Your Access Token section, click the Create My Access Token button to generate the access tokens. If it isn't obvious the keys match the code variables as such:

consumer_key = "Consumer Key (API Key)"
consumer_secret = "Consumer Secret (API Secret)"
access_token_key = "Access Token"
access_token_secret = "Access Token Secret"

Now with your access keys, you can begin using the code above to update your twitter status with various notifications.

python-twitter

The python-twitter package "provides a pure Python interface for the Twitter API." To install the python-twitter package, simply use pip.

$ sudo pip install python-twitter

That was easy.

Check Status Code

The previous code showed how we can post a new update, but what about getting our current status, or someone else's?

import twitter
api = twitter.Api(...)

# get my timeline
statuses = api.GetUserTimeline()

# get a specific user's timeline:
# statuses = api.GetUserTimeline(screen_name=SCREEN_NAME)

# show latest message only:
print statuses[0].text

# show all returned messages:
for s in statuses:
       print s.text

Catch Errors

There are a few important errors that you will probably run across, and should handle:

Authentication failure: (check your keys, especially for accientally copied space characters)

twitter.error.TwitterError: [{u'message': u'Could not authenticate you', u'code': 32}]

Too long of a message: (shorten your message)

twitter.error.TwitterError: [{u'message': u'Status is over 140 characters.', u'code': 186}]

Duplicate message: (don't repeat messages or add a unique id, like time)

twitter.error.TwitterError: [{u'message': u'Status is a duplicate.', u'code': 187}]

User does not exist: (check the user name you are trying to query)

twitter.error.TwitterError: [{u'message': u'Sorry, that page does not exist', u'code': 34}]

test

Cronjob

I updated the twitter script to accept a message from the command line, and then added it to my cronjob:

@daily      /usr/bin/python /usr/local/bin/twitter alive
@reboot     /usr/bin/python /usr/local/bin/twitter reboot

Tips

#1 To be able to group and sort messages, I would recommend using the power of the hashtag. Pick a unique identifier (maybe your username) and append a category like this:

system startup #oeey_garage_pi

#2 To avoid the "Status is a duplicate" you may also wish to append a unique ID to all messages. I like to use the current linux epoc time, like this:

system startup #oeey_garage_pi (1415571010)

The updated posting code would look like this:

import time
...
status = api.PostUpdate('{message} #{category} ({id})'.format(
        message=message,
        category=category,
        id=int(time.time())

Issues

InsecurePlatformWarning:

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning

Solution: either ignore the warning or upgrade to at least Python 2.7.9 (or change code to use pyOpenSSL).

Other Libraries

If Python is not your thing, there a number of other Twitter libraries for various other programming platforms.

That's All Folks

Good luck, and have a good time playing with Python and Twitter.

Introduction to the Google Calendar API (HOWTO)

I have used Google Calendars to schedule and control a number of projects (eg. sprinkler system, alarm clock, etc) .  The following How To will get you started.  You will of course need a google account.

Create Project

1) Start by visiting the Google Developers Console

and create a new project.

2) Select the project and navigate on the left menu to APIs & auth then APIs and enable Calendar API for this project.  You can disabled all other APIs if you only need Calendar access.

3) Next, select the Consent screen menu option from the APIs & auth menu.  Enter a Product Name and select an Email Address.  If you do not do this step, you will get a Error: invalid_client error later on.

4) Next, select the Credentials menu option from the APIs & auth menu.  Under OAuth select Create new Client ID.

5) For the Create Client ID select Installed application for Application Type and select Other for Installed Application Type and finally click the Create Client ID button.

6) After the ID has finished being created, click the Download JSON button.  Save and rename the file as something simple like client_secret.json.

This json file contains your API credentials needed to access the Google Calendar APIs.

Install Google API Libraries

1) Install the Google API Libraries using Python's PIP installer:

$ sudo pip install --upgrade google-api-python-client

gflags may or may not be needed, depending on what code you use: (may be optional)

$ sudo pip install --upgrade python-gflags

If you would prefer alternatives, view the Google APIs Client Library for Python page.

Authorize Application

Next we will need to run our application and authorize it against the desired account.

1) Clone my sample code:

# git clone https://github.com/oeey/gcalendar.git

The sample code is just a slight modification from the Getting Started Sample Code.  The Google sample code has some outdated code that will throw some obsoleted warnings.

2) The application has not been authorized to an account yet.  Run the application once and you will be asked to paste a validation URL into your browser.  Login to your desired target account (with the calendars you want to access) and then paste the validation URL into your browser.

For convenience I have a first_auth.py script that is the same script as the gcalendar.py script, but terminates after authorization.  You can run any of the scripts to complete this authorization step.

The first_auth.py is pretty simple:

from apiclient.discovery import build
from oauth2client.file import Storage
from oauth2client.client import AccessTokenRefreshError
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run_flow
from oauth2client.client import flow_from_clientsecrets

def main():
    scope = 'https://www.googleapis.com/auth/calendar'
    flow = flow_from_clientsecrets('client_secret.json', scope=scope)

    storage = Storage('credentials.dat')
    credentials = storage.get()

    class fakeargparse(object):  # fake argparse.Namespace
        noauth_local_webserver = True
        logging_level = "ERROR"
    flags = fakeargparse()

    if credentials is None or credentials.invalid:
        credentials = run_flow(flow, storage, flags)

if __name__ == '__main__':
    main()

You may notice the "fakeargparse" code. The run_flow() call wants the flags to be set from the parameters pulled from argparse. I think that is overkill for what I needed, so I just created a fake container so run_flow() wouldn't complain.

Run the first_auth.py script to collect the application authorization.

$ python first_auth.py
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcalendar&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=1039XXXXXXXXXXXXXXXXXXXXXXXXXXcs46gdj2.apps.googleusercontent.com&access_type=offline

Enter verification code:

3) Copy the URL into the browser and accept the permissions.

4) You will be presented with a code, to which you will then enter back into the prompt of the first_auth.py application.  The authorization will be stored in the credentials.dat file for future requests.

$ python first_auth.py
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcalendar&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&client_id=46XXXXXXXXXX2-bXXXXXXXXXXXXXusvh6.apps.googleusercontent.com&access_type=offline

Enter verification code: 4/WzAQfXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2vw2M2Pl7OykQI
Authentication successful.

Now that we have our API credentials and are authorized to access an account, we can begin to play with the Google Calendars.

View Upcoming Events

The upcoming.py script builds off of the first_auth.py script, cycles through the next few upcoming calendar events and displays the event titles.

...
    if credentials is None or credentials.invalid:
        credentials = run_flow(flow, storage, flags)

    http = httplib2.Http()
    http = credentials.authorize(http)
    service = build('calendar', 'v3', http=http)

    print "Upcoming Events:"
    request = service.events().list(calendarId='primary')
    while request != None:
        response = request.execute()
        for event in response.get('items', []):
            print event.get('summary', 'NO SUMMARY')
        request = service.events().list_next(request, response)

This script defaults to the primary calendar associated with the account.

Calendar ID

The previous script defaults to the primary calendar associated with the account.  If you wish to specify an alternate calendar, you will need the Calendar ID.  A calendar's ID can be found on the Calendar Details setting page (same page you can change a calendar's name on).  Look for the Calendar Address line, and the Calendar ID will be in the parenthesis.  It will look something like "a3sd5221ap2qe5ksbev3ip4@group.calendar.google.com".

Next 12 Hours of Events

Finally, to specify a time range for events, I use the following code in my gcalendar.py script.  This code will collect the next 12 hours worth of events.

  ...

    service = build('calendar', 'v3', http=http)

    # get the next 12 hours of events
    epoch_time = time.time()
    start_time = epoch_time - 3600  # 1 hour ago
    end_time = epoch_time + 12 * 3600  # 12 hours in the future
    tz_offset = - time.altzone / 3600
    if tz_offset < 0:
        tz_offset_str = "-%02d00" % abs(tz_offset)
    else:
        tz_offset_str = "+%02d00" % abs(tz_offset)
    start_time = datetime.datetime.fromtimestamp(start_time).strftime("%Y-%m-%dT%H:%M:%S") + tz_offset_str
    end_time = datetime.datetime.fromtimestamp(end_time).strftime("%Y-%m-%dT%H:%M:%S") + tz_offset_str

    print "Getting calendar events between: " + start_time + " and " + end_time

    events = service.events().list(calendarId='primary', timeMin=start_time, timeMax=end_time, singleEvents=True).execute()singleEvents=True).execute()
    #pprint.pprint(events)
    for event in events['items']:
        print event["summary]

And this is the basis for the code I use to schedule my sprinkler system with.