Tag: Python

An overview of the book "Effective Python"

For some time I have been working on moving from beginner to intermediate Python programmer. One step in that direction is reading the book “Effective Python – 59 Specific Ways to Write Better Python” by Brett Slatkin. Finally I have manged to finish the book during the holidays!

 

The book belongs to the “Effective” series of Addison-Wesley , which was started by Scott Meyers who is the author of “Effective C++” (1992).  All books in the series is about different topics of a specific subject. The topics all have separate items (sub-chapters), which are numbered. In the case of “Effective Python”, there are 59 specific topics covered. I really like the concept since it is possible to read one or two topics each day. Each topic is independent, hence no risk of not remembering what it was all about if having a longer break like a week.

The book has 8 broader chapters:
  1. Pythonic Thinking
  2. Functions
  3. Classes and Inheritance
  4. Metaclasses and Attributes
  5. Concurency and Parallelism
  6. Built-In Modules
  7. Collaboration
  8. Production

If being experienced in programming but a python newbie, chapter 1,2 and 3 is a really good introduction. The concepts and patterns are the same as in others programming languages but the chapters show how to apply it in python. For all of you that have some experience of python, it is no surprise that the python examples are shorter and easier to read and understand compared to other languages. Good advice is given together with examples regarding slicing, list comprehensions, arguments, return values, and iterators. Reading them in this condensed format saves a lot of time compared to acquiring the knowledge by reading discussions at stackoverflow.

Once understanding and tried the concepts of chapter 1-3, the chapters 7 and 8 are the most important. The chapters deal with the issues that always appear when a program starts to scale up, the team grows, and installing in a production environment. In chapter 7 the items give advice in docstrings, how to arrange packages and modules, using a root exception, and isolate dependencies using virtualenv. Chapter 8 is more about unittesting and interactive debugging, and how to configure for multiple different environments. Experienced programmers will recognize many items in this last chapter.

If you have gained experienced and developed some programs in python, chapters 4, 5 and 6 fit very well. Chapter 6 is about processes and threads, and explains well-known concepts like use locks to prevent races, coordinate threads using queues, and how to put jobs on different processor cores in python. Chapter 7 gives a short introduction into some of most important builtin modules. The last chapter to mention is 4 which contains some specific python concepts like metaclasses and attributes. Meta-classes, attributes and decorators are interesting concepts which gives new dynamic possibilities to expand classes during runtime or enforce checks every time a method of a class is called. Might be interesting if writing generic frameworks (data-binding) or mockup during unit-testing.

This is definitely a book that will increase the knowledge of a beginner to intermediate python programmer! Many of the items are easy to absorb and start using right away in your daily work, other items (or chapters) are something you might return to once you have reached the level where they make sense. Keep a copy of the book on your desk to look into once in a while, or why not start each day by a new item?

Supervising a Telldus Daemon

It is rare but sometimes it happens – the telldus daemon responsible for communicating with the Telldus Duo hardware has disappeared from the processlist, crashed or exited silently. This is not acceptable on an embedded system where processes shall have 100% uptime. The preferred way is to have some sort of supervisor which is responsible for restarting the process if it disappears.

Who is the naturally supervisor? Well, the process that started the process to be supervised. In Linux the first process started is either the old init or the newer systemd, which then starts all the other processes according to runlevels and dependencies (systemd) or sequence number (init/rc). There are a number of possible solutions for supervising using either init or systemd. A quick check on ‘google’ gives a couple of solutions, others probably exists as well:

  • daemontools – a supervisor which monitors processes, possible to use with init.
  • monit – a supervisor which monitors processes but also have a wealth of other monitoring possibility. Seems more targeted to server systems than embedded.
  • built in supervising in systemd – the systemd has a simple supervisor built in which restarts processes.
The websolution and the telldus daemon nowadays run on the raspbian called ‘Jessie’. Raspbian ‘Jessie’ has systemd as the first process that starts all other processes. The existing solution is built on init.d bash-scripts with links to the different runlevels (rcX). I decided to go with the systemd as the supervisor since it was time to transfer the init.d scripts to systemd services and only simple supervising was needed anyways.

In systemd you don’t write a script, instead you define a service having different properties and dependencies. This service configuration is put into /etc/systemd/system/<name>.service. The syntax is the same as .INI files on windows. See Service Configuration for details.

Telldus daemon as a systemd service

Explanations:

Type=forking

The telldus daemon is a forking daemon, i.e. it is forked off the starting process to be able to be automatically attached to the process having pid 1 (systemd in this case) making it independant of the terminal (tty).

 PIDfile=/var/run/telldusd.pid

The PID number is saved into this file when fork has been made, making it possible to find out the id of the process.

Restart=always

Always restart regardless the case abort, exception, signal or exit via return code.

Websolution (cherrypy) as a systemd service

Explanations:

After=network.target

Dependant on network connection before starting up.

Type=simple

Ordinary process i.e. doesn’t return if started since it is listening on network connections

User=root

Start process as root, since the webserver will switch to telldus user as soon as possible.
Environment=”PYTHONPATH=/usr/local/bin/setup”
Sets and exports the env. variable PYTHONPATH to the started process.

To make the services start call:

sudo systemctl enable telldusd
sudo systemctl enable cherrytelldus

Both solutions were tried out by killing the processes abruptly using ‘kill -kill ‘ and verified that they were restarted.

A Tellstick Sensor that Twitters

There are tellstick sensors measuring temperature and humidity, not with high precision but still ok if monitoring the house when being away. The sensors might be connected to a Tellstick NET from Telldus which sends all the measurement values up to the cloud (Telldus Live). However it is always a hassle to remember how to login to Telldus Live just to check the temperatures of the sensors.

It would be nice if one could read the temperature from some app already in use – twitter fits very well since it allows short text messages and it is possible to read both from an app and from a PC (no login required).

Telldus has an API to Telldus Live having the address: api.telldus.com. The API is a REST interface possible to access using GET requests and the text answers are in simple JSON format. It is definitely possible to to periodically fetch the temperature value of a sensor. Next step is to push the temperature value up to a twitter account. The twitter REST API was released as early as 2006 and have to be considered mature – see api.twitter.com. By using this API it is possible to automatically twitter status message and it might for example be used by IoT devices.

The Telldus API and the Twitter API use the standard OAuth (Open Authorization) to authorize clients. This standard is used by many companies including Facebook and Google. The standard makes it possible for service vendors to give limited access to certain parts of their API to registered clients. The registered clients might be applications, mobile apps, or back-end systems. OAuth requires you to generate 2 pair of keys, where one pair is authorization of the user or application (consumer key/consumer secret) and the other pair is authorization of the service request (access token/access secret). Once calling one of the REST API:s the public key is put directly in the HTTP authorization header and the secret key is used together with the other parameters to create a hash signature also going into the header.

If searching on pypi (the python package index), there is already a python library ‘tellive‘ for connecting to Telldus Live. The tellive library contains a client class which takes care of the constructing of the  REST url and the HTTP authorization header. However you still have to know a bit of how the REST api:s work to be able to use the correct arguments. Twitter on the other hand has several independent implementations calling the REST api:s. Among them is the python-twitter which seems pythonic and stable.

Now we have everything to put together a script which uses both the libraries ‘tellive-py’ and ‘python-twitter’:

The class ‘OAuthCredentials’ contains the two key pairs (consumer and access). The function ‘get_temperature’ uses the ‘tellive’ library by listing all sensors and find the value of requested sensor. The function ‘send_to_twitter’ uses the ‘python-twitter’ library to post a string containing the temperature to the twitter account belonging to the specified customer keys. Finally decide how often the script will run and make a crontab configuration.

Implementing a Web Solution for Tellstick Duo

A  web application for the Tellstick Duo targeting our household needs the following functions:

  • control individual switches (turn on/off lamps, and printer)
  • change schema mode (at which clock hour lamps will automatically turn/off)
  • also be usable on both android and windows phone on different screen sizes

It is all about either turning on/off switches or schema modes from a phone – best realized by using distinct and fairly large buttons. A status row is also needed to acknowledge an operation. Also the GUI should look “app-like”. Here is the end result:

In the blog post before this one, a walk through was made of the libraries needed to make a pyton web solution: cherrypy and jinja2. A little bit knowledge of html/javascript and the javascript library jquery mobile is also needed. Interaction with Tellstick Duo and setting up crontab is done using the already developed commands/libraries which are described in the former blog posts: Turn.py – Using Python to Control the Telldus and Setup.py – Configure Switches Using Crontab.

Below is an overview of the system design of the web application:


 html requests    +-------------------+     +--------------+
 REST API calls   |                   +---> |turn.py       |
+---------------> |    Cherrypy       |     +--------------+
                  |    Application    |
<-----------------+    python         |     +--------------+
   html           +----------+--------+---> |setup.py      |
                             ^              +--------------+
                             |   jinja2
                             |
                         +--------+
                         | +----+ |
                         |        |
                         | +----+ |
                         |        |
                         | +----+ |
                         +--------+

                       template page
                     html/jquery mobile/js
The cherrypy application instantiates the webserver which listens for connections on port 80. If a request is made for the main page, the template is read from flash and parsed by jinja2, in this case the schema modes and all the names of the switches are inserted into the page by jinja2. The webserver responds using the result from jinja2. Since jquery mobile is used in the main page each button is connected to a client side javascript callback. Inside the callback a REST API method call is made asynchronously to the webserver. The REST call arrives to the webserver which delegates it to a python function which in this case either calls turn.py (turn on/off switch) or setup.py (changes schema mode).

What is needed to change switches at certain intervals?

The whole point of having a tellstick connected to a raspberry pie is to let the raspberry control the switches in the house in a defined way. A telldus duo is also able do more like monitoring temperature and humidity sensors as well as PIR/magnetic sensors.
Ok, let’s stick to controlling the switches. One way is to build a program that falls asleep and then wakes up at certain times. In Linux however this functionality is built into the system process crontab. It is possible to configure crontab to run any command at a certain time each day for example. Crontab is very flexible and it is possible to have other configuration but the example covers what is most useful for lamp switches. How do you configure crontab? As always there is a easy to use library called ‘python-crontab’ where it is possible to configure crontab in an intuitive way.
Most often a lamp switch shall be turned on when dusk appears. However sunset is at different hours depending on the date and where you live geographically since the earth is moving around it’s axle. Luckily it is possible to calculate the dusk if the date and the geographically position is given. Another python library called ‘astral’ has everything needed since it possible to give it a location (through google map service) as well as the actual date, receiving the time of sun dawn/sun dusk/sun set.

Installing the mentioned libraries and their dependencies:

> pip install python-crontab
> pip install astral
> pip install pytz
> pip install nptime

Next post is how to write the setup program configuring the crontab using the libraries mentioned in this post!

Turn.py – Using Python to Control the Telldus

There are at least two python libraries for  communicating with the Telldus service. One of them is ‘tellcore-py’ which is an active github project (link). Installing the package by using pip (“python install python”):

 > pip install tellcore-py

The tellcore library is wrapper of the telldus C/C++ library which communicates directly with the Telldus service (the telldusd – daemon). This service was configured in last blog post and it uses the libusb library to send messages/commands via usb to the Tellstick DUO – see picture below.

                              +               
                 +---------+  |               
 tellcore.py +-> |telldusd/|  |  Tellstick DUO
                 |libusb   +---->             
                 +---------+  |               
                              +               
                                  

However in this case only interesting in a narrow use of the library i.e. turning on and off the switches. Later on if buying sensors it will be interesting collecting data through the use of this library.

Here is a small utility ‘turn.py’ written in python using the library, doing what I need – not more not less:

#!/usr/sbin/python
import sys
 import tellcore.telldus
 
 core = tellcore.telldus.TelldusCore()
 
 def turn(state, device):
     devices = [d for d in core.devices() if d.name == device or device == 'all']
 
     for d in devices:
        m = getattr(d, "turn_" + state)
        m()
   
 if __name__ == "__main__":
     if len(sys.argv) == 3:
        turn(sys.argv[1], sys.argv[2])
     else:
        print "turn "
 

Trying it out by first setting the executable flag and then running it directly by turning on/off switches

 > chmod u+x turn.py
 > turn on wallswitch3
 > turn off all

Getting Ready for Python

Really liking python more and more!

The windows support is also good – recommending to take a look at ironpython as well as python tools for visual studio (someone called it the best secret feature of visual studio).

Starting installing on the raspberry pi:

apt-get install python
apt-get install python-setuptools

apt-get install python-pip

The last two debian packages are important. They are needed to be able to add different python modules.