Tuesday, December 29, 2009

When ctypes comes to the rescue

I recently purchased a DSLR (Canon 1000D) and almost at the same time I read an article about remote controlling your camera using gphoto. I tried it out and thought it was cool. During the holidays I had some time over to spend on hacking and wanted to try controlling my camera from Python. To my disappointment there were no Python bindings included with Ubuntu for gphoto. I did some googling but couldn't find any pre-compiled bindings, what to do?

Well, I could always try doing it with ctypes.

From the docs:
ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.
Note: I haven't done any serious stuff with ctypes nor gphoto before so if you find any errors etc please post a comment.

It amazed my how easy it is to use ctypes. Here's a snippet that will take a picture (from the first camera found), download the image to local storage and then delete it from the camera's storage.
import ctypes
import os

# gphoto structures
""" From 'gphoto2-camera.h'
typedef struct {
        char name [128];
        char folder [1024];
} CameraFilePath;
"""
class CameraFilePath(ctypes.Structure):
    _fields_ = [('name', (ctypes.c_char * 128)),
                ('folder', (ctypes.c_char * 1024))]

# gphoto constants
# Defined in 'gphoto2-port-result.h'
GP_OK = 0
# CameraCaptureType enum in 'gphoto2-camera.h'
GP_CAPTURE_IMAGE = 0
# CameraFileType enum in 'gphoto2-file.h'
GP_FILE_TYPE_NORMAL = 1

# Load library
gp = ctypes.CDLL('libgphoto2.so.2')

# Init camera
context = gp.gp_context_new()
camera = ctypes.c_void_p()
gp.gp_camera_new(ctypes.pointer(camera))
gp.gp_camera_init(camera, context)

# Capture image
cam_path = CameraFilePath()
gp.gp_camera_capture(camera,
                     GP_CAPTURE_IMAGE,
                     ctypes.pointer(cam_path),
                     context)

# Download and delete
cam_file = ctypes.c_void_p()
fd = os.open('image.jpg', os.O_CREAT | os.O_WRONLY)
gp.gp_file_new_from_fd(ctypes.pointer(cam_file), fd)
gp.gp_camera_file_get(camera,
                      cam_path.folder,
                      cam_path.name,
                      GP_FILE_TYPE_NORMAL,
                      cam_file,
                      context)
gp.gp_camera_file_delete(camera,
                         cam_path.folder,
                         cam_path.name,
                         context)
gp.gp_file_unref(cam_file)

# Release the camera
gp.gp_camera_exit(camera, context)
gp.gp_camera_unref(camera)
Ok, remember that I haven't done any wrapper or anything, it almost looks like 'C' code. I have also skipped all error checking for brevity.

You can always use a c_void_p if you don't need access to the data in Python (if you only need to pass a pointer between foreign functions). I'm using c_void_p instead of defining ctypes structures for gphoto's data types such as Camera and CameraFile. I still had to define CameraFilePath since I needed access to the data in Python.

I really like ctypes because I don't have to maintain code in C to access native functionality. Maybe you won't get the same performance as with traditional bindings but in this particular case it's not an issue.


Tuesday, December 22, 2009

Hello Planet Python!

I just got added to Planet Python and want to give the readers a quick introduction to myself.

My name is Mario and I live in Sweden. I've been working with Java development since 1999, mainly with enterprise systems. In 2007 I switched to embedded development and primary focusing on embedded Linux systems.

Early this year (2009) I got curious about developing KDE/QT applications since I've been using the KDE desktop for years now. I didn't want to use Java nor C++ for desktop development, I needed a language which was simple to use, feature rich and well supported by QT and KDE. There were two candidates for the job, Ruby and Python, guess which one I choosed.

I really like Python and it feels fun coding again, it's my first choice of programming language now.

On my blog you'll find posts about everything regarding Python, language features, APIs, frameworks, etc. I hope you'll enjoy reading.

Tuesday, December 15, 2009

try/for/while else... else what?

The Python language actually has support for else-clauses in some compound statements such as try, for and while.

So how do you use them?

I'll illustrate the for usage with an example:
def has_only_alpha_string(lst):
    for s in lst:
        if not s.isalpha():
            print '[{0}] contains non alphabetic character'.format(s)
            break
    else:
        print 'All strings contains only alphabetic characters'

lst = ['Hello', 'World!']

only_alpha_string(lst)
If lst is empty or exhausted the execution will continue in the else-clause. If the break is executed, the else-clause will not be executed. The same applies to while statements, the else is only executed if no break is executed within the while statement.

Note: It's valid to have a for/while-else without a break. In that case the else-clause will always be executed.

The first time I tried it I got it all wrong (before actually reading the documentation). I expected that the else-clause should be executed only if the list sequence was empty or exhausted, but it's the other way around.

What about try-else?

For me, at least, the try-else is probably a bit more intuitive.
try:
    f = None
    f = open('a_file', 'r')
except IOError as err:
    print err
else:
    print f.read()
finally:
    if f:
        print "Closing file..."
        f.close()
If the open call is successful the else-clause is executed and of course if the open call raises an IOError exception the else-clause isn't executed. Both open and read can raise IOError exceptions, the finally-clause will always be executed even if an exception occurs in the else-clause. In this particular case I'm only interested in catching the exception raised by open. If read raises an exception, it should be forwarded to the caller.

I could also write the code as following:
try:
    f = None
    f = open('a_file', 'r')
except IOError as err:
    print err
    # return or os.exit()

print f.read()
print "Closing file..."
f.close()
The problem with this solution is that file wouldn't be closed if read raises an exception.

Hope this post made the else-clause thing in combination with try/for/while more clear.


Tuesday, December 8, 2009

Profiling your Python code

Python provides support for deterministic profiling of your application. It's very easy to setup and use. To start profiling your code, two python modules are used:
  •  cProfile - the application that collects profiling data
  • pstats - the application that makes the profling data human readable
You can read more about Python profiling stuff at The Python Profilers page.

I'll show an example of how you can use the profiler.

Say I need to calculate the sum of all odd numbers from zero to an arbitrary positive value. My initial code might end up something like this:
def odd_numbers(max):
    """ Returns a list with all odd numbers between 0 to max (inclusive) """
    l = list()
    for i in xrange(max+1):
        if (i & 1):
            l.append(i)
    return l

def sum_odd_numbers(max):
    """ Sum all odd numbers between 0 to max (inclusive) """
    odd_nbrs = odd_numbers(max)

    res = 0
    for odd in odd_nbrs:
        res += odd
    return res

def main():
    # Run this 100 times to make it measurable
    for i in xrange(100):
        print sum_odd_numbers(1024)

if __name__ == '__main__':
    main()
Now I want to find out where my code spend most if its time to help me optimize the code if possible. To profile this snippet I run:
$ python -m cProfile sum_odd.py
This will output some statistics about the code (try it), but I'll show you a more handy way to browse and examine the profile dump.
$ python -m cProfile -o profile_dump sum_odd.py
This will output the profiling statistics to a file (in non-human readable format) which can be loaded and examined with pstats. Start pstats and browse the profile dump:
$ python -m pstats
Welcome to the profile statistics browser.                    
% help                                                        

Documented commands (type help topic):
========================================
EOF  add  callees  callers  quit  read  reverse  sort  stats  strip

Undocumented commands:
======================
help

% read profile_dump
profile_dump% stats
Tue Dec  8 20:55:41 2009    profile_dump

         51405 function calls in 0.186 CPU seconds

   Random listing order was used

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    51200    0.082    0.000    0.082    0.000 {method 'append' of 'list' objects}
      100    0.099    0.001    0.181    0.002 main.py:1(odd_numbers)
        1    0.000    0.000    0.185    0.185 main.py:1(module)
        1    0.000    0.000    0.186    0.186 {execfile}
      100    0.004    0.000    0.184    0.002 main.py:9(sum_odd_numbers)
        1    0.000    0.000    0.186    0.186 string:1(module)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.001    0.001    0.185    0.185 main.py:18(main)


profile_dump%
This will of course give the same information as if you just executed cProfile without specifying an output file. The advantage of using pstats interactively is that you can view the data in different ways.

Now I want to find out in which function we spend most time. This can be done by using the sort command:

profile_dump% sort
Valid sort keys (unique prefixes are accepted):
stdname -- standard name
nfl -- name/file/line
pcalls -- call count
file -- file name
calls -- call count
time -- internal time
line -- line number
cumulative -- cumulative time
module -- file name
name -- function name
profile_dump% sort time
profile_dump% stats
Tue Dec  8 20:55:41 2009    profile_dump

         51405 function calls in 0.186 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      100    0.099    0.001    0.181    0.002 main.py:1(odd_numbers)
    51200    0.082    0.000    0.082    0.000 {method 'append' of 'list' objects}
      100    0.004    0.000    0.184    0.002 main.py:9(sum_odd_numbers)
        1    0.001    0.001    0.185    0.185 main.py:18(main)
        1    0.000    0.000    0.186    0.186 {execfile}
        1    0.000    0.000    0.185    0.185 main.py:1(module)
        1    0.000    0.000    0.186    0.186 string:1(module)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
Nice! We can see that most time is spent in the odd_numbers function. The time key specifies that we would like to sort the data by the time spent in a function (exclusive calls to other functions).

Time to optimize, change the odd_numbers function to the following snippet:
def odd_numbers(max):
    """ Returns a list with all odd numbers between 0 to max (inclusive) """
    return [i for i in xrange(max+1) if (i & 1)]
Now profile the code and load the dump in pstats:
profile_dump% stats
Tue Dec  8 21:20:19 2009    profile_dump

         205 function calls in 0.020 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      100    0.015    0.000    0.015    0.000 main.py:1(odd_numbers)
      100    0.004    0.000    0.019    0.000 main.py:5(sum_odd_numbers)
        1    0.001    0.001    0.020    0.020 main.py:14(main)
        1    0.000    0.000    0.020    0.020 {execfile}
        1    0.000    0.000    0.020    0.020 main.py:1(module)
        1    0.000    0.000    0.020    0.020 string:1(module)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
Wow! Not that bad, we decreased the number of function calls from 51405 to 205. We also decreased the total time spent in the application from 0.186 to 0.020 CPU seconds by writting proper Python code :)

Tuesday, December 1, 2009

FUSE - Filesystem in Userspace part 3 (final)

Finally, as I promised, the last blog post on implementing file systems using FUSE.

I've created a file system, shoutcastfs, which enables you to mount the Shoutcast Radio directory as a file system. The genres are represented as directories and stations as files. Each file contains the station's playlist and the files are suffixed with .pls which makes it possible to load the playlist in a media player such as Amarok by double-clicking the file.

Of course, I'm using pyshoutcast (Python shoutcast API) to access the shoutcast service.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import errno
import fuse
import stat
import os
import shoutcast

fuse.fuse_python_api = (0, 2)

_shoutcastApi = shoutcast.ShoutCast()

class RootInfo(fuse.Stat):
    def __init__(self):
        fuse.Stat.__init__(self)
        self.st_mode = stat.S_IFDIR | 0755
        self.st_nlink = 2
        self._genres = {}

    @property
    def genres(self):
        if not self._genres:
            for g in _shoutcastApi.genres():
                self._genres[g] = GenreInfo(g)
        return self._genres

class GenreInfo(fuse.Stat):
    def __init__(self, name):
        fuse.Stat.__init__(self)
        self.st_mode = stat.S_IFDIR | 0755
        self.st_nlink = 2
        self.name = name
        self._stations = {}

    @property
    def stations(self):
        if not self._stations:
            for s in _shoutcastApi.stations(self.name):
                name = '{0}.pls'.format(s[0])
                name = name.replace('/', '|')
                self._stations[name] = StationInfo(name, s[1])
        return self._stations

class StationInfo(fuse.Stat):
    def __init__(self, name, station_id):
        fuse.Stat.__init__(self)
        self.st_mode = stat.S_IFREG | 0644
        self.st_nlink = 1
        # Hope no playlist exceeds this size
        self.st_size = 4096
        self.name = name
        self.station_id = station_id
        self._content = None

    @property
    def content(self):
        if self._content is None:
            self._content = _shoutcastApi.tune_in(self.station_id).read()
        return self._content

class ShoutcastFS(fuse.Fuse):
    def __init__(self, *args, **kw):
        fuse.Fuse.__init__(self, *args, **kw)
        self.root = RootInfo()

    def split_path(self, path):
        """ Returns genre and station """
        if path == '/':
            return (None, None)

        parts = path.split('/')[1:]
        if len(parts) == 1:
            return (parts[0], None)
        else:
            return parts

    def getattr(self, path):
        genre, station = self.split_path(path)

        if genre is None:
            stat = self.root
        else:
            stat = self.root.genres.get(genre)
            if not stat:
                return -errno.ENOENT

            if station:
                stat = stat.stations.get(station)
                if not stat:
                    return -errno.ENOENT
        return stat

    def readdir(self, path, offset):
        yield fuse.Direntry('.')
        yield fuse.Direntry('..')

        if path == '/':
            entries = self.root.genres.keys()
        else:
            entries = self.root.genres[path[1:]].stations.keys()

        for e in entries:
            yield fuse.Direntry(e)

    def open(self, path, flags):
        # Only support for 'READ ONLY' flag
        access_flags = os.O_RDONLY | os.O_WRONLY | os.O_RDWR
        if flags & access_flags != os.O_RDONLY:
            return -errno.EACCES
        else:
            return 0

    def read(self, path, size, offset):
        genre, station = self.split_path(path)
        info = self.root.genres[genre].stations[station]
        if offset < info.st_size:
            if offset + size > info.st_size:
                size = info.st_size - offset
            return info.content[offset:offset+size]
        else:
            return ''

if __name__ == '__main__':
    fs = ShoutcastFS()
    fs.parse(errex=1)
    fs.multithreaded = False
    fs.main()

To try the file system, run:
$ # Download shoutcast.py
$ wget http://github.com/mariob/pyshoutcast/raw/master/src/shoutcast.py
$ mkdir mnt
$ ./shoutcastfs mnt
$ cd mnt/
$ ls
...A list of genres...
$ cd Samba
$ ls
...A list of 'Samba' stations...
$ cat [station name]
...Playlist data...
Have fun!

Monday, November 23, 2009

Fedora 12

For the moment I'm evaluating Fedora 12. I was a bit disappointed on the Kubuntu 9.10 release, the distribution felt a little bit slow and tiered. The only thing I'm satisfied with is the boot time, it starts quite fast.

The KDE4 desktop in Fedora feels kind of snappier than KDE in Kubuntu. I'm not sure if it's because Fedora is compiled for i686 and not i386 or if the team have done any optimizations. I didn't enabled desktop effects in neither distribution. I installed the nvidia proprietary driver in Kubuntu but use Nouveau (open source driver) in Fedora. I might try the proprietary driver in Fedora to see if I gain any speed.

There's one bad thing I found in Fedora in comparision with Kubuntu, KPackageKit. It takes ages before starting compared to the Kubuntu version so I'm using yum from the shell prompt instead. Well, to be honest, I'm not a heavy user of UI front-ends to package management systems. In Kubuntu I prefer using aptitude instead of KPackageKit.

I think I'll give Fedora a chance and try it out for a while.

Thursday, November 19, 2009

FUSE - Filesystem in Userspace part 2

This is the second post on implementing file systems using FUSE. In this post I'll show you how to create a read-only file system that is backed by an XML file. Each tag in the XML file is either a file or a directory. If the tag is representing a directory, it should have a is_dir="1" attribute added. When the tag represents a file it should have its content between the tags. The following XML snippet shows an example:
<root is_dir="1">

  <a is_dir="1">
    <file1.txt>File content for this file</file1.txt>
    <hello>Hello World</hello>
  </a>

  <empty_dir is_dir="1" />

</root>
The example models the following directory tree:
/
|-- a
|   |-- file1.txt
|   `-- hello
`-- empty_dir
There are some limitations in having an XML file representing a file system. You can't for example have filenames starting with a dot because tags are not allowed to start with a dot in XML.

On to the code, the following snippet implements the read-only file system which is backed by an XML file:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import errno
import fuse
import stat
import os
import time
import xml.etree.ElementTree as etree

fuse.fuse_python_api = (0, 2)

# Use same timestamp for all files
_file_timestamp = int(time.time())

class MyStat(fuse.Stat):
    """
    Convenient class for Stat objects.
    Set up the stat object with appropriate
    values depending on constructor args.
    """
    def __init__(self, is_dir, size):
        fuse.Stat.__init__(self)
        if is_dir:
            self.st_mode = stat.S_IFDIR | 0555
            self.st_nlink = 2
        else:
            self.st_mode = stat.S_IFREG | 0444
            self.st_nlink = 1
            self.st_size = size
        self.st_atime = _file_timestamp
        self.st_mtime = _file_timestamp
        self.st_ctime = _file_timestamp

class MyFS(fuse.Fuse):
    def __init__(self, xml_tree, *args, **kw):
        fuse.Fuse.__init__(self, *args, **kw)
        self.tree = xml_tree

    def getattr(self, path):
        # We do not support 'dot' files
        # since xml tags cannot start with a dot.
        if path.find('/.') != -1:
            return -errno.ENOENT

        entry = self.tree.find(path)

        if entry is None:
            return -errno.ENOENT
        else:
            is_dir = entry.get('is_dir', False)
            size = entry.text and len(entry.text.strip()) or 0
            return MyStat(is_dir, size)

    def readdir(self, path, offset):
        yield fuse.Direntry('.')
        yield fuse.Direntry('..')
        for e in self.tree.find(path).getchildren():
            yield fuse.Direntry(e.tag)

    def open(self, path, flags):
        # Only support for 'READ ONLY' flag
        access_flags = os.O_RDONLY | os.O_WRONLY | os.O_RDWR
        if flags & access_flags != os.O_RDONLY:
            return -errno.EACCES
        else:
            return 0

    def read(self, path, size, offset):
        entry = self.tree.find(path)
        content = entry.text and entry.text.strip() or ''
        file_size = len(content)
        if offset < file_size:
            if offset + size > file_size:
                size = file_size - offset
            return content[offset:offset+size]
        else:
            return ''

if __name__ == '__main__':
    tree = etree.parse('tree.xml')

    fs = MyFS(tree)
    fs.parse(errex=1)
    fs.main()
I'm using the ElementTree XML API to parse the XML file. The ElementTree API is very easy to use and supports finding elements by specifying a path which fits very well in this context. For example, the following snippet shows how to get the file1.txt element from the example XML file above and extract the content between the tags:
import xml.etree.ElementTree as etree
tree = etree.parse('tree.xml')
el = tree.find('/a/file1.txt')
print el.text.strip() # Remove any white-spaces
The code hasn't change that much since the part 1 post. I've added two new methods which adds support for opening and reading files. The MyStat class is only used as an convenient class to help creating appropriate stat objects. You might notice that I don't do a lot of checking in the code, this is because FUSE do a lot of them for me, this page list some of the assumptions you can make when implementing file system using FUSE.

Did you believe it would be this easy to create a mountable file system that uses an XML file for the layout? I didn't.

In the next (and final) post about FUSE I think I'll create some kind of 'use a service on the Internet' file system which can be useful and not just another toy fs.

Sunday, November 15, 2009

FUSE - Filesystem in Userspace part 1

Here's the first part in a series of posts on implementing file systems using FUSE.

Creating a file system can seem to be an intimidating task but with FUSE it's very easy and it gets even easier with python bindings. Any decent Linux distribution should include support for FUSE and be configured such that regular users (or a group) can mount file systems written with FUSE without having root access.

If you are running Ubuntu you need to install the python-fuse package which contains the Python bindings for FUSE.

- Enough, show me the code

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import errno
import fuse
import stat
import time

fuse.fuse_python_api = (0, 2)

class MyFS(fuse.Fuse):
    def __init__(self, *args, **kw):
        fuse.Fuse.__init__(self, *args, **kw)

    def getattr(self, path):
        st = fuse.Stat()
        st.st_mode = stat.S_IFDIR | 0755
        st.st_nlink = 2
        st.st_atime = int(time.time())
        st.st_mtime = st.st_atime
        st.st_ctime = st.st_atime

        if path == '/':
            pass
        else:
            return - errno.ENOENT
        return st

if __name__ == '__main__':
    fs = MyFS()
    fs.parse(errex=1)
    fs.main()
The example above shows how easy it is to create a mountable file system, it does nothing but is still a valid file system. The MyFS class implements a file system with no entries, it will always return ENOENT (No such file or directory) except for the root path. You should take a look at the FUSE Python Reference page which contains details about FusePython and, among other things, describes the stat object returned from the getattr method.

Paste the example to a file, MyFS.py, and make the file executable:
$ chmod 755 MyFS.py
Now create a directory which will be used as mount point and mount the file system:
$ mkdir myfs
$ ls -l
totalt 4
drwxr-xr-x 2 mario mario 4096 2009-11-15 15:59 myfs
-rwxr-xr-x 1 mario mario  648 2009-11-15 15:59 MyFS.py
$ # This will mount the file system, try ./MyFS.py --help
$ ./MyFS.py myfs
$ ls -l
totalt 0
drwxr-xr-x 2 root  root    0 2009-11-15 16:10 myfs
-rwxr-xr-x 1 mario mario 648 2009-11-15 15:59 MyFS.py
Here we can see that the owner, size and time stamp changed on myfs after we mounted the file system. You can cd into myfs, but if you try to list the directory (file system root) you'll get an error:
$ cd myfs
$ ls
ls: reading directory .: Function not implemented
You got this error because we haven't implemented the readdir method which is invoked when listing a directory. To make this work we need to add a readdir method which returns (at least) the '.' and '..' entries. Add the following snippet to the class:
    def readdir(self, path, offset):
        for e in '.', '..':
            yield fuse.Direntry(e) 
Before you continue you should unmount the file system:
$ fusermount -u myfs
This will unmount your FUSE file system as a regular user and doesn't require root access.

Retry to mount and cd into the myfs directory:
$ ./MyFS.py myfs
$ cd myfs
$ ls -a
.  ..
Alright, I think I'll stop for today. Oh, just one more thing... You won't get any print outputs nor tracebacks from your code if you don't start the file system in foreground mode. This is done by adding '-f' as argument when starting the file system.
$ ./MyFS.py -f myfs
It can be hard to tell why something doesn't work if you can't see tracebacks and print outputs.

Thursday, November 5, 2009

GMonitor

Ah, I've finally managed to make my new fresh Kubuntu 9.10 installation usable. Now it's time to do some KDE stuff and I'll show you how easy it's to create a plasmoid (applet/widget) for KDE4.

I've created a simple applet which shows the number of unread mail in a Gmail inbox. We already know how to fetch the list with unread mail, which I have explained in a previous blog post. Now I'll create a nice applet showing the count. I'm going to keep it as simple as possible, writing hard-coded values, skip error handling and so on to avoid making things more complicated than necessary.

There are several good tutorials at techbase.kde.org which explains plasma programming. They cover the basics and explains things in detail so I'll only make references to the pages instead of repeating what's already been written.

The first tutorial you should read (you don't have to read it right now, you can read it after you tried doing the applet) is Python Plasma Getting Started. The tutorial covers setting up a simple plasmoid, packaging, installing and running.

I'll name the plasmoid GMonitor because it only monitors the mailbox and shows the number of unread mail, it will not actually try to do some kind of notification (yet?!).

I've put the code on github, you can clone it with:
$ git clone git://github.com/mariob/gmonitor.git
You'll find a Makefile in the git which can be useful when doing plasmoids. The Makefile supports installing, un-installing, viewing, packaging and updating the plasmoid.

The directory tree looks like this:
gmonitor/
|-- Makefile
|-- README
|-- contents
|   `-- code
|       `-- main.py
`-- metadata.desktop
The gmonitor directory is the project home and contents/code contains the python source. The metadata.desktop file will be explained below.

To be able to install and run your applet you need to provide a metadata.desktop file to plasma. The metadata.desktop file contains important information about the applet and you can read more about the file in the Plasma Getting Started tutorial.

Feel free to change the file. The Name specifies the applet name and the Icon field gives the name to the icon to associated with this applet. These two fields are typically shown when listing applets in the 'Add Widget' dialog. There are two important fields which must be present for plasma to run your applet, the X-Plasma-API field specifies which script-engine to be used and the X-Plasma-MainScript field which script to be executed. It's also good to know that the X-KDE-PluginInfo-Name fields is used as a plasmoid identification, so it should be unique.

Some notes about the implementation. There are two classes, GMonitor and MailFrame (I know, poor name). The MaiFrame class has an icon and a label. When the mail count is set to zero (by calling setCount(cnt)), the icon is disabled, grayed and the label is set to 'No new mail'. When the mail count is set to a value greater than zero, the icon is enabled, colored and the label is set to 'Count: x', where x is the number of mail in the inbox. See the images below.



Sample of what the applet looks like

When the icon is enabled and clicked, a 'clicked()' signal is emitted. The GMonitor class connects the openBrowser() method to the 'clicked()' signal. The openBrowser() method opens a default browser in KDE to load the google mail url. The GMonitor also sets up a timer to fetch the feed in 60 second interval. The fetchFeed() method uses the KIO framework in KDE to download the content from a URL and parseFeed() parses the downloaded feed and emits a 'mailcount' signal.

The implementation depends on feedparser which can be installed on Ubuntu by running:
$ sudo aptitude install python-feedparser
You can also download and put the feedparser.py file inside the code directory.

Not that bad. It's actually possible to create a very simple 'good looking' Gmail monitor in less than 100 lines of code (excluding comments) with Python and KDE.

To view this plasmoid without installing it run (while standing in the gmonitor folder):
$ make view
This might only work on KDE4.3, not sure if the plasmoidviewer in 4.2 supports running plasmoids without first installing them. If it fails you should install the plasmoid and run plasmoidviewer yourself (or modify the makefile):
$ make install
$ plasmoidviewer pysnippet-gmonitor
If you never heard of Qt signals and slots you should read this introduction on the topic. For example, a button can 'emit' a signal when it's clicked. A slot (function/method) can be connected to a signal and each time the signal is emitted the function/method will be called.

You can find the Python KDE4.3 API docs here. Unfortunately, I don't find them as good as the Qt docs.

When running the applet, KDE will ask for the username/password when it tries to connect to Google. You can check the 'remember the password' checkbox which will make KDE cache the password as long as the desktop session is alive. If you choose not to, you'll have to enter your username/password each time the code tries to fetch the feed. A better solution is to use KWallet.

Ok, hope you enjoyed it and start doing cool plasmoids!

Monday, November 2, 2009

Kubuntu 9.10

I've installed Kubuntu 9.10 and I'm busy configuring it to a usable state. For the moment I don't have too much spare time but I'm almost done.

I hope I'll have time to do a post about creating plasmoids (KDE4 desktop widgets) in the next couple of days. I think I'll do a gmonitor applet which will show the number of unread mail in your Gmail inbox.

Wednesday, October 28, 2009

DIY: Google Mail Notifier

It's quite easy to create your own GMail notifier. Your inbox, gmail or domain,  can be monitored by fetching and parsing an atom feed.

Following code snippet prints new (unread) mail in your inbox:
# -*- coding: utf-8 -*-
import feedparser
import urllib2

# Create http basic auth handler
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password('New mail feed', 'https://mail.google.com/',
                          'USERNAME@DOMAIN', 'PASSWORD')

# Open url using the auth handler
opener = urllib2.build_opener(auth_handler)
feed_file = opener.open('https://mail.google.com/mail/feed/atom/')

# Parse feed using feedparser
d = feedparser.parse(feed_file)

# Print mail count and mails
print 'Mail count:', d.feed.fullcount

for entry in d.entries:
        print '----------------------------------------------'
 print 'Author: ', entry.author
 print 'Subject:', entry.title
 print 'Summary:', entry.summary

Just change USERNAME and PASSWORD. You also need to download feedparser which is free to use, or write one yourself.

Have fun!

ELCE 2009 presentations

The presentations from ELCE 2009 are now available from here. Some of the presentations have not been posted yet.

Monday, October 26, 2009

pyshoutcast v0.7.1

pyshoutcast version 0.7.1 has just been released. It implements a Python shoutcast API which makes it possible to list genres, stations, do search, etc, using the shoutcast yellow pages service.

Unfortunately, I just found a forum post stating that Amarok and VLC have removed support for shoutcast due to licensing issues. I'm not sure why because I can't find any info about it, I'll try to find out more details.

Anyway, to try it out you can start a interactive python prompt and paste the following code:
>>> import shoutcast
>>> api = shoutcast.ShoutCast()
>>> api.random()

Wednesday, October 21, 2009

'with' statement

Python have a great number of cool language features and one of them is the with statement. The with statement can be used in numerous ways but I think that the most common usage is to track and release some kind of resource and at the same time encapsulate the code block with a try/finally clause.

For example:
with open('my_file.txt') as f:
    for line in f:
        print line
which could be translated to something like this:
f = open('my_file.txt')
try:
    for line in f:
        print line
finally:
    f.close()
and in a general form:
with <EXPRESSION> [as <VAR>]:
    <BLOCK>
The with statement guarantees that the file will be closed no matter what happens, for example an exception is thrown or a return is executed inside the for statement etc. The result from the expression that is evaluated after the with keyword will be bound to the variable specified after the as keyword. The as part is optional and I'll show you a use case later.

But wait a minute, how does the with keyword know how to close the file? Well, the magic is called 'Context Manager'.

A context manager is a class which defines two methods, __enter__() and __exit__(). The expression in the with statement must evaluate to a context manager else you'll get an AttributeError. The __enter__ method is called when the with statement is evaluated and should return the value which should be assigned to the variable specified after the as keyword. After the block has finished executing the __exit__ method is called. The __exit__ method will be passed any exception occurred while executing the block and can decide if the exception should be muted by returning True or be propagated to the caller by returning False.

Let's look at the following example (not a good design, just an example):
class TransactionContextManager(object):
    def __init__(self, obj):
        self.obj = obj

    def __enter__(self):
        self.obj.begin()
        return self.obj

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type:
            self.obj.rollback()
        else:
            self.obj.commit()
        return False

class Transaction(object):
    def begin(self):
        print "Start transaction"

    def query(self, sql):
        print "SQL:", sql

    def rollback(self):
        print "Rollback transaction"

    def commit(self):
        print "Commit transaction"

print "Before 'with'"

with TransactionContextManager(Transaction()) as tx:
    tx.query("SELECT * FROM users")

print "After 'with'"
The code produces the following output:
Before 'with'
Start transaction
SQL: SELECT * FROM users
Commit transaction
After 'with'
It's easy to trace the execution but I'll explain it anyway.
  1. The code starts by printing "Before 'with'".
  2. The with statement is executed and instances of TransactionContextManager and Transaction are created. The context manager's __enter__ method is invoked which will invoke the Transaction instance's begin method. The begin method prints "Start transaction" and finally the __enter__ method returns the Transaction instance.
  3. The return value from the context manager's __enter__ method is bound to the variable tx.
  4. The block is executed and the query method is called.
  5. The query method prints "SQL: SELECT * FROM users".
  6. The block is done executing and the context manager's __exit__ method is invoked. Since no exception occurred the commit method is invoked on the Transaction instance. The __exit__ method returns False which means that any exceptions should be propagated to the caller, none in this case.
  7. The code ends executing by printing "After 'with'".
Let's change the Transaction class so that it throws an exception from the query method. This should execute the rollback method instead of the commit method. The exception should also be propagated to the caller since we return False from the __exit__ method.
def query(self, sql):
    raise Exception("SQL Error occurred")
And the output looks like following:
Before 'with'
Start transaction
Rollback transaction
Traceback (most recent call last):
  File "transaction.py", line 32, in 
    tx.query("SELECT * FROM users")
  File "transaction.py", line 21, in query
    raise Exception("SQL Error occurred")
Exception: SQL Error occurred
Here we see that the rollback method is called instead of commit. The rollback method is called because the code inside the with block raised an exception and a valid exc_type argument was passed to the __exit__ method. We can also see that the exception is propagated to the caller because we return False from the __exit__ method.

If we change the __exit__ method to return True instead of False we get the following output:
Before 'with'
Start transaction
Rollback transaction
After 'with'
As expected, we don't see the exception anymore. Note that the __exit__ method is not called if an exception occurs before the with statement starts executing the block.

Alright, I hope you agree with me that the with statement is very useful. Now I'll show you a final example where we don't need to bind the return value from the context manager to a variable. I've re-factored the previous example and hopefully made a better design.
class Connection(object):
    def __init__(self):
        # Default behaviour is to do auto commits
        self._auto_commit = True

    def connect(self):
        pass

    def auto_commit(self):
        return self._auto_commit

    def set_auto_commit(self, mode):
        print "Auto commit:", mode
        self._auto_commit = mode

    def rollback(self):
        print "Rollback transaction"

    def commit(self):
        print "Commit transaction"

    def executeQuery(self, sql):
        # Handle auto commit here if it's enabled
        print "SQL:", sql

class Transaction(object):
    def __init__(self, conn):
        self.conn = conn
        self.auto_commit = False

    def __enter__(self):
        self.auto_commit = self.conn.auto_commit()
        self.conn.set_auto_commit(False)
        return self.conn

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type:
            self.conn.rollback()
        else:
            self.conn.commit()

        self.conn.set_auto_commit(self.auto_commit)
        return False

conn = Connection()
conn.connect()

with Transaction(conn):
    conn.executeQuery("SELECT a user FROM users")
    conn.executeQuery("INSERT some data INTO users")
The output:
Auto commit: False
SQL: SELECT a user FROM users
SQL: INSERT some data INTO users
Commit transaction
Auto commit: True
As the example shows, we don't need to use the return value from the context manager to still have use of them. We only want to be sure that the queries inside the with block is executed in the same transaction and rollbacked if an exception occurs or commited if successful. Without the with statement the code would be implemented something like the following snippet:
conn = Connection()
conn.connect()
auto_comm = conn.auto_commit()

try:
    conn.set_auto_commit(False)
    conn.executeQuery("SELECT a user FROM users")
    conn.executeQuery("INSERT some data INTO users")
    conn.commit()
except:
    conn.rollback()
    raise
finally:
    conn.set_auto_commit(auto_comm)

Well, for me the with statement seems more clean and convenient to use than the last example.

Sunday, October 18, 2009

Back home from ELCE 2009


It has been a great visit to Grenoble. I'm pleased with the conference because most of the sessions were held by developers, which I prefer, and not marketing people. The major focus seemed to be on build systems and reducing boot time. I learned a lot about tools which can give you hints on performance issues etc. Some celebrities also showed up like Jon Masters and Alessandro Rubini.

I assume that the slides from the sessions will be available on the home page so you can take a look at them yourself.

Adeneo had a cool demo of booting up a Linux system with a Qt-app doing 3D (software rendering) in less than 5 seconds. Unfortunately I can't give you a clip since the new blogger editor doesn't support movie clips, or am I wrong?. A picture will do.


The conference had only a little exhibiton hall (which was ok for me) and here's a picture of Free Electrons' booth.


The food was a success! A lot of different finger food and of course wine. Here's an example of what the dessert looked like the second day.


The restroom has a really cool designed washbasin.


As I mentioned before I'm pleased with the conference and I believe I'll attend again next year.

Tuesday, October 13, 2009

I'm going to ELCE 2009

I'll be visiting the Embedded LInux Conference Europe 2009 which is held in Grenoble, France. It's a two day conference, 15-16 October. I'm looking forward to attend the sessions about build systems and other cool low-level kernel stuff.

Hope I'll meet some interesting people.

Saturday, October 10, 2009

KDE Hello World

I have plans to do a web radio player (I wonder if we'll ever see this app). My intention is to learn Qt/KDE programming in Python. I've played around with Qt a bit and I'm starting to get comfy. I've decided to try doing the player as a KDE app, just to learn more about KDE. Qt is great for doing cross-platform stuff but I want to take a look at what KDE offers on top of Qt. You can find docs, tutorials and more on KDE TechBase. The following snippet shows the minimum amount of code needed to create a typical KDE app in Python.
from PyKDE4 import kdecore
from PyKDE4 import kdeui
import sys

def createAboutData():
    """
    Create a KAboutData with information about this application.
    """
    return kdecore.KAboutData(
            # Program name used internally
            "hello",
            
            # Catalog name
            "",                       
            
            # Displayable program name
            kdecore.ki18n("Hello World app"),
            
            # Program version
            "0.1.0",
            
            # Short description about the program
            kdecore.ki18n("A simple KDE Hello World app"),
            
            # Program license
            kdecore.KAboutData.License_BSD,
            
            # Copyright statement
            kdecore.ki18n ("(c) 2009 Mario Boikov"),
            
            # Free form text
            kdecore.ki18n("Free form text\nsupporting newlines"),
            
            # Home page address for this program
            "http://www.pysnippet.com",
            
            # Bug report email address
            "mario@beblue.org",
            )

def main():
    about = createAboutData()
    kdecore.KCmdLineArgs.init(sys.argv, about)

    app = kdeui.KApplication()

    # INSERT APP CODE HERE

    # Start event loop
    sys.exit(app.exec_())

if __name__ == '__main__':
    main()
Nothing special will happen if you execute this application, we need some kind of window/widget to display something. To accomplish this we can create a KMainWindow with a KPushButton. We'll also add code to print Hello World when the button is clicked. The following snippet will create a window with a push button:
from PyKDE4 import kdeui
from PyKDE4 import kdecore
from PyQt4 import QtCore

class MainWindow(kdeui.KMainWindow):
    def __init__(self):
        super(MainWindow, self).__init__()

        # Create widget with vertical box layout manager
        layout = kdeui.KVBox()

        # Create push button and add to layout
        button = kdeui.KPushButton(kdecore.i18n("Push Me!"), layout)

        # Connect to button's 'clicked' signal 
        QtCore.QObject.connect(button, QtCore.SIGNAL("clicked()"),
                               self.buttonClicked)

        self.setCentralWidget(layout)

    def buttonClicked(self):
        print "Hello World!"
You might wonder why I use ki18n() when creating the KAboutData and i18n() for the MainWindow. Both functions do translations, i18n() requires an instance of KApplication before use but ki18n() don't.

To actually show the window we have to instantiate it and make it visible, the "INSERT APP CODE HERE" comment should be replaced with the following lines:
    mainWindow = MainWindow()
    mainWindow.show()
As you can see, it doesn't take too much effort to create a simple app in KDE. There's almost more code to create the KAboutData than doing the window with a push button. Here's a screen shot of what the window looks like on my Kubuntu installation:



If we compare Qt and KDE there aren't too much differences, at least with this example. The code below implements the same Hello World app made with Qt:

import sys
from PyQt4 import QtCore, QtGui

class MainWindow(QtGui.QMainWindow):
    def __init__(self, title):
        super(MainWindow, self).__init__()
        self.setWindowTitle(title)

        centralWidget = QtGui.QWidget()

        button = QtGui.QPushButton("Push Me!")

        layout = QtGui.QVBoxLayout()
        layout.addWidget(button)

        centralWidget.setLayout(layout)

        QtCore.QObject.connect(button, QtCore.SIGNAL("clicked()"),
                               self.buttonClicked)

    self.setCentralWidget(centralWidget)

    def buttonClicked(self):
        print "Hello World!"

def main():
    app = QtGui.QApplication(sys.argv)

    mainWindow = MainWindow("Hello World App")
    mainWindow.show()

    sys.exit(app.exec_())

if __name__ == "__main__":
    main()

The biggest difference is how the layout is handled and the absence of About Data.

Wednesday, October 7, 2009

Python and unit testing

Writing unit tests in python is really easy. It's the dynamic properties of the language which makes it very simple to mock class methods, instance methods and functions. This makes the task of writing isolated unit tests easy when you practice test-driven development.

Say you have the following class:
import urllib2

class UrlDownloader(object):
    def download(self, url):
        """
        Read the content from the url.
        The content is returned as a string object.
        """
        return urllib2.urlopen(url).read()
You actually don't want to download something from the net when you do a unit test for this class. You would like to have a fixed string returned from the read() method. This can be hard to achieve in some languages (for example Java) but that's not true for Python. Python supports redefining class methods, functions and other things in run-time. The following snippet shows how easy it's to mock the urlopen() function in the urllib2 module to return a mocked file-like object:
import urllib2
import unittest

class MockFileObject(object):
    def read(self):
        return "DATA"

def mock_open(url):
    return MockFileObject()

class UrlDownloaderTest(unittest.TestCase):
    def testDownload(self):
        urllib2.urlopen = mock_open
        dl = UrlDownloader()
        self.assertEqual(dl.download(""), "DATA")

if __name__ == '__main__':
    unittest.main()
In this unit test we redefine the urlopen-function (remember, this is not a member of an instance, it's a plain function) with our own which returns a mock file-like object. It's not that hard to imagen all the possibilities this gives when writing tests. There are of course frameworks that will help you with creating mocks etc, you could try The Python Mock Module which should cover most of the use cases.

Friday, October 2, 2009

File-like objects

A Python file-like object can be seen as an abstraction to input/output streams. They are actually similar to Java's java.io.Reader/java.io.Writer abstraction (I'm coming from Java). The typical usage of file-like objects can be found in frameworks and APIs. The file-like object provides a generic way of reading/writing data and would fit an HTML-parser for example. The parser is not interested of where the data come from (files, sockets, etc), only the content, so naturally the implementation should not depend on how to open resources etc.

Basic methods that need to be implemented by a file-like object are:
  • read([size]) - read at most size bytes from the file.
  • readLine([size]) - read one entire line from the file.
  • write(str) - write a string to the file.
  • close() - close the file.
You should read the documentation of the function/method that uses the file-like object, often they specify which methods that are expected to be implemented by the file-like object. If the file-like object is only used for reading, no write-functions need to be supported. More information about file-like objects can be found here.

Ok, so why should you use file-like objects in your code?

Let's take a look at the ConfigParser class which can be used to read/write configuration data. You can expect to find methods for reading and writing configuration data using files but the class does also include support for file-like objects, which is great. For example it's very easy to write unittests for this class, thanks to the support of file-like objects,  without having to create real files with test data. To create a file-like object from a string we can use the StringIO module. The module have both read and write support.

Here's some code:
import unittest
import StringIO
import ConfigParser

class ConfigParserTest(unittest.TestCase):

    def testHasSection(self):
        """
        has_section() should return True if the section exists
        and False if is doesn't.
        """
        data = StringIO.StringIO("[My section]\nSetting=1\n")
        config = ConfigParser.ConfigParser()
        config.readfp(data)
        self.assertTrue(config.has_section("My section"))
        self.assertFalse(config.has_section("Not a section"))

    def testMissingHeader(self):
        """
        readfp() should raise MissingSectionHeaderError
        when parsing a file without section headers.
        """
        data = StringIO.StringIO("\nSetting=1\n")

        config = ConfigParser.ConfigParser()
        self.assertRaises(ConfigParser.MissingSectionHeaderError,
                          config.readfp, data)

if __name__ == "__main__":
    unittest.main()
And a simple write configuration test:
import unittest
import StringIO
import ConfigParser

class WriteConfigTest(unittest.TestCase):

    def testWriteConfig(self):
        """
        Test write a simple config
        """
        data = StringIO.StringIO()
        config = ConfigParser.ConfigParser()
        config.add_section("My Section")
        config.set("My Section", "my_option", "my_value")
        config.write(data)

        expected = "[My Section]\nmy_option = my_value\n\n"
        self.assertEquals(expected, data.getvalue())

if __name__ == "__main__":
    unittest.main() 
These unittests are of course only examples and do not fully test the ConfigParser class.
 
It's easy to see that file-like objects are very handy and fits well in frameworks and APIs. There are of course many advantages of using an abstraction when reading/writing data, some of them are:
  • Implementations do not depend on real files, socket, etc, only on a file-like object.
  • Easier to fake data in unittests.
  • Caller responsible of opening/closing the resource.
  • Less error-handling in framework code, delegate exceptions to the caller.
The downside with the current file-like object abstraction is that it's tailored for text I/O. In Python 2.6 and 3.x a new module have been added named io. The module originates from the New I/O PEP. The module enriches Python with an three-layer solution, the raw I/O layer, the buffered I/O layer and finally the text I/O layer. The raw and buffered layers deals with units of bytes and the text layer deals with units of characters. There are some things you should be aware of if you use the new I/O library in Python 2.6, read more about it here.

I've implemented a shoutcast playlist parser, plsparser, using the ConfigParser class. The parser is implemented as a Python generator function. Actually, I might do a post about generator functions someday.

Thursday, October 1, 2009

Hello World

What would be more appropriate than a simple Hello World as the first code snippet!
def main():
    print "Hello World"

if __name__ == "__main__":
    main()