Wednesday, October 28, 2009

DIY: Google Mail Notifier

It's quite easy to create your own GMail notifier. Your inbox, gmail or domain,  can be monitored by fetching and parsing an atom feed.

Following code snippet prints new (unread) mail in your inbox:
# -*- coding: utf-8 -*-
import feedparser
import urllib2

# Create http basic auth handler
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password('New mail feed', 'https://mail.google.com/',
                          'USERNAME@DOMAIN', 'PASSWORD')

# Open url using the auth handler
opener = urllib2.build_opener(auth_handler)
feed_file = opener.open('https://mail.google.com/mail/feed/atom/')

# Parse feed using feedparser
d = feedparser.parse(feed_file)

# Print mail count and mails
print 'Mail count:', d.feed.fullcount

for entry in d.entries:
        print '----------------------------------------------'
 print 'Author: ', entry.author
 print 'Subject:', entry.title
 print 'Summary:', entry.summary

Just change USERNAME and PASSWORD. You also need to download feedparser which is free to use, or write one yourself.

Have fun!

ELCE 2009 presentations

The presentations from ELCE 2009 are now available from here. Some of the presentations have not been posted yet.

Monday, October 26, 2009

pyshoutcast v0.7.1

pyshoutcast version 0.7.1 has just been released. It implements a Python shoutcast API which makes it possible to list genres, stations, do search, etc, using the shoutcast yellow pages service.

Unfortunately, I just found a forum post stating that Amarok and VLC have removed support for shoutcast due to licensing issues. I'm not sure why because I can't find any info about it, I'll try to find out more details.

Anyway, to try it out you can start a interactive python prompt and paste the following code:
>>> import shoutcast
>>> api = shoutcast.ShoutCast()
>>> api.random()

Wednesday, October 21, 2009

'with' statement

Python have a great number of cool language features and one of them is the with statement. The with statement can be used in numerous ways but I think that the most common usage is to track and release some kind of resource and at the same time encapsulate the code block with a try/finally clause.

For example:
with open('my_file.txt') as f:
    for line in f:
        print line
which could be translated to something like this:
f = open('my_file.txt')
try:
    for line in f:
        print line
finally:
    f.close()
and in a general form:
with <EXPRESSION> [as <VAR>]:
    <BLOCK>
The with statement guarantees that the file will be closed no matter what happens, for example an exception is thrown or a return is executed inside the for statement etc. The result from the expression that is evaluated after the with keyword will be bound to the variable specified after the as keyword. The as part is optional and I'll show you a use case later.

But wait a minute, how does the with keyword know how to close the file? Well, the magic is called 'Context Manager'.

A context manager is a class which defines two methods, __enter__() and __exit__(). The expression in the with statement must evaluate to a context manager else you'll get an AttributeError. The __enter__ method is called when the with statement is evaluated and should return the value which should be assigned to the variable specified after the as keyword. After the block has finished executing the __exit__ method is called. The __exit__ method will be passed any exception occurred while executing the block and can decide if the exception should be muted by returning True or be propagated to the caller by returning False.

Let's look at the following example (not a good design, just an example):
class TransactionContextManager(object):
    def __init__(self, obj):
        self.obj = obj

    def __enter__(self):
        self.obj.begin()
        return self.obj

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type:
            self.obj.rollback()
        else:
            self.obj.commit()
        return False

class Transaction(object):
    def begin(self):
        print "Start transaction"

    def query(self, sql):
        print "SQL:", sql

    def rollback(self):
        print "Rollback transaction"

    def commit(self):
        print "Commit transaction"

print "Before 'with'"

with TransactionContextManager(Transaction()) as tx:
    tx.query("SELECT * FROM users")

print "After 'with'"
The code produces the following output:
Before 'with'
Start transaction
SQL: SELECT * FROM users
Commit transaction
After 'with'
It's easy to trace the execution but I'll explain it anyway.
  1. The code starts by printing "Before 'with'".
  2. The with statement is executed and instances of TransactionContextManager and Transaction are created. The context manager's __enter__ method is invoked which will invoke the Transaction instance's begin method. The begin method prints "Start transaction" and finally the __enter__ method returns the Transaction instance.
  3. The return value from the context manager's __enter__ method is bound to the variable tx.
  4. The block is executed and the query method is called.
  5. The query method prints "SQL: SELECT * FROM users".
  6. The block is done executing and the context manager's __exit__ method is invoked. Since no exception occurred the commit method is invoked on the Transaction instance. The __exit__ method returns False which means that any exceptions should be propagated to the caller, none in this case.
  7. The code ends executing by printing "After 'with'".
Let's change the Transaction class so that it throws an exception from the query method. This should execute the rollback method instead of the commit method. The exception should also be propagated to the caller since we return False from the __exit__ method.
def query(self, sql):
    raise Exception("SQL Error occurred")
And the output looks like following:
Before 'with'
Start transaction
Rollback transaction
Traceback (most recent call last):
  File "transaction.py", line 32, in 
    tx.query("SELECT * FROM users")
  File "transaction.py", line 21, in query
    raise Exception("SQL Error occurred")
Exception: SQL Error occurred
Here we see that the rollback method is called instead of commit. The rollback method is called because the code inside the with block raised an exception and a valid exc_type argument was passed to the __exit__ method. We can also see that the exception is propagated to the caller because we return False from the __exit__ method.

If we change the __exit__ method to return True instead of False we get the following output:
Before 'with'
Start transaction
Rollback transaction
After 'with'
As expected, we don't see the exception anymore. Note that the __exit__ method is not called if an exception occurs before the with statement starts executing the block.

Alright, I hope you agree with me that the with statement is very useful. Now I'll show you a final example where we don't need to bind the return value from the context manager to a variable. I've re-factored the previous example and hopefully made a better design.
class Connection(object):
    def __init__(self):
        # Default behaviour is to do auto commits
        self._auto_commit = True

    def connect(self):
        pass

    def auto_commit(self):
        return self._auto_commit

    def set_auto_commit(self, mode):
        print "Auto commit:", mode
        self._auto_commit = mode

    def rollback(self):
        print "Rollback transaction"

    def commit(self):
        print "Commit transaction"

    def executeQuery(self, sql):
        # Handle auto commit here if it's enabled
        print "SQL:", sql

class Transaction(object):
    def __init__(self, conn):
        self.conn = conn
        self.auto_commit = False

    def __enter__(self):
        self.auto_commit = self.conn.auto_commit()
        self.conn.set_auto_commit(False)
        return self.conn

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type:
            self.conn.rollback()
        else:
            self.conn.commit()

        self.conn.set_auto_commit(self.auto_commit)
        return False

conn = Connection()
conn.connect()

with Transaction(conn):
    conn.executeQuery("SELECT a user FROM users")
    conn.executeQuery("INSERT some data INTO users")
The output:
Auto commit: False
SQL: SELECT a user FROM users
SQL: INSERT some data INTO users
Commit transaction
Auto commit: True
As the example shows, we don't need to use the return value from the context manager to still have use of them. We only want to be sure that the queries inside the with block is executed in the same transaction and rollbacked if an exception occurs or commited if successful. Without the with statement the code would be implemented something like the following snippet:
conn = Connection()
conn.connect()
auto_comm = conn.auto_commit()

try:
    conn.set_auto_commit(False)
    conn.executeQuery("SELECT a user FROM users")
    conn.executeQuery("INSERT some data INTO users")
    conn.commit()
except:
    conn.rollback()
    raise
finally:
    conn.set_auto_commit(auto_comm)

Well, for me the with statement seems more clean and convenient to use than the last example.

Sunday, October 18, 2009

Back home from ELCE 2009


It has been a great visit to Grenoble. I'm pleased with the conference because most of the sessions were held by developers, which I prefer, and not marketing people. The major focus seemed to be on build systems and reducing boot time. I learned a lot about tools which can give you hints on performance issues etc. Some celebrities also showed up like Jon Masters and Alessandro Rubini.

I assume that the slides from the sessions will be available on the home page so you can take a look at them yourself.

Adeneo had a cool demo of booting up a Linux system with a Qt-app doing 3D (software rendering) in less than 5 seconds. Unfortunately I can't give you a clip since the new blogger editor doesn't support movie clips, or am I wrong?. A picture will do.


The conference had only a little exhibiton hall (which was ok for me) and here's a picture of Free Electrons' booth.


The food was a success! A lot of different finger food and of course wine. Here's an example of what the dessert looked like the second day.


The restroom has a really cool designed washbasin.


As I mentioned before I'm pleased with the conference and I believe I'll attend again next year.

Tuesday, October 13, 2009

I'm going to ELCE 2009

I'll be visiting the Embedded LInux Conference Europe 2009 which is held in Grenoble, France. It's a two day conference, 15-16 October. I'm looking forward to attend the sessions about build systems and other cool low-level kernel stuff.

Hope I'll meet some interesting people.

Saturday, October 10, 2009

KDE Hello World

I have plans to do a web radio player (I wonder if we'll ever see this app). My intention is to learn Qt/KDE programming in Python. I've played around with Qt a bit and I'm starting to get comfy. I've decided to try doing the player as a KDE app, just to learn more about KDE. Qt is great for doing cross-platform stuff but I want to take a look at what KDE offers on top of Qt. You can find docs, tutorials and more on KDE TechBase. The following snippet shows the minimum amount of code needed to create a typical KDE app in Python.
from PyKDE4 import kdecore
from PyKDE4 import kdeui
import sys

def createAboutData():
    """
    Create a KAboutData with information about this application.
    """
    return kdecore.KAboutData(
            # Program name used internally
            "hello",
            
            # Catalog name
            "",                       
            
            # Displayable program name
            kdecore.ki18n("Hello World app"),
            
            # Program version
            "0.1.0",
            
            # Short description about the program
            kdecore.ki18n("A simple KDE Hello World app"),
            
            # Program license
            kdecore.KAboutData.License_BSD,
            
            # Copyright statement
            kdecore.ki18n ("(c) 2009 Mario Boikov"),
            
            # Free form text
            kdecore.ki18n("Free form text\nsupporting newlines"),
            
            # Home page address for this program
            "http://www.pysnippet.com",
            
            # Bug report email address
            "mario@beblue.org",
            )

def main():
    about = createAboutData()
    kdecore.KCmdLineArgs.init(sys.argv, about)

    app = kdeui.KApplication()

    # INSERT APP CODE HERE

    # Start event loop
    sys.exit(app.exec_())

if __name__ == '__main__':
    main()
Nothing special will happen if you execute this application, we need some kind of window/widget to display something. To accomplish this we can create a KMainWindow with a KPushButton. We'll also add code to print Hello World when the button is clicked. The following snippet will create a window with a push button:
from PyKDE4 import kdeui
from PyKDE4 import kdecore
from PyQt4 import QtCore

class MainWindow(kdeui.KMainWindow):
    def __init__(self):
        super(MainWindow, self).__init__()

        # Create widget with vertical box layout manager
        layout = kdeui.KVBox()

        # Create push button and add to layout
        button = kdeui.KPushButton(kdecore.i18n("Push Me!"), layout)

        # Connect to button's 'clicked' signal 
        QtCore.QObject.connect(button, QtCore.SIGNAL("clicked()"),
                               self.buttonClicked)

        self.setCentralWidget(layout)

    def buttonClicked(self):
        print "Hello World!"
You might wonder why I use ki18n() when creating the KAboutData and i18n() for the MainWindow. Both functions do translations, i18n() requires an instance of KApplication before use but ki18n() don't.

To actually show the window we have to instantiate it and make it visible, the "INSERT APP CODE HERE" comment should be replaced with the following lines:
    mainWindow = MainWindow()
    mainWindow.show()
As you can see, it doesn't take too much effort to create a simple app in KDE. There's almost more code to create the KAboutData than doing the window with a push button. Here's a screen shot of what the window looks like on my Kubuntu installation:



If we compare Qt and KDE there aren't too much differences, at least with this example. The code below implements the same Hello World app made with Qt:

import sys
from PyQt4 import QtCore, QtGui

class MainWindow(QtGui.QMainWindow):
    def __init__(self, title):
        super(MainWindow, self).__init__()
        self.setWindowTitle(title)

        centralWidget = QtGui.QWidget()

        button = QtGui.QPushButton("Push Me!")

        layout = QtGui.QVBoxLayout()
        layout.addWidget(button)

        centralWidget.setLayout(layout)

        QtCore.QObject.connect(button, QtCore.SIGNAL("clicked()"),
                               self.buttonClicked)

    self.setCentralWidget(centralWidget)

    def buttonClicked(self):
        print "Hello World!"

def main():
    app = QtGui.QApplication(sys.argv)

    mainWindow = MainWindow("Hello World App")
    mainWindow.show()

    sys.exit(app.exec_())

if __name__ == "__main__":
    main()

The biggest difference is how the layout is handled and the absence of About Data.

Wednesday, October 7, 2009

Python and unit testing

Writing unit tests in python is really easy. It's the dynamic properties of the language which makes it very simple to mock class methods, instance methods and functions. This makes the task of writing isolated unit tests easy when you practice test-driven development.

Say you have the following class:
import urllib2

class UrlDownloader(object):
    def download(self, url):
        """
        Read the content from the url.
        The content is returned as a string object.
        """
        return urllib2.urlopen(url).read()
You actually don't want to download something from the net when you do a unit test for this class. You would like to have a fixed string returned from the read() method. This can be hard to achieve in some languages (for example Java) but that's not true for Python. Python supports redefining class methods, functions and other things in run-time. The following snippet shows how easy it's to mock the urlopen() function in the urllib2 module to return a mocked file-like object:
import urllib2
import unittest

class MockFileObject(object):
    def read(self):
        return "DATA"

def mock_open(url):
    return MockFileObject()

class UrlDownloaderTest(unittest.TestCase):
    def testDownload(self):
        urllib2.urlopen = mock_open
        dl = UrlDownloader()
        self.assertEqual(dl.download(""), "DATA")

if __name__ == '__main__':
    unittest.main()
In this unit test we redefine the urlopen-function (remember, this is not a member of an instance, it's a plain function) with our own which returns a mock file-like object. It's not that hard to imagen all the possibilities this gives when writing tests. There are of course frameworks that will help you with creating mocks etc, you could try The Python Mock Module which should cover most of the use cases.

Friday, October 2, 2009

File-like objects

A Python file-like object can be seen as an abstraction to input/output streams. They are actually similar to Java's java.io.Reader/java.io.Writer abstraction (I'm coming from Java). The typical usage of file-like objects can be found in frameworks and APIs. The file-like object provides a generic way of reading/writing data and would fit an HTML-parser for example. The parser is not interested of where the data come from (files, sockets, etc), only the content, so naturally the implementation should not depend on how to open resources etc.

Basic methods that need to be implemented by a file-like object are:
  • read([size]) - read at most size bytes from the file.
  • readLine([size]) - read one entire line from the file.
  • write(str) - write a string to the file.
  • close() - close the file.
You should read the documentation of the function/method that uses the file-like object, often they specify which methods that are expected to be implemented by the file-like object. If the file-like object is only used for reading, no write-functions need to be supported. More information about file-like objects can be found here.

Ok, so why should you use file-like objects in your code?

Let's take a look at the ConfigParser class which can be used to read/write configuration data. You can expect to find methods for reading and writing configuration data using files but the class does also include support for file-like objects, which is great. For example it's very easy to write unittests for this class, thanks to the support of file-like objects,  without having to create real files with test data. To create a file-like object from a string we can use the StringIO module. The module have both read and write support.

Here's some code:
import unittest
import StringIO
import ConfigParser

class ConfigParserTest(unittest.TestCase):

    def testHasSection(self):
        """
        has_section() should return True if the section exists
        and False if is doesn't.
        """
        data = StringIO.StringIO("[My section]\nSetting=1\n")
        config = ConfigParser.ConfigParser()
        config.readfp(data)
        self.assertTrue(config.has_section("My section"))
        self.assertFalse(config.has_section("Not a section"))

    def testMissingHeader(self):
        """
        readfp() should raise MissingSectionHeaderError
        when parsing a file without section headers.
        """
        data = StringIO.StringIO("\nSetting=1\n")

        config = ConfigParser.ConfigParser()
        self.assertRaises(ConfigParser.MissingSectionHeaderError,
                          config.readfp, data)

if __name__ == "__main__":
    unittest.main()
And a simple write configuration test:
import unittest
import StringIO
import ConfigParser

class WriteConfigTest(unittest.TestCase):

    def testWriteConfig(self):
        """
        Test write a simple config
        """
        data = StringIO.StringIO()
        config = ConfigParser.ConfigParser()
        config.add_section("My Section")
        config.set("My Section", "my_option", "my_value")
        config.write(data)

        expected = "[My Section]\nmy_option = my_value\n\n"
        self.assertEquals(expected, data.getvalue())

if __name__ == "__main__":
    unittest.main() 
These unittests are of course only examples and do not fully test the ConfigParser class.
 
It's easy to see that file-like objects are very handy and fits well in frameworks and APIs. There are of course many advantages of using an abstraction when reading/writing data, some of them are:
  • Implementations do not depend on real files, socket, etc, only on a file-like object.
  • Easier to fake data in unittests.
  • Caller responsible of opening/closing the resource.
  • Less error-handling in framework code, delegate exceptions to the caller.
The downside with the current file-like object abstraction is that it's tailored for text I/O. In Python 2.6 and 3.x a new module have been added named io. The module originates from the New I/O PEP. The module enriches Python with an three-layer solution, the raw I/O layer, the buffered I/O layer and finally the text I/O layer. The raw and buffered layers deals with units of bytes and the text layer deals with units of characters. There are some things you should be aware of if you use the new I/O library in Python 2.6, read more about it here.

I've implemented a shoutcast playlist parser, plsparser, using the ConfigParser class. The parser is implemented as a Python generator function. Actually, I might do a post about generator functions someday.

Thursday, October 1, 2009

Hello World

What would be more appropriate than a simple Hello World as the first code snippet!
def main():
    print "Hello World"

if __name__ == "__main__":
    main()