FUSE - Filesystem in Userspace part 3 (final)

Finally, as I promised, the last blog post on implementing file systems using FUSE.

I've created a file system, shoutcastfs, which enables you to mount the Shoutcast Radio directory as a file system. The genres are represented as directories and stations as files. Each file contains the station's playlist and the files are suffixed with .pls which makes it possible to load the playlist in a media player such as Amarok by double-clicking the file.

Of course, I'm using pyshoutcast (Python shoutcast API) to access the shoutcast service.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import errno
import fuse
import stat
import os
import shoutcast

fuse.fuse_python_api = (0, 2)

_shoutcastApi = shoutcast.ShoutCast()

class RootInfo(fuse.Stat):
    def __init__(self):
        fuse.Stat.__init__(self)
        self.st_mode = stat.S_IFDIR | 0755
        self.st_nlink = 2
        self._genres = {}

    @property
    def genres(self):
        if not self._genres:
            for g in _shoutcastApi.genres():
                self._genres[g] = GenreInfo(g)
        return self._genres

class GenreInfo(fuse.Stat):
    def __init__(self, name):
        fuse.Stat.__init__(self)
        self.st_mode = stat.S_IFDIR | 0755
        self.st_nlink = 2
        self.name = name
        self._stations = {}

    @property
    def stations(self):
        if not self._stations:
            for s in _shoutcastApi.stations(self.name):
                name = '{0}.pls'.format(s[0])
                name = name.replace('/', '|')
                self._stations[name] = StationInfo(name, s[1])
        return self._stations

class StationInfo(fuse.Stat):
    def __init__(self, name, station_id):
        fuse.Stat.__init__(self)
        self.st_mode = stat.S_IFREG | 0644
        self.st_nlink = 1
        # Hope no playlist exceeds this size
        self.st_size = 4096
        self.name = name
        self.station_id = station_id
        self._content = None

    @property
    def content(self):
        if self._content is None:
            self._content = _shoutcastApi.tune_in(self.station_id).read()
        return self._content

class ShoutcastFS(fuse.Fuse):
    def __init__(self, *args, **kw):
        fuse.Fuse.__init__(self, *args, **kw)
        self.root = RootInfo()

    def split_path(self, path):
        """ Returns genre and station """
        if path == '/':
            return (None, None)

        parts = path.split('/')[1:]
        if len(parts) == 1:
            return (parts[0], None)
        else:
            return parts

    def getattr(self, path):
        genre, station = self.split_path(path)

        if genre is None:
            stat = self.root
        else:
            stat = self.root.genres.get(genre)
            if not stat:
                return -errno.ENOENT

            if station:
                stat = stat.stations.get(station)
                if not stat:
                    return -errno.ENOENT
        return stat

    def readdir(self, path, offset):
        yield fuse.Direntry('.')
        yield fuse.Direntry('..')

        if path == '/':
            entries = self.root.genres.keys()
        else:
            entries = self.root.genres[path[1:]].stations.keys()

        for e in entries:
            yield fuse.Direntry(e)

    def open(self, path, flags):
        # Only support for 'READ ONLY' flag
        access_flags = os.O_RDONLY | os.O_WRONLY | os.O_RDWR
        if flags & access_flags != os.O_RDONLY:
            return -errno.EACCES
        else:
            return 0

    def read(self, path, size, offset):
        genre, station = self.split_path(path)
        info = self.root.genres[genre].stations[station]
        if offset < info.st_size:
            if offset + size > info.st_size:
                size = info.st_size - offset
            return info.content[offset:offset+size]
        else:
            return ''

if __name__ == '__main__':
    fs = ShoutcastFS()
    fs.parse(errex=1)
    fs.multithreaded = False
    fs.main()

To try the file system, run:

$ # Download shoutcast.py
$ wget http://github.com/mariob/pyshoutcast/raw/master/src/shoutcast.py
$ mkdir mnt
$ ./shoutcastfs mnt
$ cd mnt/
$ ls
...A list of genres...
$ cd Samba
$ ls
...A list of 'Samba' stations...
$ cat [station name]
...Playlist data...

Have fun!

Fedora 12

For the moment I'm evaluating Fedora 12. I was a bit disappointed on the Kubuntu 9.10 release, the distribution felt a little bit slow and tiered. The only thing I'm satisfied with is the boot time, it starts quite fast.

The KDE4 desktop in Fedora feels kind of snappier than KDE in Kubuntu. I'm not sure if it's because Fedora is compiled for i686 and not i386 or if the team have done any optimizations. I didn't enabled desktop effects in neither distribution. I installed the nvidia proprietary driver in Kubuntu but use Nouveau (open source driver) in Fedora. I might try the proprietary driver in Fedora to see if I gain any speed.

There's one bad thing I found in Fedora in comparision with Kubuntu, KPackageKit. It takes ages before starting compared to the Kubuntu version so I'm using yum from the shell prompt instead. Well, to be honest, I'm not a heavy user of UI front-ends to package management systems. In Kubuntu I prefer using aptitude instead of KPackageKit.

I think I'll give Fedora a chance and try it out for a while.

FUSE - Filesystem in Userspace part 2

This is the second post on implementing file systems using FUSE. In this post I'll show you how to create a read-only file system that is backed by an XML file. Each tag in the XML file is either a file or a directory. If the tag is representing a directory, it should have a is_dir="1" attribute added. When the tag represents a file it should have its content between the tags. The following XML snippet shows an example:

<root is_dir="1">

  <a is_dir="1">
    <file1.txt>File content for this file</file1.txt>
    <hello>Hello World</hello>
  </a>

  <empty_dir is_dir="1" />

</root>

The example models the following directory tree:

/
|-- a
|   |-- file1.txt
|   `-- hello
`-- empty_dir

There are some limitations in having an XML file representing a file system. You can't for example have filenames starting with a dot because tags are not allowed to start with a dot in XML.

On to the code, the following snippet implements the read-only file system which is backed by an XML file:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import errno
import fuse
import stat
import os
import time
import xml.etree.ElementTree as etree

fuse.fuse_python_api = (0, 2)

# Use same timestamp for all files
_file_timestamp = int(time.time())

class MyStat(fuse.Stat):
    """
    Convenient class for Stat objects.
    Set up the stat object with appropriate
    values depending on constructor args.
    """
    def __init__(self, is_dir, size):
        fuse.Stat.__init__(self)
        if is_dir:
            self.st_mode = stat.S_IFDIR | 0555
            self.st_nlink = 2
        else:
            self.st_mode = stat.S_IFREG | 0444
            self.st_nlink = 1
            self.st_size = size
        self.st_atime = _file_timestamp
        self.st_mtime = _file_timestamp
        self.st_ctime = _file_timestamp

class MyFS(fuse.Fuse):
    def __init__(self, xml_tree, *args, **kw):
        fuse.Fuse.__init__(self, *args, **kw)
        self.tree = xml_tree

    def getattr(self, path):
        # We do not support 'dot' files
        # since xml tags cannot start with a dot.
        if path.find('/.') != -1:
            return -errno.ENOENT

        entry = self.tree.find(path)

        if entry is None:
            return -errno.ENOENT
        else:
            is_dir = entry.get('is_dir', False)
            size = entry.text and len(entry.text.strip()) or 0
            return MyStat(is_dir, size)

    def readdir(self, path, offset):
        yield fuse.Direntry('.')
        yield fuse.Direntry('..')
        for e in self.tree.find(path).getchildren():
            yield fuse.Direntry(e.tag)

    def open(self, path, flags):
        # Only support for 'READ ONLY' flag
        access_flags = os.O_RDONLY | os.O_WRONLY | os.O_RDWR
        if flags & access_flags != os.O_RDONLY:
            return -errno.EACCES
        else:
            return 0

    def read(self, path, size, offset):
        entry = self.tree.find(path)
        content = entry.text and entry.text.strip() or ''
        file_size = len(content)
        if offset < file_size:
            if offset + size > file_size:
                size = file_size - offset
            return content[offset:offset+size]
        else:
            return ''

if __name__ == '__main__':
    tree = etree.parse('tree.xml')

    fs = MyFS(tree)
    fs.parse(errex=1)
    fs.main()

I'm using the ElementTree XML API to parse the XML file. The ElementTree API is very easy to use and supports finding elements by specifying a path which fits very well in this context. For example, the following snippet shows how to get the file1.txt element from the example XML file above and extract the content between the tags:

import xml.etree.ElementTree as etree
tree = etree.parse('tree.xml')
el = tree.find('/a/file1.txt')
print el.text.strip() # Remove any white-spaces

The code hasn't change that much since the part 1 post. I've added two new methods which adds support for opening and reading files. The MyStat class is only used as an convenient class to help creating appropriate stat objects. You might notice that I don't do a lot of checking in the code, this is because FUSE do a lot of them for me, this page list some of the assumptions you can make when implementing file system using FUSE.

Did you believe it would be this easy to create a mountable file system that uses an XML file for the layout? I didn't.

In the next (and final) post about FUSE I think I'll create some kind of 'use a service on the Internet' file system which can be useful and not just another toy fs.