Planet Ergo

I recently, very recently, purchased a Lenovo X220, fully beefed up with an i7 processor, 8GB of RAM, and the largest SSD (Intel 160GB), amongst other things.

Installing Fedora 16 (of course) went smooth (as was to be expected, as I know other people with Lenovo X220's). It's nicely locked down now, with startup, BIOS, GRUB passphrases, limited boot devices (SSD only) and an encrypted VG -which I can afford doing now without slowing down the entire system too much.

As a Kolab Systems employee, running multiple Kolab servers, naturally I install Kontact, the Kolab client, and I tend to do this using a fresh install (i.e. no copying data from the old laptop, no upgrading).

Fedora 16 includes a 4.7.4 KDE PIM stack, which turns out to not work for me. Having configured the Kolab accounts, it seems I cannot get to the messages in my Kolab INBOX -other folders work just fine.

In any case, I decided to try rawhide; the version of the KDE PIM stack included in rawhide at this moment is 4.7.95, KDE's latest release en route to 4.8 - this too, however, did not work for me.

So, I decided to try and build from GIT - my first time ever. KDE has a utility for this, called kdesrc-build. It's use is pretty straight-forward, but I had to install some build requirements on my system. This is what I have installed now:

# yum -y install \
    alsa-lib-devel attica-devel avahi-devel boost-devel bzip2-devel check-devel cups-devel \
    cyrus-sasl-devel dbus-devel dbusmenu-qt-devel enchant-devel fontconfig-devel \
    freetype-devel gamin-devel gettext-common-devel gettext-devel giflib-devel \
    glib2-devel glibc-devel glib-devel gnutls-devel gpgme-devel grantlee-devel \
    gstreamer-devel gstreamer-plugins-base-devel herqq-devel ilmbase-devel jasper-devel \
    kdebase-workspace-devel kdelibs-devel kdepimlibs-devel keyutils-libs-devel krb5-devel \
    libacl-devel libattr-devel libcom_err-devel libdrm-devel libgcrypt-devel \
    libgpg-error-devel libical-devel libICE-devel libjpeg-turbo-devel libpng-devel \
    libselinux-devel libsepol-devel libSM-devel libstdc++-devel libtasn1-devel \
    libudev-devel libutempter-devel libX11-devel libXau-devel libxcb-devel \
    libXcomposite-devel libXcursor-devel libXdamage-devel libXext-devel libXfixes-devel \
    libXft-devel libXi-devel libXinerama-devel libxkbfile-devel libxml2-devel libXpm-devel \
    libXrandr-devel libXrender-devel libXScrnSaver-devel libxslt-devel libXt-devel \
    libXtst-devel libXv-devel libXxf86misc-devel libXxf86vm-devel mesa-libGL-devel \
    mesa-libGLU-devel mysql-devel OpenEXR-devel openldap-devel openssl-devel \
    pcre-devel phonon-devel polkit-devel polkit-qt-devel PyKDE4-devel PyQt4-devel \
    python-devel qca2-devel qt-devel qt-gstreamer-devel qtwebkit-devel raptor2-devel \
    shared-desktop-ontologies-devel sip-devel soprano-devel sqlite-devel strigi-devel \
    xorg-x11-proto-devel xz-devel zlib-devel

Consider installing the "Fedora Packager" group as well;

# yum -y install @fedora-packager

After following the setup instructions, you should first initialize your copy of the various sources (otherwise failures would cause you to need to manually cleanup subversion repositories, for example):

$ kdesrc-build --src-only

This, when you run it for the first time, can take quite a while.

Once it's done, you can start building stuff:

$ kdesrc-build

This, too, can take quite a while. Furthermore, it requires a lot of energy, and drains my 7.5 hour battery life in about 90 minutes ;-)

UPDATE^1: The build dependencies for gwenview need to be added to the list of build requirements; exiv2-devel.

What is referred to as 'condstore' is actually the feature-set that IMAP4Rev1 extension added in RFC 4551. It enables the quick synchronization of flags on messages as well as other conditional STORE operations.

I'm going to have to refer you to the RFC for the full details, but suffice it to say your Kolab 2.3 deployment could greatly benefit from enabling said extension on your mailboxes - it is not enabled by default as part of Cyrus IMAP 2.3, only with Cyrus IMAP 2.4 is the 'condstore' annotation set to 'true' by default.

You will have to execute two separate actions on your Kolab server:

1. Configure new mailboxes to have 'condstore' enabled by default,
2. Configure current mailboxes with 'condstore' enabled.

You would preferrably execute these actions in this order, to reduce the chance new mailboxes are being created while you have not yet switched on 'condstore' to be enabled by default.

Enable 'condstore' by default

To enable condstore by default, in your favorite editor, add the following line to /kolab/etc/kolab/templates/imapd.conf.template:

mailbox_default_options: 2

Afterwards, write out new templates and reload Kolab with the new configuration;

/kolab/sbin/kolabconf

Enabling 'condstore' on existing mailboxes

To enable 'condstore' on existing mailboxes, login to Cyrus IMAP using the cyradm utility:

cyradm -t "" -u manager localhost

From there, you can examine one of the existing mailboxes;

localhost> info user/john.doe@example.org
{user/john.doe@example.org}:
  condstore: false
  duplicatedeliver: false
  lastpop:  
  lastupdate: 12-Sep-2011 17:56:54 +0200
  partition: default
  pop3newuidl: true
  sharedseen: false
  size: 105382261
  folder-test: true

As you can see in the example, condstore has not yet been enabled for this mailbox. To enable condstore for a particular mailbox:

localhost> mboxcfg user/john.doe@example.org condstore true

You will not want to repeat this command for all mailboxes, but instead use wildcard matching:

localhost> mboxcfg user/*@example.org condstore true

NOTE: The two commands do not return any output.

NOTE^2: Don't forget to, at your option, also enable condstore for mailboxes in the shared namespace, in other authentication realms, and perhaps even for those mailboxes that reside in the DELETED namespace (just in case they are restored to a visible namespace later on in life).

First of all, Bron Gondwana has done a great job over the course of a number of releases, to resolve some of the bugs in Cyrus IMAP that may have been around for quite a while -though most of them you may not have noticed.

Since Cyrus IMAP has been around since 1993 (give or take), it inherently contains legacy implementations of features or required functionality, sometimes to the tune of "good enough". Seeking further compliance with IETF approved RFCs and allowing for further development has spawned a 2.4 software series, with large amounts of code refactoring and cleanup, and the addition of some interesting features -more on those later.

Just this week, the Cyrus IMAP team has released cyrus-imapd-2.4.10.tar.gz. One day later, I've submitted the build to Fedora rawhide, so it should be hitting your doorstep soon enough. I plan to follow up on releases upstream with packaging more quickly and more agile in the future. Yours truly being the Release Engineer for Cyrus IMAP & SASL, as part of my work for Kolab Systems, I like how we have established a culture of release early, release often within the Cyrus IMAP project, and how core developers like Bron can make releases happen almost autonomously, while in between the last version in the 2.3 product series and the first version of the 2.4 series, a good couple of years had past.

Further endeavors include our 2.5 roadmap (which I admit could use some love), documentation (which I work on) and of course testers and developers are always welcome!

In the Cyrus SASL realm, we're working to convert the CVS repository to GIT. This takes a long time, since like Cyrus IMAP, Cyrus SASL has been around for quite a while. I must say we have done our due dilligance over the past few months on this one though, so I expect the repository conversions to GIT I've been doing to be approved soon enough.

Using GIT for the Cyrus IMAP codebase has proven to be a lot more attractive to a lot more contributors (since they can now create their own working copies and collaborate on those), and I hope we achieve the same level of success for Cyrus SASL.

Applications that we integrate with Kolab Groupware have a genuine need for caching. But what is it exactly that causes this need for caching?

Why Does One Cache?

Let's first think about why one caches in the first place. Caching is usually implemented to eliminate a bottleneck and boost performance. Data can then be obtained from the relatively quick cache -it is "close by", and it usually understands and is optimized for some form of querying- as opposed from the relatively slow original source of the data.

With Kolab, using Cyrus IMAP as the backend storage for all groupware related information like Email, Events and Contacts, it can hardly be argued the IMAP server does not perform up to specifications. Cyrus IMAP is extremely fast and scalable; it is, arguably, just as lean and mean as a cache would be.

So, Where's the Bottleneck?

Yet, the following topology does introduce and work around a bottleneck, seriously impacting and later improving performance. I'm off to explore what exactly is the bottleneck, and how to work around it.

We'll use Z-Push (Free Software ActiveSync implementation) in a regular, current scenario workflow as an example. The following happens, justifying caching, when a mobile device requests synchronization;

1. The 3rd party application connects to Cyrus IMAP to retrieve "the information". Since not all folders may need syncing, and some folders contain Email while others contain Events or Contacts, a list of IMAP folders needs to be obtained.
2. Cyrus IMAP, its efficiency in this matter aside, interacts with the mailbox storage to retrieve certain information from its mailbox database, its annotations database, the message files, etc.

Using the IMAP protocol, this means;

• Listing the folders the user is authorized for.
• Iterate over the list of folders and retrieve the following annotations for each folder:
  • "Should this folder be synchronized with the mobile device at all?"
  • "What type of groupware items does the folder contain?"
• Then, excuse my paraphrasing, on a per IMAP folder target basis, the changes that may or may not have been applied on the mobile device, need to be compared with the changes that may or may not have been applied in the IMAP folder, and vice-versa. Retrieving changes, messages, parsing them, and comparing them, and applying the changes on either end, while tracking which changes have already been communicated to one or the other end of the synchronization exercise.

Naturally, the last step is what Z-Push is in charge of, in this example. and It does have certain characteristics and interactions with Cyrus IMAP as well as with its own caches to optimize the performance, scalability and user experience.

The former notwithstanding, this is just one application integrated with Kolab responsible of maintaining its own cache. In current generations of Kolab, Horde Webmail does the exact same but using a different cache, and future generations of Kolab will include RoundCube, which again also maintains its own cache.

One Cache per Application?

Maintaining a cache per 3rd party application integrated with Kolab isn't necessarily the most sustainable route to go. Feasible? Yes. Sustainable? Perhaps not. Let's take one step back and look at the bigger picture again;

Presumably, the interaction between Cyrus IMAP and its storage can not be optimized further (which the dotted double arrow is supposed to indicate). Not without intrusive changes at the very least, that is to say, while admittedly I'm unaware of our options to further increase performance in this part of the flow of information. If you have ideas and the necessary experience, let me know and I can get you hooked up.

It is, perhaps, the IMAP protocol used in between the Cyrus IMAP server and the 3rd party application that is the bottleneck. For example, Z-Push cannot do the following over IMAP, eliminating a number of iterations and sequences issuing IMAP commands;

SELECT folder FROM folders
    INNER JOIN annotations ON folders.id = annotations.folder_id
    WHERE annotations.key = '/vendor/kolab/activesync' AND annotations.value = 'true';

Hey, this does somewhat represent what it does against the cache it maintains, having obtained the information over IMAP once (slow) it uses its cache to obtain the information (fast) in a number of subsequent synchronizations -limited to an expiry interval, expiring and updating cache, of course. This builds us the following picture, where IMAP is "slow" for the task at hand, and SQL is fast;

Ignoring the interaction that Cyrus IMAP requires with the filesystem as being a negligible performance penalty, and focussing on how the 3rd party application wishes to optimize performance, apparently it would rather perform caching (cheap), then it would want to interact over the IMAP protocol (expensive).

Suggestion #1: One Cache To Rule Them ALL

It has been suggested, since most if not all of the 3rd party applications integrated with Kolab would require some form of caching, we create "one cache to rule them all";

Although probably this is in fact entirely feasible, the following constraints to such architecture come to mind;

• Session reliability and personal information security, complex to implement but even more complex to audit, and implemented up to specifications with IMAP ACLs already,
• Duplication of ((a significant) part of) the data,
• Abstraction from caching required in all 3rd party applications, each of which has their own already (i.e. significant development effort right from the start, and continuous development effort for more 3rd party applications to integrate with Kolab Groupware), and one uniform caching specification across all of the 3rd party applications (i.e. significant design complexity).

We (within the Kolab community) regularly refer to the "one cache to rule them all" as "server-side akonadi" - currently the very efficient client-side (offline) caching in our primary smart client, Kontact.

Suggestion #2: Maintain Cyrus IMAP Databases in Networked SQL

It has also been suggested (by me, in fact), to have Cyrus IMAP use database formats other applications within a Kolab Groupware deployment could read from. By having Cyrus IMAP maintain its mailbox, annotations and perhaps even mail folder indexes and caches in a database format like SQL (instead of Berkely or skiplist), these would become available to the 3rd party applications without them having to populate the cache first, and the cache would be updated "automagically"; as a result, the level of interactions with Cyrus IMAP over the "inefficient" IMAP protocol would be further reduced -"inefficient" for the task at hand, that is.

However, this would greatly impact the scalability of Cyrus IMAP. It would, in fact, greatly impact the overall performance of Cyrus IMAP as an IMAP server. It's a valid option, but not considered feasible targetting for because of the projected performance penalties.

Suggestion #3: Use Cyrus IMAP

If you agree it's fair to label having to use the IMAP protocol to get to the data required as the bottleneck, here's the suggestion I have in mind; Add a thin, lean and mean, network-enabled, read-only C application to interface between the 3rd party applications and the Cyrus IMAP databases (on the filesystem), thus enabling the 3rd party applications to use a different protocol or querying language to obtain the data in a more efficient manner. Perhaps this would look as follows:

Benefits would include many requirements have already been implemented; Locking, networking, database maintenance, threading, thread safety, TLS/SSL, access control though IMAP ACLs and its handling and more of that stuff. The new application could, presumably, also maintain its own caching capabilities to be even quicker.

Just some early Saturday morning thoughts... let's see what the rest of the weekend brings.

Turns out the LDAP module for Python breaks the API while developing version 2.4 -with no backwards compability, but a workaround is relatively easy. If, like me, you need to be able to run code on platforms traditionally "slow" in adopting the latest and greatest, as well as those that are traditionally, relatively "fast", here's an idea:

# Catch python-ldap-2.4 changes
from distutils import version
if version.StrictVersion('2.4.0') <= version.StrictVersion(ldap.__version__):
    LDAP_CONTROL_PAGED_RESULTS = ldap.CONTROL_PAGEDRESULTS
else:
    LDAP_CONTROL_PAGED_RESULTS = ldap.LDAP_CONTROL_PAGE_OID

class SimplePagedResultsControl(ldap.controls.SimplePagedResultsControl):
    """

        Python LDAP 2.4 and later breaks the API. This is an abstraction class
        so that we can handle either.
    """

    def __init__(self, page_size=0, cookie=''):
        if version.StrictVersion('2.4.0') <= version.StrictVersion(ldap.__version__):
            ldap.controls.SimplePagedResultsControl.__init__(
                    self,
                    size=page_size,
                    cookie=cookie
                )
        else:
            ldap.controls.SimplePagedResultsControl.__init__(
                    self,
                    LDAP_CONTROL_PAGED_RESULTS,
                    critical,
                    (page_size, '')
                )

    def cookie(self):
        if version.StrictVersion('2.4.0') <= version.StrictVersion(ldap.__version__):
            return self.cookie
        else:
            return self.controlValue[1]

    def size(self):
        if version.StrictVersion('2.4.0') <= version.StrictVersion(ldap.__version__):
            return self.size
        else:
            return self.controlValue[0]

Example Usage

Where 'self.ldap' is the LDAP object;

    def _search(self,
            base_dn,
            scope=ldap.SCOPE_SUBTREE,
            filterstr="(objectClass=*)",
            attrlist=None,
            attrsonly=0,
            timeout=-1
        ):

        _results = []

        page_size = 500
        critical = True

        server_page_control = SimplePagedResultsControl(page_size=page_size)

        _search = self.ldap.search_ext(
                base_dn,
                scope=scope,
                filterstr=filterstr,
                attrlist=attrlist,
                attrsonly=attrsonly,
                serverctrls=[server_page_control]
            )

        pages = 0
        while True:
            pages += 1
            try:
                (
                        _result_type,
                        _result_data,
                        _result_msgid,
                        _result_controls
                    ) = self.ldap.result3(_search)

            except ldap.NO_SUCH_OBJECT, e:
                log.warning(_("Object %s searched no longer exists") %(base_dn))
                break
            _results.extend(_result_data)
            if (pages % 2) == 0:
                log.debug(_("%d results...") %(len(_results)))

            pctrls = [
                    c for c in _result_controls
                        if c.controlType == LDAP_CONTROL_PAGED_RESULTS
                ]

            if pctrls:
                size = pctrls[0].size()
                cookie = pctrls[0].cookie()
                if cookie:
                    server_page_control.cookie = cookie
                    _search = self.ldap.search_ext(
                            base_dn,
                            scope=scope,
                            filterstr=filterstr,
                            attrlist=attrlist,
                            attrsonly=attrsonly,
                            serverctrls=[server_page_control]
                        )
                else:
                    # TODO: Error out more verbose
                    break
            else:
                # TODO: Error out more verbose
                print "Warning:  Server ignores RFC 2696 control."
                break

        return _results

Or something like that ;-)

You are somewhere in a Sun Belt state in the U.S., and it's nearing the and of January. For some, this sounds like a hopeless situation. For others, it is the opportunity of a lifetime!

Here's 10 things to do in Tempe, Arizona, nearing the end of January, 2011, while FUDCon is taking place.

1. Bacula Backup and Recovery; Get some insight on the subject and engage a Bacula Systems Certified Trainer on the subject. I'd love to insert a "Tuesdays at noon" type of thing, but we're just not sure when exactly such sessions would be taking place. It's not important either, there's at least 9 other things to do!
2. Koji Build Systems. Do you use it? Do you use it because you need something happening downstream? Are you a third-party repository enthusiast? Let's get to it, and have at least one session on the build system suite.
3. Long Term Support (LTS), Extended Lifecycle Support (ELS), or whatever you may want to call an extention on the regularly supported time period of updates coming to a stable Fedora release. I like the latter terminology better. If you're interested in stopping it though, please stay away?
4. Spins. Custom spins, localized spins, remixes or bluntly put, dwarf forks. How can we make sure all this momentum is harnassed and nurtured up and until the point little projects become proud masters of the universe with our help.
5. Packaging Guidelines and the Package Review Process. I've had my share, how about you?
6. Cyrus IMAP; first-hand experience of a) How Fedora Rules the Universe, b) What Upstreams Expect From Distributors and c) How To NOT Behave As A Distributor.
7. The Ever So Awesome Fedora Cloud. I'm not sure where it's at, but I do think I know a little something about what cloud is all about. It's time to get that first concept implementation for Metadata DNS going, while showing off a complete Open Source stack for Any Company(TM) in a demo-environment with public access of sorts.
8. Fedora Hosted Groupware. I know Fedora Infrastructure to have looked at Zarafa, not to say I have shamelessly plugged it back in the day. In the interest of full disclosure, I now work for Kolab Systems -another Groupware ISV, this time fully open source and free software.
9. Puppet. Modules. Modules. Modules. Cross-platform consistency. Approaches. Packaging. Writing modules (why and how)
10. Beer, meat (that's me), friends, laughs (that's me too!). I can think of very little more I need in life.

When you run Kolab, or Postfix and Cyrus IMAP, you may have any number of your users subscribed to a mailing list.

As such, many mailing list posts may delivered to a large number of users separately, increasing the data footprint on disk, and processing, while only including some of your users in the loop on what may be interesting content for everyone.

You may want to have one shared folder to deliver new mailing list messages to, to which individual users can then subscribe. Here's how (in a nutshell):

• Create a Cyrus IMAP user that is called 'shared',
• Set the postuser to 'shared' in /etc/imapd.conf
• Set the sharedprefix to 'shared' in /etc/imapd.conf
• Create a shared/lists mailbox (cm shared/lists@kolabsys.com in my case),
• Create some subfolders you want mailing list posts be delivered to, like shared/lists/kolab.org/devel@kolabsys.com,
• Make sure that at the very least 'anyone' as the 'lrsip' permissions on the mailbox,
• Give yourself 'all' privileges on the mailbox,
• Make sure the recipient_delimiter is set to '+' in /etc/postfix/main.cf,
• Make sure shared@kolabsys.com is listed in /etc/postfix/transport with transport lmtp:unix:/var/lib/imap/socket/lmtp
• Reload what you need to reload,
• Send a test message to shared+shared/lists/kolab.org/devel@kolabsys.com,
• It should end up in the correct folder.

Note, for those of you not using virtual domain support in Cyrus, skip the @kolabsys.com at the end of folder names to be created and you should be good to go.

Also note that for each individual user that wants to post messages to the mailing list, they still need to be subscribed to the mailing list themselves; they can then disable delivery through the mailing list preferences pages if available, reading the list through the shared folder.

It's been a very busy past two weeks and I haven't had much time to show my face on IRC or respond to email. I've participated in or executed some interesting activities though, so please allow me to share those with you.

On Monday, almost two weeks ago, I visited Bacula Systems headquarters in Switzerland, to get a 1-on-1 training from Kern Sibbald -the original author of Bacula backup & recovery software-, in order to become a certified Bacula Systems Trainer. A quick walk-through the slides along with some insight on what Kern thought were the most important highlights to mention and/or explain during the training, a review of the exercises and one day later, you have sufficient additional information apart from the course material itself.

The next day, Eric Schirardin introduced me to how the Bacula Systems course classroom setup was to be configured, so that all the exercises could be performed without the course attendees having to go through the motions that in fact would have had little to do with Bacula itself.

Since my flight back to the Netherlands was leaving early that evening, and I had an important Kolab Systems conference call in between, I had to hurry and catch a train to the airport, find myself a nice, quiet corner somewhere, a wifi connection and call in. I made it, but that morning I had no idea how smoothly we would go through the motions to get me to be able to set up the classroom and at what time we'd be done exactly. Let's just say I had too little time to get any lunch before I caught a train just in time for me to arrive at the airport and not have any of the ambient train noises while sitting in a conference call. Then again, I don't usually eat during lunch anyways ;-)

The next day (Thursday last week), I was going to travel to Osnabrueck for a visit to a valued partner on behalf of Kolab Systems; a three-hour train-ride from Utrecht. In order to arrive at a reasonable time, I had to get up at around 6 in the morning, not something I'm very much used to, nor a fan of ;-) Either way though, the visit was planned for Thursday as well as Friday, and it was definitely worth the travel! I was staying with Christoph Wickert -you know him from his contributions to the Fedora Project and various upstream projects- in Munster throughout my stay in Germany.

On Saturday, I was on my way back home again. Since the Bacula Systems Administration Course was to start on Tuesday, and Monday was allocated to preparing the classroom (and testing the setup), most of my weekend was spent at going through the slides and exercises and preparing my story. Luckily, I was still very familiar with the course as I had recently participated in the course myself, so I could afford to visit a friend's birthday party as well.

On Monday, it turned out that VirtualBox (used to create a second backup client for the Bacula Systems Administration Course), installed on a 64-bit laptop without hardware virtualization acceleration capabilities cannot run a 64-bit guest, so I quickly installed a 32-bit guest and configured it, which actually took the majority of the single day that I had available.

Tuesday, Wednesday and Thursday, at Amaziq Source in Amsterdam, I ran the Bacula Systems Administration I Course (there's a part II coming up as well, with more advanced topics) with 6 participants from 4 countries, and I'd like to think that it was very successful. At least Bacula Systems (through their senior engineer Arno Lehmann) thought so, because I passed the test running the course according to their quality standards, and so now I'm a proud Bacula Systems Certified Trainer ;-)