Planet Ergo

Many of you have anxiously anticipated the return of shared folder management in Kolab 3 - a feature that is often used for email addresses with multiple readers, such as perhaps a sales@company.com or info@example.org.

A large part of this functionality - as often requested - has been the ability to delegate to individual users the use of the recipient address associated with these shared folders.

Your sales representatives would read incoming sales@company.com email through a shared folder (say, shared/sales@example.org), and have the ability to not respond as themselves (i.e. with their own user email address as the envelope sender), but "on behalf of" sales@company.com, using that very address as the envelope sender.

I'm pleased to announce this functionality will be a part of Kolab 3.1 - I sat down yesterday, in my own time, and enabled the Kolab Web Administration Panel client interface and API to handle shared folders. Needless to say I also made sure that PyKolab (or, should I say, its setup-kolab utility for bootstrapping your brand new Kolab server) is aware of the changes required to Cyrus IMAP and Postfix.

To show off this shared folder functionality, I've created a screencast using a vanilla Kolab 3.0 deployment, and some magic commands to pull in the latest and greatest PyKolab and Kolab WAP directly from their respective GIT repositories.

Furthermore, yesterday's exercise brought me to also show off the new "dynamic distribution group" functionality - now exposed in the Kolab WAP, and re-inserted "allow" and "deny" recipients and senders to groups. You would, for example, create a kolab-users@example.org distribution group that, rather than a static list of members, holds a search query for all "(objectclass=kolabinetorgperson)" LDAP objects, hence creating a dynamic distribution group. Useful for announcements, but limit who can send to the distribution group using the "kolabAllowSMTPSender" attribute.

Now that Kolab 3 is released, it's time to gather some feedback as well. Many of you have installed Kolab 3, and it's time for you to participate!

I'm going to start off with an easy one, but before I do so, please allow me to provide you with some rationale. First, please allow me to take a minute of your time to have you read chapter 3.1 of our Administrator Guide - you can stop once you hit section 3.1.1.

I personally think the recipient policy is a very powerful feature, and I therefore wanted to show this off with any given installation - so I've provided defaults that apply this policy in one of the many ways that it is possible.

Now, I understand that many of you have no purpose for this feature, and that many of you want it changed. We've received questions, and complaints (each of these is to be taken as a compliment, or in the case of Kolab Systems -whom I work for- a sales opportunity). We've provided you with documentation on the why, and the how (to change the behaviour), and the how (to completely disable this feature). We've received enhancement requests to allow this powerful feature to be applied to a user's 'uid' attribute value, too.

So now is the time to gather some of your feedback, and see if we can improve the defaults for the use-cases you have, and the way you think the recipient policy could be improved. I'm hereby inviting you to do so; send me a direct email at kanarip@kanarip.com, stating;

1. Whether you would like to see the defaults deployed on your system(s) to include the recipient policy,
2. Whether you think the documentation on changing and/or disabling the recipient policy is clear and visible enough (did you find what you were looking for when you Googled?),
3. Whether you have any suggestions that would improve the recipient policy for your needs and use of it.

Perhaps stating the obvious but nonetheless worth mentioning: Your sending such feedback to my personal address, and not my professional address. I'll treat it confidentially, and so rest assured that Kolab Systems is not going to take the email you send as a sales opportunity nor that you've opted in to a marketing scheme - I won't allow it, not without your explicit prior consent.

That said, please do know that you have Kolab Systems to thank for most -not to say all- of the work that has gone in to Kolab 3. If you wanted to leave a "thank you" note, I suppose contact@kolabsys.com is the place to do so! This will most likely sollicit a response though, mind you ;-)

Dear businesses that post us marketing material through email,

I read my mail in plaintext, like many other people do. Not in the least because I can set my favorite font type, size and color and have absolutely nothing interfere with that preference.

It's frustrating when you send me an email that tells me "my client cannot read this email". Surely, the clients for Kolab Groupware (which I use, and develop, and support, and architect) can read your email, but you neglected to provide a legible equivalent of what you wanted to tell me.

I'm sure you're very fond of your undoubtedly stunningly beautiful markup, after all you employ people to work on that for you, but ignoring the fact people might actually just want to read some content is bluntly ignorant, and offensive.

I wonder whether you care enough. All I want is to just read about what you wanted to tell me without oversized headers, too small fonts for paragraphs, social media logos, login links, and screaming background colors being thrown in my face.

Not only do you not care enough to send me a legible version of your message, you don't even care enough to correct the message that tells me what is displayed on my screen is not what you wanted to say. The facts of the matter are relatively straight-forward;

• My client might not be able to render the marked up form of the message from the HTML, but it probably is able to.
• I will likely have preferred plain text content over HTML content, if that is the part that is being displayed on my screen.
• While there is no fancy WYSIWYG editor the people you employ can demand training for, you could use any application that does not have a markup toolbar, and that will be your WYSIWYG editor for the plain text version of your message.

So, please allow me to help you resolve this problem; An email that you sent with HTML can contain two(!) parts, and you can mark those parts as an alternative equivalent of the other part.

For the part that is the HTML equivalent, feel free to knock yourselves out.

For the plain text equivalent, please provide some useful content. I recommend you consider marking a header and lists such as you would do marking up the source of a wiki page, but whatever you do don't invent something you think is nice yourselves. Perhaps consider wrapping the lines at some 72 characters. A link you may need to refer to in the text can be referred to like this

(...)
We have a new product[1].
(...)

[1] http://new.product

Those links do not (need to) include the social media links, and "my account", and opt-in, and opt-out, and subscription status, and contact, and mobile site, and the web version of the marketing collateral (really, some do include all of the above).

Kolab Groupware has a strong focus on security, and data integrity - not just your own mailbox but the flow of traffic between you and your peers as well.

Please allow me to take the opportunity to explain to you some of the background of what Kolab Groupware does, and why. In this blog post, I'm zooming in on our use of the submission port (587).

There's a standard 3 ports associated with receiving email messages:

1. Port 25/tcp, or SMTP, which a mail exchanger uses to receive email from the Internet, and from other systems in your network that need to use this mail exchanger as their smarthost (conditions apply). Most commonly, port 25 accepts messages from the Internet for local recipients, or mail traffic from trusted systems inside your network, that are destined for the outside world.
   Typically, no authentication is required, and the envelope sender of a message is not guaranteed to actually be the real human being that would receive your reply should you reply to the envelope sender address.
2. Port 465/tcp, or SMTP with implicit SSL a.k.a. SMTPS a.k.a. SSMTP, which a mail exchanger rarely uses to receive messages from the internet, and only sometimes uses to receive messages from other systems on your network.
   Usually, the internal network is considered as trusted (to some extent). You would want to use SMTPS in networks where the network is not trusted, perhaps including requiring authentication using client-side SSL certificates. At the very least, the implicit SSL nature of SMTPS ensures the communication stream is encrypted.
3. Port 587/tcp, or Submission, which is intended for use by actual human beings. This is the service Kolab Groupware chooses to make use of.

You would not want to use ports 25 nor 465 for submission of data from your groupware users, because both usually include bluntly allowing what is called $mynetworks - the list of IP Address (Spaces) that are trusted networks.

To illustrate, consider the following; You have a LAMP stack that sends out notifications to subscribers via the SMTP port number 25 on localhost. The envelope sender could be something like apache@example.org or noreply@example.org. You would typically have configured (it may actually be the default configuration) "mynetworks = 127.0.0.0/8". The setting "smtpd_sender_restrictions" or "smtpd_recipient_restrictions" typically includes "permit_mynetworks".

Now you install an awesome webmail program called Roundcube, and you configure it to use localhost:25 as its SMTP server. Anyone able to log in to Roundcube is now able to configure an identity with any envelope sender address making messages sent out using that identity appears as if they were coming from that actual envelope sender.

So, joesmo@example.org could now send messages pretending to be the CEO of Example.org, Inc., simply by adding an identity of "The Bossman ", and would go completely unchecked while doing so.

What you can't do is restrict $mynetworks further than 127.0.0.0/8 - your LAMP stack would fail attempting to send messages to your subscribers.

You will want to use a service that does not trust *anyone*, *requires* authentication and *enforces* the user sending the message is authorized to send messages using the envelope sender address of the very message.

I say "authorized to send messages (...)", because one feature that you may need is delegation; the secretary sending a message or two on behalf of the bossman is not a very uncommon scenario.

This is where Kolab Groupware kicks in. We implement pieces of software you are probably already intimately familiar with, and then take it up a notch.

It will not accept messages appearing to originate from an envelope sender address that is within a local domain name space from anyone or anything, unless the actual sender is verifiably authorized to send using the envelope sender address the message will appear to have come from, or unless the sender system is explicitly trusted.

By means of a demo, at Kolab Systems itself, not only do we run a couple of Kolab Groupware deployments, but we also service some mailing lists for third party Free Software projects - hence I can only verify the actual envelope sender address, and not the domain name space:

$ (
> sleep 1
> echo "HELO $(hostname -f)"
> echo "MAIL FROM: vanmeeuwen@kolabsys.com"
> echo "RCPT TO: vanmeeuwen@kolabsys.com"
> echo "DATA"
> sleep 1
> echo "Subject: Nice blog post, thanks"
> echo "Hey there,"
> echo ""
> echo "I think your blog post is awesome."
> echo "."
> sleep 1
> echo "QUIT"
> ) | telnet ext-mx01.kolabsys.com 25

Then do the same against your own SMTP server, with a set of address of your own.

We've done a couple of announcements and blog posts in a variety of locations about this new thing called Kolab 3.0 - you may have heard about it.

It's been a little while since you got your last update, and many people walk up to me on the street in my hometown of Cardiff in Wales - of all places - asking me when they can expect Kolab 3.0 beta to be released. Nah, just kidding.

All kidding aside, Kolab Groupware has a very, very large user community. While I recognize many are in anticipation for Kolab 3.0 to be called stable, I have to admit a large proportion of our users have Debian-based deployments.

Long story short, the packaging for APT repositories has lagged for quite a while. Thankfully, with the help of community members, the packaging for Debian Wheezy has picked up pace and we're closing in on being able to bless it.

Since I am a credit-where-credit-is-due kinda guy, please allow me to say thank you on your behalf to the following people, who volunteer their time and effort; in no particular order;

• Paul Klos
• Michael Kiefer
• Johannes Graumann
• Robert Tasarz

It's the new eleventh commandment the Kolab Groupware community shall live by: "11. Thou shall applaud their efforts".

As Debian packaging is picking up, we find more and more people are willing and able to sink their teeth into it and report issues.

To me, personally, and I would literally be the person to pull the trigger on releasing Kolab 3.0 beta, it would be detrimental not to say offensive and/or disrespectful if we were to slap the beta label on to Kolab 3.0 without giving this relatively new momentum in Debian packaging a chance to complete its efforts. Offensive and disrespectful not only to those doing the work, but the large proportion of our community awaiting the work to complete for them to enjoy as well.

So, that's what we're waiting for - for Debian packaging to be in a state where those doing the work and therefore also being at the receiving end of bugs to tell me they are satisfied - noted I keep a very keen eye on progress.

To keep in touch with the progress being made, please consider subscribing to our Kolab development mailing list, and more importantly, register for a bugzilla account to be able to add yourself to CC: on the APT packaging tracker bug.

If you want to try Kolab 3.0 on Debian, please don't hesitate to read up on our documentation with installation and configuration instructions.

While only a warning in setup-kolab, some people set up a Kolab Groupware server with with an improper system fully qualified domain name, with all sorts of undesirable effects. I'm actually seriously considering making this warning a fatal error.

Why does a system require a proper FQDN? Is it actually required?

I'll answer the second question first, since it has the shortest possible answer: No.

Of course a proper FQDN is not absolutely required. But it is strongly recommended. For example, the default "localhost" and "localhost.localdomain" names you'll find in /etc/hosts (with address 127.0.0.1) are not "fully qualified domain names".

Right after you install a system (unless you have a fancy network infrastructure with (dynamic) DNS and DHCP, or a prepared DNS entry for a static IP address), the only way a system knows how to refer to itself is using these names "localhost" and "localhost.localdomain".

With a Kolab Groupware setup, this is somewhat problematic. Not because there's a guarantee it won't work, but because it is an insensible default, and secundary components that install alongside Kolab Groupware won't work as expected.

So what is a proper FQDN? Well, it exists of at least two parts: the host's name, and a domain name. For example, if the host's name is 'kolab01', and the domain name is 'example.org', the FQDN would be 'kolab01.example.org'. Please note that the domain name itself 1) exists of the domain name you (probably) registered and the top-level domain name you registered it with - apologies to those in the know, for bastardizing the terminology, and 2) may actually be a sub-domain name, such as 'ch.kolabsys.com'.

When you set a system's FQDN it is important that it exists of at least these three parts divided by a dot (.) character. Why?

A domain name (for example, 'example.org') is used for three purposes: 1) its IN A is probably pointed at a webserver, so that people can navigate to http://example.org equally well as http://www.example.org, 2) its IN A is used as a fallback recipient SMTP server address should no IN MX records exist, 3) it contains the SOA and NS records (authority statements, signatures, and other things).

Because of 1), the server to which the IN A for the domain name is pointing, it is unlikely (for larger deployments) that this is also the server on which the groupware deployment runs. Furthermore, it is best practice to set your MX records, possibly to the same system, but MX records nonetheless, so that you may have a level of redundancy (by supplying more than one MX record).

That said, the Kolab Groupware component that is the 389 Directory server refers to itself using the configured system's FQDN. While this can work should the IN A for "example.org" never change, it is more problematic should the system only be using 'localhost' or 'localhost.localdomain' - your 389 graphical console won't be able to connect to the administrator server with that set.

Furthermore, it is just bad, bad practice. A domain like 'example.org' is like a particular forest, and one of your systems is one of the trees. Unless you had a very specific deployment in mind, I'm sure you'll want to be able to refer to a single tree rather than only being able to refer to the entire forest as a whole.

That said, this is not what setup-kolab is developed to handle either. It'll chop off the first part of the "FQDN", so using 'example.org' will lead the setup to conclude you're going to want to set up for domain 'org'. It'll use that domain to build you a standard root dn as well, which in this example case would become 'org'. Settings that relate to login realms, and other sorts of stuff all over the place will result in unexpected behaviour, not to say fail.

In our web client based on Roundcube, we have had a need to get very large LDAP directories to function as an address book. While "large" is a relatively subjective term, in terms of LDAP "large" simply means running out of time, or meeting the look-through or size limits. Additionally, of course, PHP itself is subject to memory size and maximum execution time restrictions.

We weren't the first to run into these problems, of course, but with a little help we came up with a very, very good solution. Using Virtual List View (VLV) control and Server-Side Sort (SSS) control, browsing very large address books page-by-page takes maybe a tenth of a second per page. Roundcube address book problem solved (thanks Thomas, thanks Robert!)

But are they complex?

VLV and SSS controls are both very complex (the former more than the latter), with configuration on the LDAP server required and client-side BER encoding magic involved. I am not a programmer, let alone traditionally educated, so to me the PHP code (or the code's capabilities, rather) resulting from this effort is simply awesome. To properly function, it requires a patch to PHP to be applied, which I've now submitted for upstream inclusion. Kolab Groupware ships its own PHP packages solely to include this patch. There's many other gotchas to take into account as well.

Gotcha!

To use server-side sorting, you have to create an index on the LDAP server with a single attribute containing a list of attribute names using which entries can be sorted. When using LDAP from a client, you have to specify that exact same sort order.

To use Virtual List View controls, you require a successful hit on the server-side sorting (as that is the index). You also have to configure the LDAP server with a search (for which it requires a base, filter and scope). When using LDAP from a client, in addition to hitting the server-side sorting, you have to hit those three exact same search parameters as well.

That may not sound so difficult...

Complications with the User Interface Context

However, an LDAP client like Roundcube lists and/or searches differently based on the context.

One context is the address book. Typically one would browse an addtress book sorted by "Surname, Givenname" or "Givenname Surname". These one or two attributes (displayName can be put to very good use here) would then be included in the server side sorting. But then again, who sensibly browses a very large address book page by page? A 1000 entries with a page size of 40 results in 25 pages already - address book sizes I speak of when I say "large" are literally hundreds of thousands of entries.

So the keyword is searching - but searching is a complex realm in and by itself. If you were allowed to search only on the attributes included in a SSS and VLV, searching is virtually useless (all puns intended). Searching would basically allow you to list VLV entries starting with the first result your search had found.

You would like to search by location, phone number, department, organization, email address and other attributes. This is difficult to do efficiently, but can be done. The show-case is in what I'm sort of announcing in this blog post.

After your search results are in, you would still like them sorted somehow. If you would like to know who lives in Narnia and search for it, your results should not look like so:

1. Narnia
2. Narnia

but like so (for example):

1. Doe, Jane
2. Doe, John

A second context is auto-complete. In it's very nature, auto-completion is always a search (and not simply a paginated list of all entries), similar to the search for people living in Narnia. You can imagine how a set of search results could need to be paginated if there were many people living in Narnia.

Furthermore, in the user interface context where you use auto-complete (the compose window, the ACL entry management on folders), no address book interface pop-up is available, as only a small list with hits is displayed. The pages are much smaller, increasing the change the result set needs pagination.

Naturally, auto-completion searches for a limited set of attributes - but also most likely different attributes than an address book listing. Most likely, a search for the purpose of auto-completion looks at a display name ("Doe, J. (Engineer)", perhaps), a given name ("Jane", "John"), a surname ("Doe"), any email address (mail, alias, mailalternateaddress, mailforwardingaddress attributes), etc.

Imagine these searches would occur without the help of pre-sorted indexes. A million-and-one LDAP entries may need to be searched, all results would need to be obtained, the results would then need to be sorted, to then return only the first 10-15 entries. Wouldn't it be great if you could search, sort and paginate with the help of pre-sorted indexes?

But... I did mention earlier, the three magic search parameters need to match the server-side VLV search configured. As it turns out you can actually specify additional search parameters, but only in a very specific way. More awesome stuff I'm announcing in this blog post.

And all of this is only in Roundcube?

Within the Kolab Groupware realm, we have more components that hook into LDAP and perform listings and searches. Two of them are the Kolab Web Administration Panel (or actually, its API) and Syncroton.

So I've had multiple agenda's to attend to; 1) avoid needing to do actual programming for the Web Administration Panel, 2) avoid requiring Roundcube libraries with both the WAP and Syncroton and 3) further enhance our LDAP capabilities for yet even more awesome stuff.

I give to you Net_LDAP3 - not yet a PHP PEAR module, because it's a work in progress, and I don't like their license requirements and I don't like their function naming conventions (camel-case, basically).

My target is to integrate all of the LDAP functionallity required with Roundcube, our Kolab plugins for Roundcube, the WAP and Syncroton.

I've already achieved the following milestones:

• Unless specifically configured, or specifically disabled, or not a supported control on the LDAP server, the Net_LDAP3 module automatically discovers available settings for VLV/SSS listings (method list_entries())
• A method search_entries() with an argument for additional search parameters that automatically uses VLV/SSS configured or discovered but with the additional filter settings,
• Validation of the desired sorting,
• Automatic selection of one sorting configuration out of available sorting configurations,
• Allow a method login() to find a user entry before binding as the user being logged in,
• The execution of a registered hook to obtain or set configuration items using a routine external to Net_LDAP3,
• The execution of a registered hook to use for logging,
• Trace and debug levels

Obviously there's still many things on my TODO list, which ends with patching Roundcube, the WAP and Syncroton to use this module.

With the release of the Kolab 3.0 alpha1, we now enter a stage in the development process and release management called testing. Usually an alpha release is sort of a development snapshot, but to be honest with you we didn't change as much as you might think, between 2.4 and 3.0.

One of the things that needs testing is the upgrade of your system to Kolab 3.0, be it from Kolab 2.2, 2.3 or 2.4.

I am, therefore, writing some documentation on performing that upgrade.

Shortly, I'll be off to travel to Tallinn, Estonia, to attend Ye Olde Annual KDE Conference aptly dubbed Akademy.

This is going to be something new to me, to be honest. First, I'm new to Estonia - never been there before in my life. Second, I'm a newbie in the KDE community. I haven't got the slightest clue what's awaiting me after I touch down, but for the stories my fellow Kolabian Paul Adams has shared with me. All of them funny, not all of them good...

Furthermore, I must admit in all honestly, I myself am a GNOME user. By convenience, more than anything else, and though it's not supposed to be bad, of course, yet I feel like I might be walking into the lion's nest ;-)

I'm confident I'll enjoy myself - but not in the way you're thinking of right now. I've heard Tallinn is a very beautiful city, and I regret not going to see much of it - if this conference is anything like the dozens of other conference I've been to. I'm sure there'll be plenty of KDE people who turn to be on speaking terms with a GNOME user after we have a good old-fashioned how-much-beer-can-you-chuck contest.

Yet though, all fun aside, the strongest impression I need to make is not at any random bar. It's on stage, right in front of all those people.

I'll be giving a talk titled "KDE Releases that Just Work(TM)" - this is supposed to be where I make my move. I've been practicing different stances to emphasize different points all week long.

I'll speak to subjects I think I'm an expert on, and that I think are open for vast improvement within the KDE project. There's a couple of gotchas though;

1. I'm not the only one to think he's an expert
2. I'm not the only expert

For release engineering and quality assurance and so ultimately the quality of releases to really improve, consensus between a large variety of people needs to be established, not all of them with anywhere near the same agenda, let alone anywhere near the same opinion.

This sort of effort touches processes, tools, workflows and ultimately what the individual developer is doing. The trick is to find consensus on a way to improve (the use of) all those tools, to make all of that touching further enable the people that actually do the work - those that'll give me the evil stare for using GNOME - and make them feel empowered, and achieve greater satisfaction.

Many Kolab "users" (system administrators?) want Debian APT packages for Kolab Groupware, version 2.4 preferably, of course.

Over the past few weeks, different people have mentioned they could volunteer some time to create and maintain packages (needed for testing, eh?) and other people offered to test (the yet not existing packages). However, no packages actually appeared in any repository and no commits to the GIT source code management repositories we use for the packaging were being made.

So, I (an RPM packager) have taken it on - APT packages. A Linux distribution I'm not all too familiar with, a packaging mechanism I'm not all too familiar with, and guidelines and processes and rules I'm completely unaware of.

When I say packaging for Debian feels like a waste of time to me, it's not that I mean it's fruitless - supposedly many people will enjoy the packages - it's that I'm extremely inefficient doing so. Things that are obvious to more experienced APT packagers will cause me to run in a circle a few times, if you know what I mean.

Furthermore, under the misnomer of Getting It Done(TM), I bluntly copy what I have for the RPM side of things, and I ignorantly ignore lintian warnings that probably have a meaning (not necessarily caused by RPM-ishes, BTW).

First on my list was some of my own code (pykolab), until I found out 389 Directory Server isn't available for Debian. Pykolab likes 389 Directory Server very much, as do I myself. They have a permanent lease on a room in the hotel right across the street from where I live, if you will.

Then I found that it is, and that the work on it had started as early as 2005, but that it still is not made available through the standard repositories. Push comes to shove, I rebuilt all the relevant components and we're now shipping it as part of the Kolab 2.4 repositories.

Long story short, it's still a work in progress, but those of you able and/or willing to scrutinize my work; the following is a list of packages for which you should be able to find the corresponding GIT SCM repository at http://git.kolabsys.com/apt/

$ wget -q http://mirror.kolabsys.com/pub/debian/kolab-2.4/dists/squeeze/developmen... -O- | gunzip -c | grep ^Package | awk '{print $2}'
389-admin
389-admin-console
389-adminutil
389-console
389-ds-base
389-ds-console
389-dsgw
cyrus-imapd
idm-console-framework
jss
kolab
ldapjdk
libapache2-mod-nss
libmozilla-ldap-perl
openldap
pykolab
svrcore 

This last week and a half or so, I've been working on implementing Resource Management based on the new Kolab XML format version 3, which uses xCal for events.

Individual Resources

Individual resources -that is to say, the way I've implemented them- consist of a Calendar folder in IMAP (a Kolab Shared Folder in IMAP, with type 'event'), name ("label", if you will), and a recipient email address. Events for which a resource is invited are added to the relevant Calendar folder. For example, a resource "Volkswagen GTI" (of type "Car") gets recipient email address: resource-car-volkswagengti@example.org, and a shared event folder "shared/Resources/Volkswagen GTI@example.org".

Resource Collections

As described in the Resource Management article, we are considering resource collections ("cars", "confererence rooms") to be eligible for selection when a user seeks to make reservations for such resource. This enables users to indicate they want to make reservations for "a car" rather then a specific car. I've implemented this by collecting the individual resources in a collection as groups, which naturally have a title (the common name of the group, in fact).

When a user selects such a resource collection, and sends out the reservation request iTip, it is sent out to the canonical address for the resource collection (in this example resource-collection-car@example.org). A Kolab content filter named Wallace detects the fact the message contains an iTip, and that resource-collection-car@example.org (one of the attendees) is a resource collection.

Wallace (or actually, its 'resources' module) then takes the individual resources that are in the collection, and detects whether there is a conflict between the existing events scheduled for each individual resource.

If none of the individual resources have an event already scheduled, Wallace randomizes a selection and will attempt to use that resource instead of the original resource collection. In true iTip-style, this means a delegation response needs to be sent back to the organizer, with the delegator being the resource collection, and the delegatee being the individual resource selected.

This allows a user to invite a resource collection, but indicate a preference for a particular resource - though not implemented yet. Think, for example, a user could invite resource-collection-car+resource-car-volkswagengti@example.org.

Overall Thoughts

The implementation (of the aforementioned) is not fully done yet. I have a bunch of TODOs remaining:

L10n and/or i18n

On my personal TODO list is to allow this folder path to be localized, so that for an installation that has German configured as its locale, the folder name becomes "shared/Ressourcen/Volkswagen GTI@example.de", or, in Dutch, "shared/Middelen/Volkswagen GTI@example.nl". I'm awarding myself bonus points if I can also make the "resource-" and "resource-collection-" be localized somehow, though I realize it is in fact an arbitrary prefix altogether and could be made to dissappear.

The name used for the actual folder (as well as the transliteral, normalized ascii string for the email address) is a name the user supplies in the Kolab Web Administration Panel.

Delegation, Cancellation

While I pride myself these actions or iTip message types can be implemented quickly, it's not actually been implemented yet.

Caching

In my tests (sending tens of thousands of randomized reservation requests around), it can take Wallace a significant chunk of time to resolve conflicts as it operates against raw IMAP and thus has to retrieve all events, one by one, at least until a conflict is detected. This clearly doesn't scale, so Wallace will need some caching. I'm tempted to allow re-use of the caches that Roundcube already maintains, though there's little spare room for large quantities of events that users normally don't need caching for.

Naturally, there would also be the opportunity to use existing Free/Busy information, if any, and/or to create and/or update such Free/Busy information or trigger such update to happen. The backend for this Free/Busy information could be using the same caches as well...

Long story short, I'm not done with regards to caching and event conflict detection quite yet!

Recurring Events

Recurring events, or event occurrences to be exact, are not yet being parsed (fully). With libkolab however, it should become relatively easy to do with the existing codebase now still a part of kcalcore (from KDE).

Reservation Request Responses

The responses sent out to reservation requests do not look as pretty as I would like them to look, regrettably. I think the user experience (compared to what I have now) could be greatly improved.

To stick-shift, or not to stick-shift.

There's no design yet for parameters to a resource to be stored and taken into account, so that a user might select "a room with 6+ seats, a beamer and a wired internet connection for the speaker / for each participant". Similarly, Wallace might take into account a general reservation request with 7 attendees should not be allocating a room with 4 seats.

Users Should Not View the Resource Calendars

See the following screenshot for why users should not be able to access the resource calendars directly; Firstly, the number of resource calendars to a deployment be a very large. Secondly, these resource calendars may just contain too many events.

That is to say, of course, a particular type of user could get access to the resource calendar itself, such as a secretary, but it would likely be write access.

Click to enlarge the image.

It's Tempting to go Out-of-Specification

It's tempting to go outside of the specification, and attempt to automate things further. If a user were to propose a meeting time and no room is available, what could Kolab do? Kolab could issue a counter-proposal changing the event to a time a room is available, for example. Kolab could also CANCEL the event.

Also, when a user selects a resource collection (such as, in this example, "a conference room"), the specification says to send a delegation iTip message back to the organizer only. It is then, according to the specification, up to the organizer to choose whether or not to send out an updated event (with the actual room) to the other attendees. But, of course, a "room" may be bound to a physical location, and I would think one would want the attendees to be updated period, and not leave the choice whether or not to send the updated information out to the attendees to the organizer.

After more than 3 years, the review request for package rubygem-passenger can finally move forward.

A bundling exception was finally granted, because someone is putting in the work to make the bundled, forked and patched version of Boost ultimately dissappear - by bringing the required changes upstream.

While I proposed doing so early on in the saga, though I could not do it myself and would have needed to find someone else to do so, that wasn't acceptable - as is tradition in the Fedora Project by now, the "discussion" on my plan of attack went absolutely nowhere, with people arguing in circles, simply for the sake of arguing, and without ever having been, being or feeling endangered of being involved in any way, shape or form.

Patience is golden, the cost of running Passenger in my own build systems can now be avoided. Time for 3.0.12 to be packaged for Rawhide, built against Ruby 1.9, and the review to move along!

One part we're doing some significant work on for Kolab 3.0 is the Kolab Web Administration Panel (or WAP, for short).

Again, you ask? You may remember the look-and-feel revamp Kolab Systems performed in Kolab 2.3 - this one is different though.

I hear you think, "Don't they all say that all the time?", and you're probably right. Pictures or it didn't happen, is what I would say. I'll do you one better; Want to click around yourself, perhaps?

Check out admin.klab.cc/~vanmeeuwen/kolab-wap/public_html/. Login with your full email address (not an alias, see bug #594), such as 'jeroen.vanmeeuwen@klab.cc', but not 'vanmeeuwen@klab.cc', and the same password you are using for our http://webmail.klab.cc demo environment.

Don't have an account yet? Don't worry. Send your givenname and surname to sysadmin-main+kolab@klab.cc and one of our admins will create an account for you.

Please note it is a work under construction, and this is a "Release Early" under the "Release Often" mantra. We have a list of known issues. Feel free to add one or two, the more the merrier.

Should you want to try yourself, perhaps against your own Kolab Groupware (test?) installation, please clone the GIT repository like so:

$ git clone git://git.kolab.org/git/kolab-wap

and follow the instructions in the INSTALL file. Note that you will need a MySQL server, and a system that allows you to have an /etc/kolab/kolab.conf file in .INI format (that means, no installing this on existing OpenPKG Kolab 2.3 servers) - work is undergoing to document the full extent of the configuration options available. An example is available here.

Many people have been wondering why the documentation for Kolab Groupware currently refers to version 2.4, and what that version 2.4 is all about. We have to take a couple of steps back in order to address the question, and review what has happened, and why.

Documentation... regrettably, it's one of those things that tends to lag behind the actual development and release of a software project. Perhaps it's because many people contributing to the software development have little interest in writing up how their software should or could be used. Perhaps some of the people that would write it all up do not have confidence in their documentation skills - perhaps a language barrier is a part of that. In any case, so far, the majority of documentation for Kolab Groupware version 2.4 has therefore been written by yours sincerely, though admittedly parts of it are attributable to others who've created wiki articles on a particular subject, such as the chapter on Combatting Spam.

Kolab Systems, the vendor of Kolab Groupware, and the company that enables me to pay my bills, has a vested interest in the adaptability and continued development of the Kolab Groupware solution. We have found that, using the traditional deployment model for Kolab 2.3, and with the Kolab 2.3 code-base, integration into existing environments, scalability and redundancy had proven to be... suboptimal.

Suboptimal not in that it was impossible to do, but the code-base would have needed customizations that in turn would prove to be mutually exclusive with deployment in other environments, again spawning the need for customizations. This is a virtuous circle, but would have taken a very long time to increment in value and move forward. Long story short, for integration and adaptability purposes, the code-base needed some serious re-factoring.

The Kolab components we are speaking of mainly in this context would include the Kolab daemon, it's Kolab perl library, which is used by kolabconf, and the web administration panel. Inherently, the changes would also impact client software, such as Horde and Roundcube.

On August 26th, 2010, work on PyKolab began. Its goals were set out to be broad yet very clear; allow for sufficient option value to integrate with existing environments sustainably.

PyKolab, today, includes the following components;

• Kolab daemon
• Kolab SASL Authentication daemon
• Kolab SMTP Access Policy
• Kolab Content Filter
• Kolab Command-line utilities
• Format interpreters

All of this was, and still is, somewhat mutually exclusive with Kolab 2.3. It could therefore not be released as part of Kolab 2.3. Furthermore, Kolab 2.3 is product series for which the reference implementation continues to be OpenPKG, and that does not ship Python.

So, we're looking at something that is not Kolab 2.3 to ship PyKolab with. After some deliberations, it was decided that it could also not be called Kolab 3.0 as we were looking to make many other changes, many of which are outlined in the agenda for our last meeting on February 27th, that would, for most of them, each by themselves justify a major release (see KEP #5, "Server Product Versioning").

Because its availability had become urgent, it was decided to dupe the product series "Kolab 2.4". Perhaps, in hindsight, it was not the wisest of decisions, but it happened nonetheless.

So why were most of you not involved in this decision?

First, urgency - a specific customer needed the changes that were in PyKolab right-away.

Second, we had not yet defined any sort of process around development cycles, let alone agreed upon a set of processes with you, the community.

Third, the changes involved are significant. No, let me rephrase - the changes involved are extra-ordinarily humongous. An attempt to outline what was going on with the Kolab SMTP Access Policy implementation outlines the amount of confusion surrounding what was going on, but please allow me to illustrate some further shock-waves that would have hit the development mailing list too early;

• The reference platform for server implementation is native packages on Red Hat Enterprise Linux 5 - not OpenPKG packages on Debian.
• The reference implementation for the authentication and authorization database was the technical equivalent of Netscape Directory Server 7.2, and today is 389 Directory Server, not OpenLDAP.
• Kolab 2.4 ships Cyrus IMAP 2.4 without any Kolab specific patches.
• Kolab 2.4 does not use the Perl Kolab daemon, nor perl-Kolab, nor kolabconf, nor kolab_bootstrap.
• Kolab 2.4 ships an LDAP schema that is stripped down (significantly) as to not include organization-specific schema extensions, and not include server configuration items.

All of this, the replacement of components you are comfortable with in Kolab 2.3, and development of new components and new features all needs documentation. This new documentation, of the software I have developed and we are using as the basis to work towards the future, Kolab Groupware 3.0, is what is on http://docs.kolab.org. Now you know why it refers to 2.4.

So why were most of you not informed about 2.4?

Kolab Groupware 2.4 requires a serious transition between what you run today, and what you would be running, regardless of whether you run an OpenPKG-based deployment, or native packages available through, for example, the Debian repositories.

Packages for distributions other then Red Hat Enterprise Linux are barely available, still - a shortage in resources contributes to this, and you, the community, is not yet participating in making anything happen in this area. This of course is quite a vicious circle as well. The changes involved require clear documentation - something I'm working on, but that'll remain a work in progress for quite some time to come. The setup process (you probably know as kolab_bootstrap) is not automated - something I'm working on. The Kolab web administration panel that is compatible with the other changes involved is under active development. Etcetera, etcetera.

How can you help?

The more things you throw back over the fence, the better. We'd like to do a good job, you know, and if you raise an issue you do not have to feel like you're also responsible for resolving it. Here's some ideas of what you could do.

1. Read the documentation at http://docs.kolab.org/.
   I recommend starting with the Architecture and Design guide. It's only about 124 pages all-inclusive...
   Let us know whether it makes sense, whether you understand what it says / what it is about, what you're missing, etcetera. If you don't understand what you read in the documentation, the issue is with the documentation. I recommend using the issue tracker to create tickets, or discuss on the development mailing list.
2. Help define how -you think- things should work.
   You of all people know what you want better than anyone else. We're interested to learn about your use-cases, deployment scenarios, requirements. I recommend using the development mailing list for this.
3. Write documentation.
   There's many things that currently require some text still. You'll recognize the blanks throughout the documentation currently available.
   Writing documentation is one way to enjoy one of the steepest learning curves you'll ever get. Note that any documentation  you do contribute doesn't have to be perfect right off the bat, language- or otherwise, it is a collaborative and an evolutionary process.
4. Give-It-A-Go(TM).
   Follow the instructions in the Community Installation Guide using an Enterprise Linux 5 system, such as Red Hat Enterprise Linux 5, or CentOS 5 - for the part that has been completed. If you favor a different distribution or a different version, please consider contributing to the packaging effort. Contact me (RPM) or Christoph Wickert (APT) for more information. To avoid any confusion, Christoph is an excellent RPM packager as well, BTW.
   Let us know where things go wrong or things don't make any sense.

You turn your attention away from your computer - you should lock your desktop. It is not just about level of trust you put in the people that you think may indeed gain physical access to your system. If like myself, you run a home office, physical access to your system is pretty restricted. If though, again like myself, you travel frequently and attend events, you better get used to locking your desktop as soon as your attention diverts away from the screen. Here's 10 reasons;

1. Privacy
   Your desktop is your own, and what you choose to do on your computer is nobody's business. An open chat-window with a (girl)friend may hold contents that could make the people dearest to you believe something seems what it is not - I reckon all of us sometimes say things in the privacy of a conversation with someone (we think) we know. You rarely know who's scraping your screen and what they may think of it, though.
2. Oops!
   Sometimes screens dim or power off and sometimes people use a key like space or enter to wake up the computer. Sometimes these buttons trigger an action such as submitting a form, or execute a command. Using the delete button to "wake up a computer" of course isn't any better.
   FWIW, I use the left-hand shift key, as it is the furthest away from any key that does anything significant.
3. Confidential Communications and Documents
   If you work for a company, sometimes somebody sends you information that is supposed to be for your eyes only. If the world knew you to not lock your desktop, how could they trust you with information that is supposed to not leak out?
   Furthermore, communications and documents may be subject to a non-disclosure agreement, either directly with you or with your employer. Whom you trust doesn't matter when it's about who's subject to an NDA, you know.
4. "Switch User", not "Log Out, Log In"
   Your favorite desktop's unlocking dialog likely offers a user that needs the computer the capability to log in as a different user, maintaining your session, whereas it is common for somebody else to just use your open browser window to check his/her email / Facebook / Google+ / Twitter a little - using sites you may be automatically logged on to when using your user account.
5. Your Cat / Dog / Hamster
   If you have a pet, surely you've noticed how they are kee to walk over your desk possibly accidentally hitting any of those key you have managed to avoid using to wake up your computer...
6. su access es mi access
   With an unlocked desktop, you are surrendering the access to your files - whether they be personal, private, confidential or public. Litterally anyone gaining physical access can modify, remove or share data from your desktop.
   Furthermore, if you're anything like me, you probably have access to other systems. This may be through a certificate or two, a VPN connection, or an SSH key. You may also have a PGP key to sign and encrypt email with. Let's be honest, most people run agents or have no passphrase on these things at all. Go figure.
7. Unsaved Work
   I'm fairly certain you do some things on your desktop that are of some significance to someone, especially while you perform tasks for work. Somebody else closing off an application you have running does not necessarily mean they save changes to documents, or save them to a location where you can find that version.
   Frankly, though, if you manage to save all your changes constantly, perhaps using the Ctrl+S keyboard shortcut, I fail to recognize how you manage to do so but fail to press Ctrl+Alt+L (or Super+L if you will) when you step away.
8. Practical "Jokes"
   There are screensavers that just never go away. There's applications that will force you to admit something rather embarrasing as well. Whether you trust other people (that you shouldn't trust) with confidential information, do you trust them to not pull a practical joke on you as well?
9. Keyloggers and other hackydihack
   Once you've left your desktop unattended, and unlocked, how do you make sure that your desktop is secure from that point on forward? We know of keyloggers, and we know of programmable USB sticks that can present themselves as a HID keyboard and start typing as soon as they are plugged in. Admittedly, probably only a geeky forensic scientist can figure out what happened afterwards.
10. You Owe $x a Beer
    My personal favorite is using some mailing list I know you are subscribed to (and I am too), to send on your behalf - using your account, an invitation for beers on your dime. Naturally I disclose the fact you are buying beers because you have not locked your desktop.
    FWIW, most of the time I don't really have to have access to your unlocked desktop in order to be able to do this kind of thing using your account, since it's pretty likely lists you and I are both subscribed to (that are actually eligible for this, i.e. nothing like full-disclosure), run on servers I manage.

Now that I've got it going, I could not stop myself from sharing this with you.

What do I mean by multi-domain mailman list server, you may be wondering? Well, imagine you have two development projects, foobar and bazbah, and each has their own domain name; foobar.org and bazbah.org.

If you where to take a standard mailman installation, you could only have one announce, devel and users, and you would need to choose to which of the two projects to attach those lists.

Usually this makes mailman site administrators choose for another option: prefix the name of the list with the name of the project. Then juggle around the multiple domain name spaces for each list and hope the footers, archives and other pages work out (there's a command in mailman, called withlist, with which you can fix_url after changing the URL for a mailing list... right).

An alternative of course is to run multiple mailman instances. Being the RPM package management oriented guy that I am though, I want those multiple instances to be packaged properly, of course, and available through YUM repositories.

Furthermore, I'm a self-diagnosed Security Enhanced Linux fanboy.

So, what did I do?

First, I got the sources for the mailman package in Fedora Rawhide (no point in taking an older one). I modified the RPM spec file so that it would take an optional %{domain} parameter all the way through. Since I'm using something Fedora Rawhide on Enterprise Linux 6, I needed to clause the various references to systemd stuff as well. You can see the result here.

Then, I made me this little script that makes things happen including using the original spec to spit out specs for individual mailman instances, build the source RPMs, and submit the build to Koji.

Presumably, you'll get httpd_t SELinux AVC denials against mailman_cgi_exec_t;

type=AVC msg=audit(1330454256.738:564): avc: denied { search } for pid=12091 comm="httpd" \
name="cgi-bin" dev=dm-0 ino=1183684 scontext=unconfined_u:system_r:httpd_t:s0 \
tcontext=system_u:object_r:mailman_cgi_exec_t:s0 tclass=dir
type=AVC msg=audit(1330454256.738:565): avc: denied { getattr } for pid=12091 comm="httpd" \
 path="/usr/lib/mailman-lists.syncrotron.org/cgi-bin" dev=dm-0 ino=1183684 \
scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mailman_cgi_exec_t:s0 tclass=dir

This is resolved with a relatively small policy, easily deployed with the Puppetmanaged.org SELinux module. Include in your node manifest:

selinux::policy { "httpd_mailman": }

Furthermore, you should set the context for the files deployed by the RPM packages, as is shown in the Puppetmanaged.org mail module from line #94 onward.

After some hard work, it it finally is: the new Kolab website.

It's seriously improved over the old website, but of course there's still some work to do. We'd like to be able to link to 'login' and 'register' pages, for example - so that you guys can all register and post to the forums, etc.

If you want to help out - we also have a test instance we can play with - please let us know on the websites-team mailing list. We seek skills in Drupal (6) + modules, JavaScript, CSS and of course, HTML.

Kennelijk is er een model waaronder een bedrijf, hede ten dage, kan opereren zonder een algemeen email adres voor bijvoorbeeld de afdeling Klantenservice, of om klachten naar op te sturen. Voorbeeld #1 en #2.

Beide voorbeelden zijn van Nuon, waarmee ik grappig genoeg, edoch niet grappig -na onvoldoende voldoenende communicatie vooraf, een dispuut probeer te openen over een factuur van enkele honderden euros.

Onbegrijpelijk, want Twitteren en Facebooken doen ze allemaal. Deze heeft zelfs een YouTube kanaal (voor filmpjes over de inmiddels bedreigde soort "tevreden klant", ongetwijfeld).

Gegeven welk formulier dan ook, daarbij, kun je niet eens een goed betoog opsturen - het is danwel maximaal 1.000 karakters (voor een vraag), danwel maximaal 2.000 karakters (voor een klacht). Het spijt me zeer, maar mijn klacht is (erg dichtelijk samengesteld) toch zeker 3.800 karakters.

De zaak draait erom, dat Nuon eist dat ik meer betaal over 2011 dan over 2010. Uitleggen kunnen ze het niet, en ikzelf heb de ballen verstand van meters - ik vertrouw een derde partij, stom genoeg, de netbeheerder. Ik heb wel iets verbruikt, natuurlijk, maar laat ze eerst maar eens zien dat de gegevens die ze denken te hebben kloppen, en uitgelegd kunnen worden. Afgeleid over de cijfers die ik van ze krijg ben ik tenminste 1 keer het klokje rond gegaan, en dan wel voor een apartement dat het afgelopen jaar (2011) heeft leeggestaan (het enige wat stroom consumeert is de verwarmingsketel die bevriezing van het hele zwikkie voorkomt). Ik verwacht, redelijkerwijs, een significant lager verbruik dan voor een apartement in gebruik (2010). Het zal op het laatste kilowattuurtje niet neerkomen, maar ik heb zo langzamerhand toch ongeveer een driedubbele vakantie naar een tropisch eiland in termijn-bedragen bij ze uitstaan - en nog willen ze meer.

Dit, voor mij, spelt oplichting. Factureren, inclusief de dreiging van het inschakelen van een incasso bureau en algehele afsluiting, om vervolgens zonder uitleg nog meer te gaan factureren, en (tot 2 jaar!!) later significante bedragen te crediteren, doet bij mij de emmer overlopen.

The Kolab web administration panel is being re-factored, or should I say, developed from the ground up.

We're splitting the user interface and the backend logic into a frontend "client" and a backend API, in order to allow for easier integration within other management and corporate products. The frontend client, the user interface I'm about to show off, therefore calls upon the API to get to certain settings, logic and other fancy stuff. Let's see what it can do already;

See here the client login screen. Once you press submit, it actually calls the API to create you a session, and tokens are being bounced back and forth. The way this works (or is supposed to work) exactly is documented in our Architecture and Design guide. Currently, we only allow login with either the LDAP Distinguished Name of a user, or the primary mail attribute value - this is to be improved still.

After (successful) login, you'll be taken to an overview screen - I'm thinking this is to, in the future, display tasks pending (for sysadmins), and/or overall status of one's Kolab deployment.

You can see there's some icon work to be done, still, and that we're not yet "fully featured". We have an idea of how it's supposed to work, of course, but we're nowhere done implementing all of it. The key functionality is to first implement administration for "users" and for "groups".

While this development deployment of the new Kolab Web Administration Panel runs against https://webmail.klab.cc, with currently over a hundred users or so(!), and with our larger deployments running with hundreds of thousands of users, we have considered user administration to become 'search based' as opposed to 'list and browse based'.

We've still provided a listing of (20) users, which is currently not configurable in size, with browsing capabilities. This, including a variety of other changes Kolab Systems has developed, should allow very large user databases in LDAP to still function properly with this interface. Please note that I've logged in here with my personal credentials - not the System Administrator credentials, which is why I'm only seeing a sub-set of users that are actually in the system.

Now on to one of the nicest features we've developed; Automatic generation of certain form fields when adding a user (of a certain type). We currently ship with one default user type only; the "Kolab User". We're working on allowing administrators to define their own user and group types, so that adding a user or group offers you a quick way to identify required attribute values, and allows other attribute values to be generated using the information you supply. Have a look;

We're adding a user here. I give the user a given name attribute value of "John". Nothing is happening yet, but here it comes;
As soon as I leave the textbox in which I am to supply the given name attribute value, the frontend client calls the API to generate "other attribute values" given this new information. You can here see that the display name is being filled out automatically.
Now, having supplied the surname, please find the display name is completely filled out. The mechanism applied here contributes to consistency for multiple adminstrators working to add users to the same Kolab Groupware deployment. But that's not all of it!
In the System tab, other attribute values are filled out automatically as well - again using API calls. In the background, the API is using the same settings as are provided to the new Kolab daemon that I've developed, so that the daemon's "recipient_policy" plugin - if enabled, of course - does not have to change the same attribute values afterwards.

These "auto" form fields, calling the API in the background, are going to become fully configurable. That is to say, 1) which attribute values are to be generated automatically, 2) using what form data already provided, 3) rules to authorize someone to override automatically generated values, and 4) per user/group type.

I've mentioned "user and group types" a couple of times before, so let me illustrate what that's supposed to be able to provide you with.

Sometimes, you just want to add "a Kolab User" - this entry is supposed to have a mail attribute representing a valid email recipient address (and email envelope sender address), amongst other things. Suppose however you want the user to just be "a POSIX user" - but without Kolab Groupware. You would add a user type called "POSIX user", and set the mandatory, recommended, automatic and other form fields (and values), and from that point on forward, whenever you add a user, you get to choose what type of user that should be.

Of course you can make this "whatever you like" - if all "Kolab Users" are also supposed to have POSIX attributes, and Samba attributes, and the like, you can just add these form fields to the existing "Kolab User" type.

As I'm addicted to using Alt+Tab to switch between open application windows, you can imagine that in GNOME 3, where Alt+Tab only switches between grouped applications (read: two open terminal windows are grouped) is... a little different. You would go to that terminal application icon, and then a little downward arrow would indicate multiple windows had been grouped.

If you were to release Alt at that point, you would be presented with the window from that group you last opened.

I had been using the arrow keys to navigate to the window I actually wanted (down to expand group, left-and-right to select window), but now I have a new thing to get addicted to.

I just discovered this magic key-combo, so I'm pleasantly surprised and I didn't want to withhold all that positive energy from you; Alt+`

Go fetch!

Lydia and I have decided on a mechanism to learn Spanish. First things first, it's crucial we vastly increase our vocabulary.

Every week, starting this week (#2, 2012), we point at objects, think of activities, just tell eachother the time, every day life if you will, up and until we collect 50 or so, and find the Spanish translation.

Then, at the end of the week (this is going to be the exercise over the weekend) we learn those ~50 translations and start talking jibberish - It'll be a mixture between Spanish, Dutch and English, I imagine.

The more words we learn, as our vocabulary grows, the more we are (going to attempt) to actually speak Spanish. We should thus be able to learn about ~2.600 translations a year, and while our pocketsize translation dictionary holds about 37.000, we should be OK for the next decade and beyond :P

I recently, very recently, purchased a Lenovo X220, fully beefed up with an i7 processor, 8GB of RAM, and the largest SSD (Intel 160GB), amongst other things.

Installing Fedora 16 (of course) went smooth (as was to be expected, as I know other people with Lenovo X220's). It's nicely locked down now, with startup, BIOS, GRUB passphrases, limited boot devices (SSD only) and an encrypted VG -which I can afford doing now without slowing down the entire system too much.

As a Kolab Systems employee, running multiple Kolab servers, naturally I install Kontact, the Kolab client, and I tend to do this using a fresh install (i.e. no copying data from the old laptop, no upgrading).

Fedora 16 includes a 4.7.4 KDE PIM stack, which turns out to not work for me. Having configured the Kolab accounts, it seems I cannot get to the messages in my Kolab INBOX -other folders work just fine.

In any case, I decided to try rawhide; the version of the KDE PIM stack included in rawhide at this moment is 4.7.95, KDE's latest release en route to 4.8 - this too, however, did not work for me.

So, I decided to try and build from GIT - my first time ever. KDE has a utility for this, called kdesrc-build. It's use is pretty straight-forward, but I had to install some build requirements on my system. This is what I have installed now:

# yum -y install \
    alsa-lib-devel attica-devel avahi-devel boost-devel bzip2-devel check-devel cups-devel \
    cyrus-sasl-devel dbus-devel dbusmenu-qt-devel enchant-devel fontconfig-devel \
    freetype-devel gamin-devel gettext-common-devel gettext-devel giflib-devel \
    glib2-devel glibc-devel glib-devel gnutls-devel gpgme-devel grantlee-devel \
    gstreamer-devel gstreamer-plugins-base-devel herqq-devel ilmbase-devel jasper-devel \
    kdebase-workspace-devel kdelibs-devel kdepimlibs-devel keyutils-libs-devel krb5-devel \
    libacl-devel libattr-devel libcom_err-devel libdrm-devel libgcrypt-devel \
    libgpg-error-devel libical-devel libICE-devel libjpeg-turbo-devel libpng-devel \
    libselinux-devel libsepol-devel libSM-devel libstdc++-devel libtasn1-devel \
    libudev-devel libutempter-devel libX11-devel libXau-devel libxcb-devel \
    libXcomposite-devel libXcursor-devel libXdamage-devel libXext-devel libXfixes-devel \
    libXft-devel libXi-devel libXinerama-devel libxkbfile-devel libxml2-devel libXpm-devel \
    libXrandr-devel libXrender-devel libXScrnSaver-devel libxslt-devel libXt-devel \
    libXtst-devel libXv-devel libXxf86misc-devel libXxf86vm-devel mesa-libGL-devel \
    mesa-libGLU-devel mysql-devel OpenEXR-devel openldap-devel openssl-devel \
    pcre-devel phonon-devel polkit-devel polkit-qt-devel PyKDE4-devel PyQt4-devel \
    python-devel qca2-devel qt-devel qt-gstreamer-devel qtwebkit-devel raptor2-devel \
    shared-desktop-ontologies-devel sip-devel soprano-devel sqlite-devel strigi-devel \
    xorg-x11-proto-devel xz-devel zlib-devel

Consider installing the "Fedora Packager" group as well;

# yum -y install @fedora-packager

After following the setup instructions, you should first initialize your copy of the various sources (otherwise failures would cause you to need to manually cleanup subversion repositories, for example):

$ kdesrc-build --src-only

This, when you run it for the first time, can take quite a while.

Once it's done, you can start building stuff:

$ kdesrc-build

This, too, can take quite a while. Furthermore, it requires a lot of energy, and drains my 7.5 hour battery life in about 90 minutes ;-)

UPDATE^1: The build dependencies for gwenview need to be added to the list of build requirements; exiv2-devel.

I was watching this documentary I don't recall the name of. I remember recognizing Stephen Hawking though, which is one of the reasons one of the statements made in the documentary caused me to raise my eyebrow and question the rationale put forth.

Summarizing, the narrator said that in essence, no time exists inside (close to?) a black hole. A clock that would travel into a black hole would stop ticking, or so the audience were told. I suppose it's fair enough to reason time no longer exists inside a black hole, though from where I am sitting -no astrophysics background whatsoever- it sounds like quite the assumption.

Anyways, it was argued that the Big Bang itself, the event that supposedly caused the universe to exist as we know it (even though we know very little of it), was some sort of black hole imploding on itself.

It was then argued, that, therefore, no time existed before the Big Bang. The documentary further argued, that since no time existed before the Big Bang, it was therefore impossible for some sort of grand designer to have existed, let alone create the universe.

Now, I'm not exactly in favor of pointing to a grand designer that created it all, but this documentary narrates flawed reasoning for such grand designer to not have existed.

The link that is being drawn between one type of black hole (the ones we think we have in our universe, that exist with surroundings, in which time exists) and another type of "black hole" (the one we can only speculate about, for which it is assumed no surroundings existed, in which time could have existed), in that in each type of black hole, no time exists nor existed, neglects the fact that outside of such black hole, surroundings may exist or may have existed, other localities if you will, in which thus also time may have existed. It's not like current black holes cause time to not exist anywhere outside of it, right?

What is referred to as 'condstore' is actually the feature-set that IMAP4Rev1 extension added in RFC 4551. It enables the quick synchronization of flags on messages as well as other conditional STORE operations.

I'm going to have to refer you to the RFC for the full details, but suffice it to say your Kolab 2.3 deployment could greatly benefit from enabling said extension on your mailboxes - it is not enabled by default as part of Cyrus IMAP 2.3, only with Cyrus IMAP 2.4 is the 'condstore' annotation set to 'true' by default.

You will have to execute two separate actions on your Kolab server:

1. Configure new mailboxes to have 'condstore' enabled by default,
2. Configure current mailboxes with 'condstore' enabled.

You would preferrably execute these actions in this order, to reduce the chance new mailboxes are being created while you have not yet switched on 'condstore' to be enabled by default.

Enable 'condstore' by default

To enable condstore by default, in your favorite editor, add the following line to /kolab/etc/kolab/templates/imapd.conf.template:

mailbox_default_options: 2

Afterwards, write out new templates and reload Kolab with the new configuration;

/kolab/sbin/kolabconf

Enabling 'condstore' on existing mailboxes

To enable 'condstore' on existing mailboxes, login to Cyrus IMAP using the cyradm utility:

cyradm -t "" -u manager localhost

From there, you can examine one of the existing mailboxes;

localhost> info user/john.doe@example.org
{user/john.doe@example.org}:
  condstore: false
  duplicatedeliver: false
  lastpop:  
  lastupdate: 12-Sep-2011 17:56:54 +0200
  partition: default
  pop3newuidl: true
  sharedseen: false
  size: 105382261
  folder-test: true

As you can see in the example, condstore has not yet been enabled for this mailbox. To enable condstore for a particular mailbox:

localhost> mboxcfg user/john.doe@example.org condstore true

You will not want to repeat this command for all mailboxes, but instead use wildcard matching:

localhost> mboxcfg user/*@example.org condstore true

NOTE: The two commands do not return any output.

NOTE^2: Don't forget to, at your option, also enable condstore for mailboxes in the shared namespace, in other authentication realms, and perhaps even for those mailboxes that reside in the DELETED namespace (just in case they are restored to a visible namespace later on in life).

First of all, Bron Gondwana has done a great job over the course of a number of releases, to resolve some of the bugs in Cyrus IMAP that may have been around for quite a while -though most of them you may not have noticed.

Since Cyrus IMAP has been around since 1993 (give or take), it inherently contains legacy implementations of features or required functionality, sometimes to the tune of "good enough". Seeking further compliance with IETF approved RFCs and allowing for further development has spawned a 2.4 software series, with large amounts of code refactoring and cleanup, and the addition of some interesting features -more on those later.

Just this week, the Cyrus IMAP team has released cyrus-imapd-2.4.10.tar.gz. One day later, I've submitted the build to Fedora rawhide, so it should be hitting your doorstep soon enough. I plan to follow up on releases upstream with packaging more quickly and more agile in the future. Yours truly being the Release Engineer for Cyrus IMAP & SASL, as part of my work for Kolab Systems, I like how we have established a culture of release early, release often within the Cyrus IMAP project, and how core developers like Bron can make releases happen almost autonomously, while in between the last version in the 2.3 product series and the first version of the 2.4 series, a good couple of years had past.

Further endeavors include our 2.5 roadmap (which I admit could use some love), documentation (which I work on) and of course testers and developers are always welcome!

In the Cyrus SASL realm, we're working to convert the CVS repository to GIT. This takes a long time, since like Cyrus IMAP, Cyrus SASL has been around for quite a while. I must say we have done our due dilligance over the past few months on this one though, so I expect the repository conversions to GIT I've been doing to be approved soon enough.

Using GIT for the Cyrus IMAP codebase has proven to be a lot more attractive to a lot more contributors (since they can now create their own working copies and collaborate on those), and I hope we achieve the same level of success for Cyrus SASL.

I'm having a great time at FUDCon in Panama. For one, the weather is much better then back in Wales... :P Parties are at the villa of les Chiles Loco (the crazy Chileans), and last untill about 3am in the morning. Plenty of beer and plenty of barbeque steak usually do the trick for me :)

My first session on Thursday, titled "Why You Are All Idiots", was post-poned because someone from Dell needed to have his session on what seems to have been the busiest FUDCon day so far. Irony has it, the talk was on "Virtualization with KVM", and while my schedule was freed up, I went to assist Dell Panama with their KVM / Live-migration demonstration setup they were going to show on Friday :)

So, after all, my first session was on Friday, and on "Cloud", following two other presentations on virtualization, and the second was on "Software, RPM, Packaging, Guidelines and Build Systems... but why?", starting off in a small series of talks on packaging and the koji build system.

Other people are having many interesting talks as well -but in Spanish. Since I only hablo pocitto espanol (and that's about it), it is sometimes hard to follow. For most of today -outside of my "Why You Are All Idiots" and two lightning talks, I've been working to get a working GNOME3 desktop environment on my laptop.

Looking forward to FUDPub.

Please allow me to describe my day of travelling yesterday;

Having woken up at 05:30am ET, I started at Detroit airport, as I just so happened to be in the area already. From Detroit, a Continental (now United) flight to Newark (oops, high building, zig, oops, high building, zag, touch-down). Short layover, continued my trip to Miami -from the air it looks much like in the CSI series.

Note to self: do not wear black shoes, black trousers and a black T-shirt when travelling through Miami as you have to go outside to the designated smoking area which has no shaded area anywhere near it.

From Miami, I had an endurable Copa flight to Panama City, where immigration lines are as bad as they are in the U.S. Alejandro picked me up, dropped me off at the hotel, and took me back to Cuidad del Saber, where I met up with the rest of the guys -about 18 hours after I started.

One too many beers and a barbeque later, Alejandro drove me back to the hotel for some well-deserved, long-needed rest.

Today, I've been preparing some of the sessions I'm going to pitch or try and squeeze in the schedule, including but not limited to the following titles;

• Cloud
• Why You Are All Idiots
• Software, Packaging, RPM, Guidelines, Build Systems... but why?

I suppose other sessions I could/should also be doing are on Spins, Fedora Trademark / EMEA / NPO, Extended Life-cycle Support, Configuration Management with Puppet and Fedora in the Enterprise. I can pitch'em all, but I'll have to see which ones actually make it on the rather tight schedule ;-)

Applications that we integrate with Kolab Groupware have a genuine need for caching. But what is it exactly that causes this need for caching?

Why Does One Cache?

Let's first think about why one caches in the first place. Caching is usually implemented to eliminate a bottleneck and boost performance. Data can then be obtained from the relatively quick cache -it is "close by", and it usually understands and is optimized for some form of querying- as opposed from the relatively slow original source of the data.

With Kolab, using Cyrus IMAP as the backend storage for all groupware related information like Email, Events and Contacts, it can hardly be argued the IMAP server does not perform up to specifications. Cyrus IMAP is extremely fast and scalable; it is, arguably, just as lean and mean as a cache would be.

So, Where's the Bottleneck?

Yet, the following topology does introduce and work around a bottleneck, seriously impacting and later improving performance. I'm off to explore what exactly is the bottleneck, and how to work around it.

We'll use Z-Push (Free Software ActiveSync implementation) in a regular, current scenario workflow as an example. The following happens, justifying caching, when a mobile device requests synchronization;

1. The 3rd party application connects to Cyrus IMAP to retrieve "the information". Since not all folders may need syncing, and some folders contain Email while others contain Events or Contacts, a list of IMAP folders needs to be obtained.
2. Cyrus IMAP, its efficiency in this matter aside, interacts with the mailbox storage to retrieve certain information from its mailbox database, its annotations database, the message files, etc.

Using the IMAP protocol, this means;

• Listing the folders the user is authorized for.
• Iterate over the list of folders and retrieve the following annotations for each folder:
  • "Should this folder be synchronized with the mobile device at all?"
  • "What type of groupware items does the folder contain?"
• Then, excuse my paraphrasing, on a per IMAP folder target basis, the changes that may or may not have been applied on the mobile device, need to be compared with the changes that may or may not have been applied in the IMAP folder, and vice-versa. Retrieving changes, messages, parsing them, and comparing them, and applying the changes on either end, while tracking which changes have already been communicated to one or the other end of the synchronization exercise.

Naturally, the last step is what Z-Push is in charge of, in this example. and It does have certain characteristics and interactions with Cyrus IMAP as well as with its own caches to optimize the performance, scalability and user experience.

The former notwithstanding, this is just one application integrated with Kolab responsible of maintaining its own cache. In current generations of Kolab, Horde Webmail does the exact same but using a different cache, and future generations of Kolab will include RoundCube, which again also maintains its own cache.

One Cache per Application?

Maintaining a cache per 3rd party application integrated with Kolab isn't necessarily the most sustainable route to go. Feasible? Yes. Sustainable? Perhaps not. Let's take one step back and look at the bigger picture again;

Presumably, the interaction between Cyrus IMAP and its storage can not be optimized further (which the dotted double arrow is supposed to indicate). Not without intrusive changes at the very least, that is to say, while admittedly I'm unaware of our options to further increase performance in this part of the flow of information. If you have ideas and the necessary experience, let me know and I can get you hooked up.

It is, perhaps, the IMAP protocol used in between the Cyrus IMAP server and the 3rd party application that is the bottleneck. For example, Z-Push cannot do the following over IMAP, eliminating a number of iterations and sequences issuing IMAP commands;

SELECT folder FROM folders
    INNER JOIN annotations ON folders.id = annotations.folder_id
    WHERE annotations.key = '/vendor/kolab/activesync' AND annotations.value = 'true';

Hey, this does somewhat represent what it does against the cache it maintains, having obtained the information over IMAP once (slow) it uses its cache to obtain the information (fast) in a number of subsequent synchronizations -limited to an expiry interval, expiring and updating cache, of course. This builds us the following picture, where IMAP is "slow" for the task at hand, and SQL is fast;

Ignoring the interaction that Cyrus IMAP requires with the filesystem as being a negligible performance penalty, and focussing on how the 3rd party application wishes to optimize performance, apparently it would rather perform caching (cheap), then it would want to interact over the IMAP protocol (expensive).

Suggestion #1: One Cache To Rule Them ALL

It has been suggested, since most if not all of the 3rd party applications integrated with Kolab would require some form of caching, we create "one cache to rule them all";

Although probably this is in fact entirely feasible, the following constraints to such architecture come to mind;

• Session reliability and personal information security, complex to implement but even more complex to audit, and implemented up to specifications with IMAP ACLs already,
• Duplication of ((a significant) part of) the data,
• Abstraction from caching required in all 3rd party applications, each of which has their own already (i.e. significant development effort right from the start, and continuous development effort for more 3rd party applications to integrate with Kolab Groupware), and one uniform caching specification across all of the 3rd party applications (i.e. significant design complexity).

We (within the Kolab community) regularly refer to the "one cache to rule them all" as "server-side akonadi" - currently the very efficient client-side (offline) caching in our primary smart client, Kontact.

Suggestion #2: Maintain Cyrus IMAP Databases in Networked SQL

It has also been suggested (by me, in fact), to have Cyrus IMAP use database formats other applications within a Kolab Groupware deployment could read from. By having Cyrus IMAP maintain its mailbox, annotations and perhaps even mail folder indexes and caches in a database format like SQL (instead of Berkely or skiplist), these would become available to the 3rd party applications without them having to populate the cache first, and the cache would be updated "automagically"; as a result, the level of interactions with Cyrus IMAP over the "inefficient" IMAP protocol would be further reduced -"inefficient" for the task at hand, that is.

However, this would greatly impact the scalability of Cyrus IMAP. It would, in fact, greatly impact the overall performance of Cyrus IMAP as an IMAP server. It's a valid option, but not considered feasible targetting for because of the projected performance penalties.

Suggestion #3: Use Cyrus IMAP

If you agree it's fair to label having to use the IMAP protocol to get to the data required as the bottleneck, here's the suggestion I have in mind; Add a thin, lean and mean, network-enabled, read-only C application to interface between the 3rd party applications and the Cyrus IMAP databases (on the filesystem), thus enabling the 3rd party applications to use a different protocol or querying language to obtain the data in a more efficient manner. Perhaps this would look as follows:

Benefits would include many requirements have already been implemented; Locking, networking, database maintenance, threading, thread safety, TLS/SSL, access control though IMAP ACLs and its handling and more of that stuff. The new application could, presumably, also maintain its own caching capabilities to be even quicker.

Just some early Saturday morning thoughts... let's see what the rest of the weekend brings.

Turns out the LDAP module for Python breaks the API while developing version 2.4 -with no backwards compability, but a workaround is relatively easy. If, like me, you need to be able to run code on platforms traditionally "slow" in adopting the latest and greatest, as well as those that are traditionally, relatively "fast", here's an idea:

# Catch python-ldap-2.4 changes
from distutils import version
if version.StrictVersion('2.4.0') <= version.StrictVersion(ldap.__version__):
    LDAP_CONTROL_PAGED_RESULTS = ldap.CONTROL_PAGEDRESULTS
else:
    LDAP_CONTROL_PAGED_RESULTS = ldap.LDAP_CONTROL_PAGE_OID

class SimplePagedResultsControl(ldap.controls.SimplePagedResultsControl):
    """

        Python LDAP 2.4 and later breaks the API. This is an abstraction class
        so that we can handle either.
    """

    def __init__(self, page_size=0, cookie=''):
        if version.StrictVersion('2.4.0') <= version.StrictVersion(ldap.__version__):
            ldap.controls.SimplePagedResultsControl.__init__(
                    self,
                    size=page_size,
                    cookie=cookie
                )
        else:
            ldap.controls.SimplePagedResultsControl.__init__(
                    self,
                    LDAP_CONTROL_PAGED_RESULTS,
                    critical,
                    (page_size, '')
                )

    def cookie(self):
        if version.StrictVersion('2.4.0') <= version.StrictVersion(ldap.__version__):
            return self.cookie
        else:
            return self.controlValue[1]

    def size(self):
        if version.StrictVersion('2.4.0') <= version.StrictVersion(ldap.__version__):
            return self.size
        else:
            return self.controlValue[0]

Example Usage

Where 'self.ldap' is the LDAP object;

    def _search(self,
            base_dn,
            scope=ldap.SCOPE_SUBTREE,
            filterstr="(objectClass=*)",
            attrlist=None,
            attrsonly=0,
            timeout=-1
        ):

        _results = []

        page_size = 500
        critical = True

        server_page_control = SimplePagedResultsControl(page_size=page_size)

        _search = self.ldap.search_ext(
                base_dn,
                scope=scope,
                filterstr=filterstr,
                attrlist=attrlist,
                attrsonly=attrsonly,
                serverctrls=[server_page_control]
            )

        pages = 0
        while True:
            pages += 1
            try:
                (
                        _result_type,
                        _result_data,
                        _result_msgid,
                        _result_controls
                    ) = self.ldap.result3(_search)

            except ldap.NO_SUCH_OBJECT, e:
                log.warning(_("Object %s searched no longer exists") %(base_dn))
                break
            _results.extend(_result_data)
            if (pages % 2) == 0:
                log.debug(_("%d results...") %(len(_results)))

            pctrls = [
                    c for c in _result_controls
                        if c.controlType == LDAP_CONTROL_PAGED_RESULTS
                ]

            if pctrls:
                size = pctrls[0].size()
                cookie = pctrls[0].cookie()
                if cookie:
                    server_page_control.cookie = cookie
                    _search = self.ldap.search_ext(
                            base_dn,
                            scope=scope,
                            filterstr=filterstr,
                            attrlist=attrlist,
                            attrsonly=attrsonly,
                            serverctrls=[server_page_control]
                        )
                else:
                    # TODO: Error out more verbose
                    break
            else:
                # TODO: Error out more verbose
                print "Warning:  Server ignores RFC 2696 control."
                break

        return _results

Or something like that ;-)

I recently purchased a SIP phone, one of those hardware devices, so I'm now online on Fedora Talk as well. It's not that softphones such as Ekiga or Twinkle wouldn't work, they just wouldn't work as well while I'm connected to a VPN (with my laptop). A Siemens C475 IP DECT solves the problem of lag for me ;-)

I'm on extension 5100680, so if you need me, and you have Fedora Talk, give me a quick call!