Planet Ergo

You are somewhere in a Sun Belt state in the U.S., and it's nearing the and of January. For some, this sounds like a hopeless situation. For others, it is the opportunity of a lifetime!

Here's 10 things to do in Tempe, Arizona, nearing the end of January, 2011, while FUDCon is taking place.

1. Bacula Backup and Recovery; Get some insight on the subject and engage a Bacula Systems Certified Trainer on the subject. I'd love to insert a "Tuesdays at noon" type of thing, but we're just not sure when exactly such sessions would be taking place. It's not important either, there's at least 9 other things to do!
2. Koji Build Systems. Do you use it? Do you use it because you need something happening downstream? Are you a third-party repository enthusiast? Let's get to it, and have at least one session on the build system suite.
3. Long Term Support (LTS), Extended Lifecycle Support (ELS), or whatever you may want to call an extention on the regularly supported time period of updates coming to a stable Fedora release. I like the latter terminology better. If you're interested in stopping it though, please stay away?
4. Spins. Custom spins, localized spins, remixes or bluntly put, dwarf forks. How can we make sure all this momentum is harnassed and nurtured up and until the point little projects become proud masters of the universe with our help.
5. Packaging Guidelines and the Package Review Process. I've had my share, how about you?
6. Cyrus IMAP; first-hand experience of a) How Fedora Rules the Universe, b) What Upstreams Expect From Distributors and c) How To NOT Behave As A Distributor.
7. The Ever So Awesome Fedora Cloud. I'm not sure where it's at, but I do think I know a little something about what cloud is all about. It's time to get that first concept implementation for Metadata DNS going, while showing off a complete Open Source stack for Any Company(TM) in a demo-environment with public access of sorts.
8. Fedora Hosted Groupware. I know Fedora Infrastructure to have looked at Zarafa, not to say I have shamelessly plugged it back in the day. In the interest of full disclosure, I now work for Kolab Systems -another Groupware ISV, this time fully open source and free software.
9. Puppet. Modules. Modules. Modules. Cross-platform consistency. Approaches. Packaging. Writing modules (why and how)
10. Beer, meat (that's me), friends, laughs (that's me too!). I can think of very little more I need in life.

It's been a very busy past two weeks and I haven't had much time to show my face on IRC or respond to email. I've participated in or executed some interesting activities though, so please allow me to share those with you.

On Monday, almost two weeks ago, I visited Bacula Systems headquarters in Switzerland, to get a 1-on-1 training from Kern Sibbald -the original author of Bacula backup & recovery software-, in order to become a certified Bacula Systems Trainer. A quick walk-through the slides along with some insight on what Kern thought were the most important highlights to mention and/or explain during the training, a review of the exercises and one day later, you have sufficient additional information apart from the course material itself.

The next day, Eric Schirardin introduced me to how the Bacula Systems course classroom setup was to be configured, so that all the exercises could be performed without the course attendees having to go through the motions that in fact would have had little to do with Bacula itself.

Since my flight back to the Netherlands was leaving early that evening, and I had an important Kolab Systems conference call in between, I had to hurry and catch a train to the airport, find myself a nice, quiet corner somewhere, a wifi connection and call in. I made it, but that morning I had no idea how smoothly we would go through the motions to get me to be able to set up the classroom and at what time we'd be done exactly. Let's just say I had too little time to get any lunch before I caught a train just in time for me to arrive at the airport and not have any of the ambient train noises while sitting in a conference call. Then again, I don't usually eat during lunch anyways ;-)

The next day (Thursday last week), I was going to travel to Osnabrueck for a visit to a valued partner on behalf of Kolab Systems; a three-hour train-ride from Utrecht. In order to arrive at a reasonable time, I had to get up at around 6 in the morning, not something I'm very much used to, nor a fan of ;-) Either way though, the visit was planned for Thursday as well as Friday, and it was definitely worth the travel! I was staying with Christoph Wickert -you know him from his contributions to the Fedora Project and various upstream projects- in Munster throughout my stay in Germany.

On Saturday, I was on my way back home again. Since the Bacula Systems Administration Course was to start on Tuesday, and Monday was allocated to preparing the classroom (and testing the setup), most of my weekend was spent at going through the slides and exercises and preparing my story. Luckily, I was still very familiar with the course as I had recently participated in the course myself, so I could afford to visit a friend's birthday party as well.

On Monday, it turned out that VirtualBox (used to create a second backup client for the Bacula Systems Administration Course), installed on a 64-bit laptop without hardware virtualization acceleration capabilities cannot run a 64-bit guest, so I quickly installed a 32-bit guest and configured it, which actually took the majority of the single day that I had available.

Tuesday, Wednesday and Thursday, at Amaziq Source in Amsterdam, I ran the Bacula Systems Administration I Course (there's a part II coming up as well, with more advanced topics) with 6 participants from 4 countries, and I'd like to think that it was very successful. At least Bacula Systems (through their senior engineer Arno Lehmann) thought so, because I passed the test running the course according to their quality standards, and so now I'm a proud Bacula Systems Certified Trainer ;-)

I've been working on a backup strategy for our Zarafa groupware environment, using Bacula backup & restore software.

Since we have a Zarafa support subscription at Operator Groep Delft, we have a utility called zarafa-backup. Regrettably, this utility only allows you to create a Full or Incremental backup. Here's how the utility works:

• If you run zarafa-backup for the first time, it'll create a .data.zbk file per mailbox, with a .index.zbk file indicating the contents of this one .data.zbk file.
• If you run zarafa-backup the second time, provided the .index.zbk file is still around, it'll create a .data.zbk.1 file with only the new content of the mailbox, and update the index file.
• If you run zarafa-backup the third time, provided the .index.zbk file is still around, it'll create a .data.zbk.2 file with only the new content of the mailbox, and update the index file.
• etc, etc, etc.
• If you want to create a Full backup at some point, destroy the .index.zbk file and run zarafa-backup - it'll think it's the first time you run the utility since there's no existing .index.zbk file for the mailbox.

This is incompatible with a decent backup strategy. Creating a full backup in a larger environment is rather expensive, so you would want one full, say, every so many months or so. With small incrementals stacking over a longer period of time though (31 days in a month), you can imagine that restoring yesterday's mailbox content can quickly become one full backup and a million-and-one incremental backups. You would need to restore the yesterday's .index.zbk file along with all .data.zbk files from your backup suite. The incremental backup may be spanned over several locations, though, and your expiration policy or storage limitations may require you to expunge data from tape or disk after a couple of months. At some point, restoring just becomes such a huge effort it outweighs the downside of creating a new full backup.

Let's state some numbers, just for fun, and for further reference:

• 1.000 mailboxes
• 1GB mailbox quota

Both attachments and message content (text) count towards the quota, but in the most extreme scenario, all mailboxes are full. The theoretical maximum volume to backup would be about 1TB.

The actual volume will always be lower; Even though some users have a tendency to complain they run out of quota, it turns out that, given a progress bar counting upwards to their maximum, and the simple fact that they will not receive any more email when they run out of quota, causes them to clean out their mailboxes. That is, if nobody ever gives in to requests to increase the quota, and you automaticalle expire everything that is irrelevant (calendar entries in the '90s) The average usage currently is about 15%. The total footprint as you can see is a lot less then the theoretical maximum of 1TB.

However, this "15%" (or ~150GB) will take a little over 24 hours to export through zarafa-backup. As running the utility impacts the overall performance of the groupware environment (the backend zarafa-server process is heavily utilized and zarafa-backup in that sense behaves like a big, fat client), you can see how this (if you expect higher average usage) might result in a policy that requires you to create a full backup somewhere between Christmas and Newyears. Other reasons for the volume to increase beyond what is reasonable to create a full backup for on a regular basis may include company growth, more temporary hires, what-have-you.

Ergo, a proper backup strategy is required. Something that let's you restore a mailbox with yesterday's contents, but without the hassle of getting 364 incrementals. Somehow, you would need to preserve the .index.zbk files along with the .data.zbk files on a daily, weekly, monthly, bi-monthly or even yearly basis, so that you can use those to backup "what has been changed since the last full", so that you can expire all the little incrementals of "what had been changed per day the last month".

So, let's assume you have the following backup requirements:

1. Restore mailboxes per month for 6 months
2. Restore mailboxes per week for 1 month
3. Restore mailboxes per day for 3 weeks

This means you can restore any mailbox day-by-day for the past three weeks, week-by-week for the past month, and month-by-month for the past 6 months. Presumably, your Bacula Schedule resource would look like:

Schedule {
    Name = "Zarafa-Monthly"
    # A full backup starts every first weekend and takes little
    # over 24 hours to complete.
    Run = Level=Full 1st Saturday at 11:59
    # During the work-week, we backup every day
    Run = Level=Incremental Monday-Friday at 23:05
    # In weekends other then the first, ...
    Run = Level=Differential 2nd-5th Saturday at 23:05
}

This starts a full backup every month, which you may want to decrease. I for one specifically don't want to do that. I'm saying: "If the backup impacts production too much, give me more power." As you can see I start the backup on Saturday's though, so it has more then 44 hours to complete before it's Monday morning 8am. Admittedly, our environment is mostly idle during the weekend except for a few ActiveSync clients. Here's the FileSet:

File Set {
    Name = "Zarafa-Bricklevel"
    Include {
        Options {
            Signature = SHA1
        }

        # No trailing slash!
        File = /var/lib/zarafa/bricklevel
    }
}

As you can see, I'm only interested in /var/lib/zarafa/bricklevel/. Then, last but not least, the most interesting part of the configuration; the Job and Pools:

Job {
    Name = "Zarafa-Backup"
    Type = Backup
    Level = Incremental
    Client = "mail01.ogd.nl"
    File Set = "Zarafa-Bricklevel"
    Schedule = "Zarafa-Monthly"
    Storage = "Zarafa-File"
    Messages = "Standard"
    Pool = "Default"
    Full Backup Pool = "Zarafa-Full-Pool"
    Incremental Backup Pool = "Zarafa-Inc-Pool"
    Differential Backup Pool = "Zarafa-Diff-Pool"
    Write Bootstrap = "/var/lib/bacula/bootstraps/Zarafa-Backup.bsr"

    Run Script {
        Runs When = Before
        Runs on Client = Yes
        Fail Job on Error = Yes
        Command = "/usr/local/sbin/zarafa-backup %l --before"
    }

    Run Script {
        Runs When = After
        Runs on Client = Yes
        Runs on Failure = No
        Command = "/usr/local/sbin/zarafa-backup %l --after"
    }
}

Pool {
    Name = "Zarafa-Full-Pool"
    Pool Type = Backup
    Recycle = Yes
    AutoPrune = Yes
    Volume Retention = 6 months
    Maximum Volume Jobs = 1
    Label Format = "Zarafa-Full-"
    Maximum Volumes = 9
}

Pool {
    Name = "Zarafa-Diff-Pool"
    Pool Type = Backup
    Recycle = Yes
    AutoPrune = Yes
    Volume Retention = 40 days
    Maximum Volume Jobs = 1
    Label Format = "Zarafa-Diff-"
    Maximum Volumes = 9
}

Pool {
    Name = "Zarafa-Inc-Pool"
    Pool Type = Backup
    Recycle = Yes
    AutoPrune = Yes
    # Keep a Volume for 3 weeks minimum
    Volume Retention = 21 days
    # Rotate volume every week, whether failed jobs are in
    # there or not.
    Maximum Volume Jobs = 5
    Label Format = "Zarafa-Inc-"
    # 5 jobs (e.g. one week) in one volume;
    # 3 volumes per 21 days
    # + a little leeway
    Maximum Volumes = 6
}

You may have noticed the RunScripts I'm using are something custom. That's right! Please see the attached zarafa-backup script.

NOTE: This is just what I whipped up just now, I haven't truly tested it. Please be careful when implementing it in your environment ;-)

NOTE^2: zarafa-backup is a perfect utility to restore only part of a mailbox, too. You can choose categories of items such as only a user's Email, Calendar, Tasks or Contacts. This is why you need to consider using zarafa-backup, as opposed to a snapshot-and-sync type of backup.