Planet Ergo

I was setting up a network infrastructure the other day, and needed trunking over bonded (Linux) network interfaces connected to a Cisco switch for a virtualization <-> storage network (Network #1), and "the rest" (production network, management network, etc.). Here's just some quick notes:

• My rule of thumb was: The higher the VLAN number or IP space network number, the more private the network is.
• I decided the Management network was to be VLAN 666 (how appropriate) with IP space 10.66.6.0/24. The storage network inherently became 10.66.7.0/24 (VLAN 667) because it was even more private then the management network.
• I decided that any public servers would be in VLAN 2 (very public, very close to the Internet, etc.).

Let's suppose these were all the networks I needed.

On the Hypervisor, I configured eth0 and eth1 as slave interfaces for bond0. bond0 itself though was not supposed to have any IP address configuration.

# cat ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
HWADDR=<some-mac-address>
ONBOOT=yes
MASTER=bond0
SLAVE=yes
# cat ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
HWADDR=<some-mac-address>
ONBOOT=yes
MASTER=bond0
SLAVE=yes
# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes

For a bonded interface, you need to choose the exact mode of operation, and because the interfaces were bonded to increase the throughput, I chose 802.3ad. For it to actually happen, you need to configure the bonding kernel module through /etc/modprobe.conf:

alias eth0 bnx2
alias eth1 bnx2
alias bond0 bonding
options bond0 mode=4 miimon=100

NOTE: Listing the physical interfaces first is mandatory.

Still, we have no network configuration. The only network configuration I needed was a semi-physical interface in the storage network, and a set of 802.1q encapsulated interfaces for the rest of the network communications. Ergo, I created the following interfaces:

• bond0.2 (public)
• bond0.666 (management)
• bond0.667 (storage)

The only interface out of the three that would actually get an IP address though was the storage network interface bond0.667. The configuration would look as follows:

DEVICE=bond0.667
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
TYPE=Ethernet
IPADDR=10.66.7.1
NETMASK=255.255.255.0
NOZEROCONF=yes

The other two interfaces (bond0.2 for the internet and bond0.666 for the management) are a little more tricky. They needed to be bridged interfaces, in order to allow virtualized guest nodes to be positioned in either one of those two networks. The configuration for bond0.2 therefore looked as follows:

DEVICE=bond0.2
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
TYPE=Ethernet
BRIDGE=br2

Bridge interface br2 was to be used to connect the virtualized guest nodes to. Its configuration looks like:

DEVICE=br2
TYPE=Bridge
BOOTPROTO=none
ONBOOT=yes
VLAN=yes
STP=yes
DELAY=5

Note that the bridge interface does not have its own IP address, or we would be connecting the Hypervisor directly to the Internet (and we don't want to, FWIW).

The management network interface though, which also needed to be bridged, does have its own IP address (in the management network, of course):

# cat ifcfg-bond0.666
DEVICE=bond0.666
BOOTPROTO=none
ONBOOT=yes
VLAN=yes
TYPE=Ethernet
BRIDGE=br666
# cat ifcfg-br666
DEVICE=br666
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
VLAN=yes
IPADDR=10.66.6.1
NETMASK=255.255.255.0
GATEWAY=10.66.6.254
NOZEROCONF=yes
STP=yes
DELAY=5

Now, we're done for the Linux Hypervisor part of the infrastructure. Lets get to the Cisco side of things!

All that a Cisco Catalyst 3560G really requires is that you:

• Create the VLANs you want to use. This used to be through enable mode's "vlan database" command, which is still a valid way of configuring the available VLANs, but will be (may already have been on newer equipment) deprecated for a series of commands in configure mode.
• Create the "channel-group" with the appropriate interfaces.
• Enable trunking on the channel-group interface.

Ergo, here we go (from enable mode):

conf t
  int range gi0/1-2
    no shut
    speed 1000
    duplex full
    switchport trunk encapsulation dot1q
    switchport mode trunk
    switchport trunk allowed vlan 2,666,667
    no switchport trunk native vlan
    description **some etherchannel**
    channel-group 1 mode active

You should now get an interface called Po1, with the following configuration:

show running-config interface Po1
Building configuration...

Current configuration : 142 bytes
!
interface Port-channel1
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 2,666,667
 switchport mode trunk
end

You should be good to go by now.

Software packaging... *sigh*

There's too many systems, and too many platforms required to build for each of those systems. Hence, I'm submitting the Debian packaging and build toolchain to Fedora.

I've started with packaging debhelper, debconf, po-debconf, dh-make and pbuilder. I've also submitted them to Fedora for review. If you have a spare moment, please chime in on the following bugzilla's:

• debhelper
• dh-make
• debconf
• po-debconf
• pbuilder

Next on my list are:

• debomatic
• cowbuilder
• debianutils
• debootstrap
• devscripts

Where <server> is my fileserver with a bunch of terabyte drives, my NFS clients get this message every once in a while (like every minute or so):

kernel: nfs server <server> not responding, still trying
kernel: nfs server <server> OK

Sometimes, in between these two messages, there's a couple of seconds. That can't be good ;-)

Most of the time, it's just the server being overloaded. Since most of my boxes at home are pimped yet slow desktop form-factor machines, this can happen. Google will show you the exact right hits on this phenomenon.

However, I found out there's another cause as well. Note how /etc/sysconfig/nfs has a setting, commented out by default, that says:

# Number of nfs server processes to be started.
# The default is 8.
#RPCNFSDCOUNT=8

Such I changed to 24, restarted NFS et voila! Yet another teeny weeny setting to tweak ;-)

MySQL remains a little bit of a mystery to me. I can only spend so much time on it, I only have so many database servers to play with and the necessity to tweak the performance comes from the fact the database server runs on a VMware virtualization platform with huge performance penalties, more so then sub-optimal performance of the database server or engine itself.

However, look at the following picture:

It shows you the amount of queries performed on a MySQL database server, per second average over a 5 minute poll interval. This is nice, and has shown us more then once that somewhere between 1000 and 1200 queries per second, the VMware guest is at its maximum capacity. This doesn't really help us to anticipate future usage of the database server though, as the graph just visualizes last day's usage. Luckily, Munin has a year graph too ;-)

Now, if you look at this graph, you can obviously see the trend go up and up and up. However, the trend is too vague. How steep is this usage going up exactly, and deriving from these statistics, how soon am I going to run into the guest's maximum capacity? One can make the trend go up very steep using this graph, but one can also think the usage is barely going up at all.

Admittedly, these numbers aren't all that high and the volume isn't all that huge. I would love for some admins and DBA's of some real databases to pop up and say their thing ;-)

Anyway, the workaround I came up with is to monitor and graph the Queries per Second Average. If graphed over a longer period of time (I've only recently started doing this), this does somewhat represent long-term usage trends. So, the following picture shows the Queries per Second Average (total number of queries over uptime):

Nothing really exciting in the day graph, is there? It's the year graph that, over time, I hope to get more valuable input from:

I'm hoping that this average of close to 500 now, is going to become an average of closer to 600 some time in the (near) future, and that I can accurately predict when it's going to hit 700 or so. Given that this number represents overall usage, including off-peak hours, its trend rising does mean something; it's just not directly related to the peak-hour usage.

Suppose that with an average close to 400 sub-optimal performance due to maxing out the system resources is very incidental. Suppose that with an average close to 500 this type of incident becomes more frequent (it's inevitable you will hit that point some time). Suppose you have this average in a graph going up and up you can then accurately (somewhat accurately) predict when the average is going to hit 600 -since the trend is what you're graphing here.

For those of you interested in doing this type of trending on their own environment (and let me know how it works out for you); here's the Munin plugin.

As some of you might know, I blogged to ask for a document management system and something that could deal with references. Extremely fast, I had a very useful responds that Zotero might be what I was looking for (many thanks!). Zotero is not only what I was looking for, but has so many options that I cannot stop myself telling how wonderful it is to anyone that seems to be listening. The reactions very from: “really nice program you have” to “WOW, that would have made my life so much easier” (yes that's right, no negative replies). I think that everyone (not only the people I meet) should have the opportunity to benefit from this Firefox plug-in. So, now I’m telling the world!

First of all: Zotero is ridiculously easy to work with. Besides, the documentation on http://www.zotero.org/ makes it possible for people that are “as good with computers as my mum” to fully understand and use all the facets of Zotero. Secondly, automatically obtaining information required for referencing by doi-number or ISBN saves a lot of time: instead of writing everything down yourself, you only need to check what Zotero come up with. If it is not to your liking you can easily adjust title (incl title case and lower case), author and so on. Furthermore, you can add notes and tags, categorize your documents, all to find them later on in only a few seconds. At the moment, I have about 400 items in my library and the search speed amazes me. I don’t need to make myself a cup of thee I only blink and before I have opened my eyes Zotero came up with a search list through all my data (that is: scientific articles in pdf format, notes and tags made in Zotero and documents written in OpenOffice)! And just to get back to the referencing part: you can export references in Word, OpenOffice, Gmail, LaTeX and possibly also other programs (I, however, don’t use those programs so I cannot give you feedback concerning that). Anyhow, this means that citation of references can in most cases simply be done by drag and drop. Furthermore, you can adjust the format in which the citation is made. And if that is not good enough: Zotero allows you to share your library with other people and allows you to access your library from different computers due to their server. Obviously the space on the server is limited and if I would want to use this I have to buy more space or work on only one computer since I just have too many files in my library. Anyway, the space they give you for free is more than enough to get a feeling of the value of this plug-in (I had almost 200 articles in my library before Zotero warned me that I didn’t have enough space on the server to sync my library).

A couple of things I haven’t tried yet and questions I have for the long run are:

-          Will I be able to use this program without major crashes up to at least the duration of my research (3 years)? Most likely the answer is yes, but I have to see that before I can believe it. I’m also not sure how to restore data if that would happen.

-          How does portable FireFox (combined with Zotero) work? In case I don’t have internet at hand... or can I simply open my browser and have zotero at hand? (I strangely enough never use my computer without internet. You may call me addicted.

-          What is best for exporting my references to LaTeX? Especially if I want to refer to many scientific articles? I have tried it with 1 article and that was ok. but I think that it can be done with less effort from my side (or maybe I’m just getting lazy seeing what is possible at date).

-          LyX , well that’s just one of those things I also have to try, I guess.

For now, I’m just extremely happy I found Zotero. Therefore, great many thanks to JFM who said that this might be it for me. You were right J.

If someone has any questions about this plug-in or how to deal with it and cannot find it on Zotero’s homepage, I’d be happy to help you!

A system administrator's job is always in between two -or most likely more then two- fires; one is the existing infrastructure, and the others consist of every other stakeholder in that infrastructure. The worst stakeholders are users, especially those that will burn you down without knowing what et al is involved, without any respect or perspective as to the position you are in as a sysadmin, juggling with more requirements to the infrastructure then just their personal needs. In the end though, ironically, they are also the reason why. Noted, most of my users are all sysadmins themselves, which makes any kind of disposition the more awkward, and any kind of decision be evaluated a lot differently by many more. In that regard, being a sysadmin in a company full of sysadmins is similar to participation in the Free Software world, if you will.

It came to that point. That point where a decision needed to be made on whether to scale up, flesh out and divide in order to conquer, or accept the downtime as a consequence of incidental shortage of resources. That time will come in most environments, and you can only prepare for it. Hence, I'm going to list a few tips on things to do, and things not to do, in order for you to -hopefully- be a little better prepared when you're in a similar position ;-)

First, I'll draw you up an overview picture of the environment in this case.

There's this web-server and it doesn't consume a lot of resources most of the time, and you feel confident it's going to make it through this week unharmed. It does Zarafa Webaccess, and ActiveSync through Z-Push. Both of these are heavily used by many users -think in terms of hundreds. It's a mission-critical environment to many, since groupware includes their email and calendaring, and however pitiful, this apparently still is the primary way for my colleagues to communicate.

Then, on Tuesday, during office hours, a peak in resource usage, and many users start complaining about dramatically decreased performance. Fair enough, it seems.

This web-server just so happens to be on the back-end Zarafa groupware server. Infrastructural architecture aside, tt was juggling it's available resources between the back-end and the front-end server. As a result, not only Zarafa Webaccess and ActiveSync (z-push) users were impacted, but our IMAP and Outlook users were as well -and it's the Outlook users (Account Managers, Senior Consultants, ...) that we think are truly mission-critical.

This is an incident, and it's, or so it seems, isolated. Incidents though, if I remember correctly, become problems when they occur two or more times. This situation I'm describing happened to us on Tuesday, had happened to us just one time before. The last occurrence was months ago, and we've been running this environment for about 3 or 4 years. This last incident months ago just so happened to be on the day of the release of the iPhone 3GS -a "funny" coincidence, probably. Either way, incidents, I think, have some retention to it. It's not like it's a let i++ kinda for loop that goes on and on and on, if you know what I mean.

Either way, I fixed it right on the spot, once more, by freeing up some resources. The incident did cause me to think about a different, more robust architecture though, but I wasn't in any hurry to implement such different environment/infrastructure architecture. I did some thinking, some research, I made some preparations, but I wasn't going to implement them quite yet.

Wednesday, everything is fine. Thursday, everything is fine. Friday, the very same situation happens again. Now, to me, that's a second strike, if not a third, and I'm going to take on this problem as a special project and make sure it is dealt with accordingly, my perspective being long-term sustainability and all that.

So, after a little study as to the exact cause, the proposed changes are:

• move Webaccess and ActiveSync to a different, dedicated web-server (virtualization makes that easy enough)
• position, in between external Outlook clients of the back-end mail-server, a reverse proxy (heavy stuff)
• position, in between external Zarafa Webaccess and ActiveSync (z-push) users, a reverse proxy (heavy stuff once more)

A lot of thought, again, went into long term sustainability. For one, I want to put front-end, public facing processing on a node positioned in the perimeter network, as much as I can. I want my perimeter network node to go down but never ever the backend server, if you will, by whatever means.

And so I did make those changes by the time the second occurrence of this incident was about to get the entire groupware environment to a grinding halt, as part of an emergency change, that very same Friday afternoon -the worst time to implement any kind of change.

It took me all about 5 seconds to merge and push it through our configuration management -with Puppet. Since no DNS entries had changed yet, no firewalls were forwarding to anything different yet, this change was non-intrusive. So far, the changes I made are basically an extra VirtualHost for an existing reverse proxy, and I created a new web-server.

The intrusive part of this emergency change, the part that makes me do this blog post, is still to come. Either way though, the fact that it was a non-intrusive change so far enabled me to test the implementation without a staging environment. Note that in the environment I work with right now, we only turn to a staging environment for god-awful intrusive changes (Zarafa upgrades, LDAP migrations, Disaster Recovery tests, Restores from Backup, etc.), since we have the top Senior experts on any technology on our payroll.

Anyway, as a result, I now had everything in place to flip the proverbial switches, without anyone noticing anything. Everything was tested (/etc/hosts FTW!), and so I switched the internal DNS entry for our Webaccess and ActiveSync over to the new web-server.

I shouldn't have.

I made a big, big mistake.

I assumed -which in itself very accuratly describes the fsckup-, that a host name of webmail.domain.tld was used for *webmail* only, and not, say, FTP sites or even just Outlook clients.

The new, internal web-server had no business reverse proxying Outlook clients to the back-end mail-server, yet more and more Outlook clients configured to use server webmail.domain.tld started attempting to connect to the new web-server.

The reverse proxy for external clients did have business reverse proxying such connections, but that is no back-end server in any way. The back-end web-server for webmail.domain.tld wasn't configured to do any kind of reverse proxying for Outlook clients, and justifiably so. It had been configured to perform any of the tasks that involve *webmail*, and that alone.

Luckily, I had the configuration in place on that reverse proxy I mentioned. Copy, paste, commit, push, pull, apply, run, done. No matter what the IP address for webmail.domain.tld was internally, one would either end up with the back-end mail-server, or the new (back-end) web-server that, for the time being, also reverse proxied the Outlook connection to the back-end mail-server. That too was solved pretty quickly, but for the users to understand why or how exactly they had no email or calendaring for 5 seconds... different story.

The Users

Users will argue "it used to work", when also arguing to not understand why "it doesn't work right now". Well, you know, sometimes changes are necessary in order for anything to continue to work. Sometimes the implementation of such changes impacts you as a user, but only serves other users. Sometimes, hopefully even less frequently, those other users justify the change to be an emergency change, hitting you right in the face during production hours. I'm sorry, you were saying?

Then, and this is especially the case with me, I require a feedback cycle. I myself am not an Outlook nor ActiveSync user, so I need someone else to tell me what their needs and expectations are. Not in terms of "it doesn't work anymore", but accompanied with the details that allow me to pin-point the exact cause and then solve it.

Some of these causes can be found in arbitrary ActiveSync software not accepting a wildcard-certificate as valid, even though it is in fact valid for the rest of the world. This kind of software would check whether the certificate CN is exactly equal to the fully qualified domain name of whatever you're hitting, but won't expand matching characters. That level of detail is beyond many sysadmins, let alone users. While neither party is at fault per se, users do look in the direction of a sysadmin "to solve the problem" because "it doesn't work".

"Right. Thanks. Can you check a couple of things for me, like, view the certificate you are getting?" I had my suspicions, but I'm not familiar with whether such software will even allow you to view the certificate it's rejecting.

"I don't understand how this works and I have no time to Google all day as I have a job to do... Can't you just solve it?" is a commonly heard answer.

"Right. So, well, no, I can't just go around and attempt to fix arbitrary things in arbitrary ways. I didn't ask you to Google all day and solve the problem by yourself, I asked you to provide that feedback in order for me to be able to confirm my suspicions as to the cause of your problems."

The Settings

I've always understood reverse proxying Outlook Anywhere is a challenge, given the sheer volume and the way it sets up a connection and expects that connection to be around for a relatively long time. Hence, I'm going to share some of my configuration, with comments in-line.

First, the reverse proxy:

# Need a specific IP address other then the one used for all the other reverse proxied
# websites, unless you use the very same certificates for all sites (no nss here yet).
<VirtualHost 10.0.0.19:443 10.0.0.98:443>
    ServerAdmin kc-ux@ogd.nl
    # Some Outlook clients may still have been configured with 'webmail.ogd.nl'
    ServerName webmail.ogd.nl
    # The new configuration for Outlook clients is to use the following DNS names
    ServerAlias outlook-anywhere.ogd.nl outlookanywhere.ogd.nl
    DocumentRoot /var/www/html/

    ErrorLog logs/webmail.ogd.nl-error_log
    CustomLog logs/webmail.ogd.nl-access_log combined

    SSLEngine on
    SSLProxyEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
    # Note that this can be a wildcard certificate as far as Outlook clients is concerned, but
    # not for ActiveSync clients.
    SSLCertificateFile /etc/pki/tls/certs/webmail.ogd.nl.cert
    SSLCertificateKeyFile /etc/pki/tls/private/webmail.ogd.nl.key
    SSLCACertificateFile /etc/pki/tls/certs/webmail.ogd.nl.ca.cert

    KeepAlive On
    # Crank up the KeepAliveTimeout
    KeepAliveTimeout 300
    # Prevent the connection from ever being reset unexpectedly.
    MaxKeepAliveRequests 0

    # Prevent the logs from filling up.
    SecRuleRemoveById 960010
    SecRuleRemoveById 960012
    SecRuleRemoveById 960013
    SecRuleRemoveById 960015
    SecRuleRemoveById 960032
    SecRuleRemoveById 960902
    SecRuleRemoveById 970902
    SecRuleRemoveById 970903

    ProxyRequests Off

    # This is where we actually create the reverse proxy. Note that the parameters
    # to ProxyPass make it a keepalive enabled connection, and we keep retrying in case
    # of errors, instead of throwing the error back at the client.

    # One for the Zarafa back-end mail-server
    ProxyPass /zarafa https://backend-mailserver.ogd.nl:237/zarafa keepalive=on retry=0
    ProxyPassReverse /zarafa https://backend-mailserver.ogd.nl:237/zarafa

    # And one for the webmail back-end web-server
    ProxyPass / https://webmail.ogd.nl/ keepalive=on retry=0
    ProxyPassReverse / https://webmail.ogd.nl/

</VirtualHost>

The Impact/Implications

Given the new situation, we now have the following situation;

• If anything is hammered (from the Internet) to the point where it is going down, it'll be a simple reverse proxy,
• If anything from the Internet is invalid in any way, we can stop it as close to the source as possible, so it'll never end up on our back-end Infrastructure,
• No webmail or ActiveSync client is going to cause the web-server to need so many resources it's getting in the way of the resources required by the back-end Zarafa mail-server, making the overall environment just that little more robust,
• We finally got our Outlook clients to no longer use "webmail.domain.tld" in their configuration,
• Problems with webmail and ActiveSync (timeouts and connections being reset) have now been resolved merely through the feedback cycle triggered by the sometimes disruptful implementation of these changes.

The Tips

• Assume that your predecessors have not had the same perspective you have now, and as a result may very well have caused Outlook clients to connect to the Zarafa server using a hostname of www.yourcompany.com
• Profile your environment's resource usage as much as you can. The ability to anticipate a shortage of resources works magically well. For example, if you know one incident with a 1.000 queries per second made your MySQL database server go haywire, monitor the trend of Average Queries per Second, which in healthy environments should only go up and up over longer periods of time, allowing you to anticipate on whether and when any limitations are going to be met.
• Use Configuration Management through a Source Code Management system. Not only does it allow you to quickly revert changes applied, it is also your audit trail, a large part of your documentation, and the means to verify overall system consistency. Two years from now, you may not work for your company any longer. Somebody else is going to have to figure out why the "keepalive=on retry=0" is in the configuration for webmail.domain.tld, and why it is so darn essential. This also goes to using Consistent Changesets (no changeset breaks the overall configuration and all changesets can be reverted by themselves), as well as proper Commit messages (the documentation of the change is in the change itself).
• Try to relieve as much stress as possible and don't pick up the phone when you're expecting a user to be on the other end. Make the communication with users somebody else's priority number 1, so that you can focus on what this arbitrary ActiveSync client might actually be doing totally unexpectedly, as opposed to getting even more stressed out over "it doesn't work" types of other people's frustrations.
• Move forward. Move backwards a little in order to be able to move forward at all, if necessary. Nobody (not you, not the other sysadmins, not your manager, nor the users) is helped in any way by fast hot-fixing or dirty patch-work if a situation can happen again and again. Rather, consider whether there's a (possibly) more sustainable, true way forward, even if that includes taking a small step backwards for a little while, and then implement that instead. That is, of course, unless you can hot-fix the situation temporarily, and await with the actual implementation of the permanent change 'till after production hours.

Once again another interesting and informative day at the ELN Workshop in Amsterdam.

The topics of discussion in the final day were compliance, risk, patents and industry development. Different organizations made presentations in the morning regarding licensing compliance and how it pertains to the internal part of the organization, as well as, through to distribution. Intra-organization compliance systems were discussed in short and the various mediums that could be used in order to insure that the vendor(s) is complaint. Most all were in agreement that it was vital for an organization to integrate a compliance policy into the supply chain. Different measures on how this compliance policy could be constructed were discussed also.

The second topic discussed was Risk and how to evaluate and contain it. A balanced discussion on risk from both the legal and business sides of things gave a realistic case view on what open source companies are facing in the current and expanding market. Members of the ELN have also created and are developing a Risk Grid to address procurement of F/OSS and it's function within the supply chain. This grid aims to assist professionals in the field to regulate their actions with their suppliers and buyers by allocating  risk to the appropriate actor within the supply chain.

A patent panel also discussed briefly about the pro's and con's of patents within the organization and argued how these patents can hinder, as well as, support innovation within an organization. The affects of patents on licensing and the various complications that arise were also discussed, but agreed that another workshop will be set aside to discuss patents in more detail.

Finally, Future developments were discussed and their impacts on the F/OSS industry. Technological innovations and shifts were discussed and weighed against the developments of the F/OSS movement in the past. We are seeing quite a shift from software as a product to more of a service driven industry with such developments as the cloud and embedded MID's. Awareness must be made now that assists professionals in evaluating legal risk in accordance to licensing and procurement, through educating professionals on the importance of compliance and how to properly implement a compliance policy within their respective organizations. Moreover, it is generally agreed that there must be more interaction with developers/engineers and management/legal counsel to work together and build a "best case" practice that incorporates the developer/engineers issues and needs together with the organizations compliance. The most sound method of accomplishing this is through working together to reach our common goal with special emphasis on proper education and documentation.

I would like to thank the Free Software Foundation Europe and everyone who attended the ELN workshop and especially thank them for allowing me to participate in this closed discussion. It definitely opened  my eyes to topics in which I have only scratched the surface on. The complexities of the topics discussed were just another confirmation that we need to work even harder together to meet the ever growing issues faced by all levels of the F/OSS movement. I look forward to working together with you at the ELN and sharing the knowledge and contributions from the unique position I hold between industry and community.

The fastes way to get my boyfriend respond is via this medium... Emails and phonecalls don't work. Letters, well... don't even bother to send those to him. Luckily, there is a way to get him involved in my (computer)life. I just had a phonecall and he was really helpful explaining me what is and is not an error I should worry about. @Kanarip: I'll thank you later (K).

At the moment I wish I was more eager to learn how to deal with computers in a more advanced way. I think I'm more advanced than an ordinary user, but well... not good enough yet to understand why my computer crashing every freaking day. It is annoying. Anyhow, what sort of information should I have trying to solve this is my question?

Few things I noticed:
- First crash occurred 28th of February (can I somehow obtain information about what sort of software I installed that day or just before that day?)
- I'm installing a lot of programs lately. These were: Zotero, XdrawChem, WinDrawChem, PSPP, Chemical Calculator, SpecViewer. Furthermore, I keep all my software up to date.
- My memory is used to the fullest almost constantly. (for example: 44% in use by programs and 55% in use as cache).
- My network is often down (damn wireless)
- I am almost always on the internet when I work on my laptop.
- I've taken a screen-shot of the errors I have had lately...

Especially the attachment screenshot.jpg worried me (though it was gone after a restart). I hope my laptop will survive my PhD, which I just started... Can anyone say that it is not that bad and that it is only inconvenient at times? My laptop has his maintenance soon enough (2 weeks from now, at least if my boyfriend allows me to use him for this).

AttachmentSize

allert.png152.77 KB
error2.png109.47 KB
error3.png74.24 KB
error3b.png117.87 KB
latesterror.png1.41 MB
Screenshot.png194.94 KB

The FUDCon in Toronto brought me, besides some really nice memories, a laptop referred to as OLPC. Last night I had a dinner party. I saw this an opportunity to explain them about the project and let the 30year old kids play with it. The first thing we noticed was that we are brainwashed by normal operating systems. We don't know how you can get to the main menu after you entered a program. However, we were all very enthusiastic about the project and we are sure that the kids in the age range well below us are more flexible and learn it quickly via trail and error. One of the guys explained that a similar project was run in Portugal. I'm googling and see that the XO is the big winner during a trail in Portugal. Somehow I cannot suppress this feeling of being proud, whereas I honestly haven't done a thing for the project. Only... well, let's just say I'm a good person to spread the news on open source software. Since I could explain why the kids can truly benefit from this type of laptop more than software that isn't open source, one of my friends saw the light: "SOOOO COOOOOOL". And indeed it is. It is good to develop yourself via educational exploration and have the possibility to look at what others have written to develop the software (you may be using). This way of learning most likely doesn't work for all of us, but it is an under-appreciated manner of learning if you'd ask me (in the Netherlands that is).

I was invited by the FSF Europe to take part in the ELN Workshop in Amsterdam for a discussion between industry professionals, lawyers and engineers with how licensing impacts their respective organizations. The workshop presentations consisted of highly recognized F/OSS industry professionals from Europe and the US. The first day of the event was well organized and quite refreshing since there were many open minded discussions that are leading up to implementation and management of governance in F/OSS projects, as well as, working towards a collaborative grid/matrix for developers/engineers to use when choosing a license to release their code under with a view to compliance.

Moreover, it was nice to hear presentations of up to date casework of GPL violations, casework from professionals about industry adoption of F/OSS and how the OSS industry is gathering momentum and legitimacy at lightning speeds ;-) Also, it was nice to hear the current information about the recent legal developments on a country by country level in Europe and also the recent development at the EU parliament level. Thank you especially to the small, but hard working group of lobbyists that attended!!

The organizers and presenters did a great job of keeping theoretical concepts to a minimal and successfully explained their issues and topics in layman terms. I thank you all for your openness and collaboration and must say I have never experienced an event quite like this with legal council and industry professionals that was so open, free flowing and transparent without having the feeling or worry of those luring second agendas.

I look very much forward to continuing our discussion and developments on Day 2.

Another random day at the office: Can you imagine this happens to all of your users?

Solution: Tell the domain member (client machine) that the active directory domain name space is in the trusted zone. I mean, wtf... ;-)

I think I'm going to have to give you a summary of the second and third day of the Bacula Systems Administration Course, part I that I participated in late March, since apparently I've been too busy with different stuff to even finish the blog post I was originally supposed to do on the second and third day.

The second day was a more hands-on day where we practiced with backup and restores including a full meltdown of the Director and Catalog server; having the backup server go down and not be recoverable other then from your previous backup gets you in a catch-22 kind of situation. It's not at all that hard though, since Bacula specifically addresses this type of situation in its documentation, taking you through the process step-by-step, with a grant total of not even 10 steps.

You have to have done it once, even if it's just to experience this situation once before it just happens to you in real life and you don't know what to do.

Since the day was full of exercises and practicing, it was an exhausting day altogether. Though again on the second day, much like the first day, I didn't run into any trouble at all, so I could very smoothly work through the tasks assigned ;-)

Later on in the evening, we were all invited to dinner in a restaurant nearby, where -of course- we enjoyed a couple of beers and some good stories.

On day three of the course, we got to relax a little more. I think the exact words when we came in in the morning were "Welcome, glad you survived the second day."

Various types of backup strategies also were brought to our attention. A couple of case studies showed us exactly what was on Kern's mind while he discussed certain types of backup strategies along with his choice for Pools, Volumes and other settings, and his calculations on the number of Jobs per Volume, the maximum number of Volumes, and retention periods.

Although that too isn't all that difficult (it's merely a little too abstract if you will), it's great to hear the grand master himself explaining how it works.

This is one of those blog posts that has been sitting in my Documents/Blog folder, but was never posted :-(

The larger the MySQL database(s), the less likely you are going to backup MySQL using mysqldump. With larger databases, for one, the contents of the database may change while you're exporting the database. Luckily, there's a --single-transaction option to mysqldump, but again in the case of larger MySQL databases, this directly impacts production.

I've seen many posts on using LVM snapshotting to backup MySQL, but never a concise one; I've had to figure out on my own what to do exactly in order for a restore of such backup to be successful.

Being as it may, I wanted to share it. Here's the steps I take;

1. Remove the old database backup (this is a storage limitation on my side),
2. Flush everything MySQL may have cached to disk,
3. Lock MySQL,
4. Create a LVM snapshot,
5. Unlock MySQL,
6. Sync the contents of the snapshot anywhere you like.

Now, between step 2 and 4 is the trick. It seems that with InnoDB, you can only tell MySQL to flush to disk & lock in a single session, since MySQL is unlocked automatically when you end the session. So, here's the trick I use:

(
    date > /var/log/backup-mysql.log && \
    echo "FLUSH TABLES WITH READ LOCK;" && \
    lvcreate --size 10G --snapshot --name lv_mysql_snap /dev/vg_db01/lv_mysql >> /var/log/backup-mysql.log 2>&1 && \
    echo "SHOW MASTER STATUS;" && \
    echo "UNLOCK TABLES;" && \
    date >> /var/log/backup-mysql.log && \
    echo "\quit" \
) | mysql >> /var/log/backup-mysql.log 2>&1

Creating a subshell with the output of that subshell piped through MySQL keeps the session open while creating a snapshot of the logical volume. This, as opposed to many other brain-farts I have, is actually tested, and we use it at my company to backup our MySQL servers -and restore the backup to a testing environment every once in a while.

The flushing, locking and snapshot creation (in our case) takes about 3ms.

I've been working on a backup strategy for our Zarafa groupware environment, using Bacula backup & restore software.

Since we have a Zarafa support subscription at Operator Groep Delft, we have a utility called zarafa-backup. Regrettably, this utility only allows you to create a Full or Incremental backup. Here's how the utility works:

• If you run zarafa-backup for the first time, it'll create a .data.zbk file per mailbox, with a .index.zbk file indicating the contents of this one .data.zbk file.
• If you run zarafa-backup the second time, provided the .index.zbk file is still around, it'll create a .data.zbk.1 file with only the new content of the mailbox, and update the index file.
• If you run zarafa-backup the third time, provided the .index.zbk file is still around, it'll create a .data.zbk.2 file with only the new content of the mailbox, and update the index file.
• etc, etc, etc.
• If you want to create a Full backup at some point, destroy the .index.zbk file and run zarafa-backup - it'll think it's the first time you run the utility since there's no existing .index.zbk file for the mailbox.

This is incompatible with a decent backup strategy. Creating a full backup in a larger environment is rather expensive, so you would want one full, say, every so many months or so. With small incrementals stacking over a longer period of time though (31 days in a month), you can imagine that restoring yesterday's mailbox content can quickly become one full backup and a million-and-one incremental backups. You would need to restore the yesterday's .index.zbk file along with all .data.zbk files from your backup suite. The incremental backup may be spanned over several locations, though, and your expiration policy or storage limitations may require you to expunge data from tape or disk after a couple of months. At some point, restoring just becomes such a huge effort it outweighs the downside of creating a new full backup.

Let's state some numbers, just for fun, and for further reference:

• 1.000 mailboxes
• 1GB mailbox quota

Both attachments and message content (text) count towards the quota, but in the most extreme scenario, all mailboxes are full. The theoretical maximum volume to backup would be about 1TB.

The actual volume will always be lower; Even though some users have a tendency to complain they run out of quota, it turns out that, given a progress bar counting upwards to their maximum, and the simple fact that they will not receive any more email when they run out of quota, causes them to clean out their mailboxes. That is, if nobody ever gives in to requests to increase the quota, and you automaticalle expire everything that is irrelevant (calendar entries in the '90s) The average usage currently is about 15%. The total footprint as you can see is a lot less then the theoretical maximum of 1TB.

However, this "15%" (or ~150GB) will take a little over 24 hours to export through zarafa-backup. As running the utility impacts the overall performance of the groupware environment (the backend zarafa-server process is heavily utilized and zarafa-backup in that sense behaves like a big, fat client), you can see how this (if you expect higher average usage) might result in a policy that requires you to create a full backup somewhere between Christmas and Newyears. Other reasons for the volume to increase beyond what is reasonable to create a full backup for on a regular basis may include company growth, more temporary hires, what-have-you.

Ergo, a proper backup strategy is required. Something that let's you restore a mailbox with yesterday's contents, but without the hassle of getting 364 incrementals. Somehow, you would need to preserve the .index.zbk files along with the .data.zbk files on a daily, weekly, monthly, bi-monthly or even yearly basis, so that you can use those to backup "what has been changed since the last full", so that you can expire all the little incrementals of "what had been changed per day the last month".

So, let's assume you have the following backup requirements:

1. Restore mailboxes per month for 6 months
2. Restore mailboxes per week for 1 month
3. Restore mailboxes per day for 3 weeks

This means you can restore any mailbox day-by-day for the past three weeks, week-by-week for the past month, and month-by-month for the past 6 months. Presumably, your Bacula Schedule resource would look like:

Schedule {
    Name = "Zarafa-Monthly"
    # A full backup starts every first weekend and takes little
    # over 24 hours to complete.
    Run = Level=Full 1st Saturday at 11:59
    # During the work-week, we backup every day
    Run = Level=Incremental Monday-Friday at 23:05
    # In weekends other then the first, ...
    Run = Level=Differential 2nd-5th Saturday at 23:05
}

This starts a full backup every month, which you may want to decrease. I for one specifically don't want to do that. I'm saying: "If the backup impacts production too much, give me more power." As you can see I start the backup on Saturday's though, so it has more then 44 hours to complete before it's Monday morning 8am. Admittedly, our environment is mostly idle during the weekend except for a few ActiveSync clients. Here's the FileSet:

File Set {
    Name = "Zarafa-Bricklevel"
    Include {
        Options {
            Signature = SHA1
        }

        # No trailing slash!
        File = /var/lib/zarafa/bricklevel
    }
}

As you can see, I'm only interested in /var/lib/zarafa/bricklevel/. Then, last but not least, the most interesting part of the configuration; the Job and Pools:

Job {
    Name = "Zarafa-Backup"
    Type = Backup
    Level = Incremental
    Client = "mail01.ogd.nl"
    File Set = "Zarafa-Bricklevel"
    Schedule = "Zarafa-Monthly"
    Storage = "Zarafa-File"
    Messages = "Standard"
    Pool = "Default"
    Full Backup Pool = "Zarafa-Full-Pool"
    Incremental Backup Pool = "Zarafa-Inc-Pool"
    Differental Backup Pool = "Zarafa-Diff-Pool"
    Write Bootstrap = "/var/lib/bacula/bootstraps/Zarafa-Backup.bsr"

    Run Script {
        Runs When = Before
        Runs on Client = Yes
        Job Fail on Error = Yes
        Command = "/usr/local/sbin/zarafa-backup %l --before"
    }

    Run Script {
        Runs When = After
        Runs on Client = Yes
        Runs on Failure = No
        Command = "/usr/local/sbin/zarafa-backup %l --after"
    }
}

Pool {
    Name = "Zarafa-Full-Pool"
    Pool Type = Backup
    Recycle = Yes
    AutoPrune = Yes
    Volume Retention = 6 months
    Maximum Volume Jobs = 1
    Label Format = "Zarafa-Full-"
    Maximum Volumes = 9
}

Pool {
    Name = "Zarafa-Diff-Pool"
    Pool Type = Backup
    Recycle = Yes
    AutoPrune = Yes
    Volume Retention = 40 days
    Maximum Volume Jobs = 1
    Label Format = "Zarafa-Diff-"
    Maximum Volumes = 9
}

Pool {
    Name = "Zarafa-Inc-Pool"
    Pool Type = Backup
    Recycle = Yes
    AutoPrune = Yes
    # Keep a Volume for 3 weeks minimum
    Volume Retention = 21 days
    # Rotate volume every week, whether failed jobs are in
    # there or not.
    Maximum Volume Jobs = 5
    Label Format = "Zarafa-Inc-"
    # 5 jobs (e.g. one week) in one volume;
    # 3 volumes per 21 days
    # + a little leeway
    Maximum Volumes = 6
}

You may have noticed the RunScripts I'm using are something custom. That's right! Please see the attached zarafa-backup script.

NOTE: This is just what I whipped up just now, I haven't truly tested it. Please be careful when implementing it in your environment ;-)

NOTE^2: zarafa-backup is a perfect utility to restore only part of a mailbox, too. You can choose categories of items such as only a user's Email, Calendar, Tasks or Contacts. This is why you need to consider using zarafa-backup, as opposed to a snapshot-and-sync type of backup.

Today was the first day of the Administration Course 1 for Bacula, by Bacula Systems. I am here sponsored by the distributor in the Netherlands, Amaziq Source, as a "thank you" for creating two sales leads which where followed up by Bacula Systems and Amaziq Source, and solidified -with two of my former Operator Groep Delft customers.

One of my former customers was using Commvault, which apparently turns out to be a big dissappointment when you decide to upgrade your backup environment. It seems Commvault only sells you the new, expensive license if you also purchase a number of days of expensive consultancy along with it. At these times, I can only grin, from ear to ear.

The other customer had decided on its own to move away from their existing Symantec backup suite, partly because it Just Didn't WorkTM, and partly because of the attractiveness of Open Source. Either way, when I worked at the customer site -on a job that had really nothing to do with backup- we were informally discussing the backup solution they were looking at at that time, and they let me know it was Zmanda. When I told them Zmanda is not(1) Open Source(2), even though they love to pretend to be, I got to explain why not (just try to download the source and you'll see for yourself) and most importantly, I got to mention an alternative. Enter Bacula, which they've now purchased a support subscription for with Bacula Systems.

Back to the original topic of this blog post, the course itself, to me at least didn't seem very advanced, since I've been occasionally operating Bacula for a number of years. I have to say that was just the first day though, and tomorrow is promosing to be way more challenging (I think the actual word used was "boggling"). I realize though the average system administrator used to any other type of Backup & Recovery software may require this course to get used to the different terminology used within Bacula, as it is not your average click-and-pray type of program.

Even in my case though, the course helps in creating a correct understanding of the features and configuration of Bacula, and as such helps me to futher increase my knowledge of Bacula. Even more so, since this course is being mentored by the original author of Bacula; Kern Sibbald. This man, for whom I've got much respect, has been working at Bacula for over 8 years already. It's one of the main reason why I'm in this course in lovely Switzerland; I might as well have followed the course in the Netherlands but I wanted to meet more of the Bacula people, Mr. Sibbald in particular.

We learned about the architecture of Bacula, which of course I thought I was pretty much familiar with already, but I did get to hear some more, new details on how in greater environments, multiple Directors, Catalogs and Storage Daemons can be aligned and balanced out -and some more details I all wrote down in the handout.

After the initial "overview", we went into the feature and configuration details of Bacula, merely scratching the surface of each configuration directive of each component. I got some things that were on my mind cleared up, as I've been working with all kinds of different backup suites throughout my carreer with Operator Groep Delft... ;-)

I can't wait 'till tomorrow, but first: dinner and drinks