Backups for kolab2

From Kolab Wiki

Jump to: navigation, search


Contents

Components

IMAP store backups (cyrus)

Please visit on the this very good page on the cyrus wiki.

Online backups can be made with rsync, although if you have high traffic, you may run in to some problems. One method that has been suggested is to run an online rsync, then immediately afterwards, run it with cyrus offline. This way you will have a snapshot similar to an LVM snapshot backup.

Don't forget to also add

/kolab/bin/ctl_mboxlist -d > mailboxlist.txt

to your backup, this will dump the list of mailboxes to a text file.

Note: This assumes that you are running the backup as user kolab-r. If you run the backup as root (e.g. from a cron job), you can use the following version:

su - kolab-r -c "/kolab/bin/ctl_mboxlist -d" > mailboxlist.txt 

If your mailboxes.db ever gets damaged, it can be restored with (using "su" if necessary):

/kolab/bin/ctl_mboxlist -u < mailboxlist.txt

You should also backup the annotation database, which indicates the type of the contents of each folder (such as calendar, or contacts). The annotation database is stored at /kolab/var/imapd/annotations.db.

You can run the annotation backup like this:

/kolab/bin/cvt_cyrusdb -C /kolab/etc/imapd/imapd.conf  /kolab/var/imapd/annotations.db skiplist backup.db flat

The CVS tree also contains a script and some documentation for checking this file for corruption, and backing it up. The script can be run as a cron job and notifies the administrator if the database is corrupt. (Or you could simply back the file up by copying it).

IMAP store recovery (cyrus)

Restoring emails is really simple: User's mail store is in /kolab/var/imapd/spool/domain/%yourdomain%/user/%username% (replace %username% with the username only, not username@domain). You can copy mail directly from the backup to this mail store directory, then run /kolab/bin/cyrreconstruct as the kolab-r user.

This, however, requires to have a working mailbox list of your mail store. You should have dumped it beforehand, when you did the backup, with the command

/kolab/bin/ctl_mboxlist -d > mailboxlist.txt

If you haven't, you can generate this file manually. Just go and assemble a mailbox list file with the following structure. Note: Make sure you use tabs and not blanks between the records!

yourdomain.de!user.username    default username@yourdomain.de lrswipcda
yourdomain.de!user.username.Calendar   default username@yourdomain.de lrswipcda
yourdomain.de!user.username.Contacts   default username@yourdomain.de lrswipcda
yourdomain.de!user.username.Journal    default username@yourdomain.de lrswipcda
yourdomain.de!user.username.Notes      default username@yourdomain.de lrswipcda
yourdomain.de!user.username.Sent       default username@yourdomain.de lrswipcda
yourdomain.de!user.username.Tasks      default username@yourdomain.de lrswipcda
yourdomain.de!user.username.whatever     default username@yourdomain.de lrswipcda
..........

When reading this file with the following command, the mailboxes.db will be reconstucted:

/kolab/bin/ctl_mboxlist -u < mailboxlist.txt 

Afterwards, you can go ahead and read the individual email back into cyrus. You can either specify a certain user (if you are restoring on a per-user basis) or do it without any parameters. In the latter case, all mailbox paths will be checked for emails:

kolab-r@mykolabhost$ /kolab/bin/cyrreconstruct -rf user/user@mydomain.com
discovered mydomain.com!user.username.folder1
discovered mydomain.com!user.username.folder2
user/username/folder1@mydomain.com
user/username/folder2@mydomain.com

To reconstruct a particular folder, use a format like:

kolab-r@mykolabhost$ /kolab/bin/cyrreconstruct -rf user/user/folder/subfolder@mydomain.com

If you run cyrreconstruct as another user than kolab-r, you will get strange errors like "Operating System Error" etc. Also, logging in to cyradm with user=manager and then running reconstruct user/username@mydomain.com will not work, it seems like because you are logged in as manager, not kolab-r. If you are not restoring emails from a prior kolab installation but from any other cyrus imap store, or in case you are transferring folders across users, make sure that you remove the following files from each folder: SEE NOTE

cyrus.cache
cyrus.header
cyrus.index

You can remove them by running this:

 find /kolab/var/imapd/spool/domain/{yourdomain}/user/{nameofimporteduser} -name "cyrus.*" -exec rm '{}' \;

NOTE: This may not be necessary if you run cyrreconstruct -rfx (note the -x), this will recreate the metadata (ie user etc). Doing this will make all the mail in the mailboxes appear as NEW. From the man page:

   -x
   When processing a mailbox which is not in the mailbox list (e.g. via the -p or -f options),
   do not import the metadata from the mailbox, instead create it anew (this specifically affects
   atleast the mailbox's seen stat unique identifier, user flags, and ACL).

ANOTHER NOTE: If you are transferring between users (for example, I needed to put all one users mail into another's mailbox), the permissions on all the mailboxes will still be for the old user. You may get errors like "Mailbox does not exist". This can be sorted out by using the sam command in cyradm:

 /kolab/bin/cyradm -U manager localhost
 IMAP PASSWORD: 
 localhost> lam *newuser*
 ## This should show a list of all the mailboxes of the new user, if there are no overlaps with others, run the next command ##
 localhost> sam *newuser* newuser@mydomain.com lrswipcda
 ## This should show a list of the mailboxes it applies the acl to ##

See IMAP ACL for more details of ACLs After doing this, log out of cyradm (quit) and log into the mail client, you may need to re-subscribe to the mail folders in the client.

IMAP folder recovery from backup (cyrus)

If a user has accidentally deleted a folder (or some mails), here's a procedure to restore the data:

Using cyradm, create a restore folder:

 /kolab/bin/cyradm --user manager localhost
   cm user/UserName/restore@domain.name
   quit

Then restore the files and subdirectories from the backup into the restore folder. Next, make sure that all files and folders are owned by user/group kolab-r/kolab-r

 chown -R kolab-r:kolab-r restore

Now run cyrreconstruct as user kolab-r:

 su - kolab-r -c "/kolab/bin/cyrreconstruct -r -f user/UserName/restore*@domain.name"

Afterwards, do not forget to subscribe to the restore folder, and sub-folders. The -r is a little broken with cyrus version before 2.3.12 this is why the star "*" in the mailbox is useful. cyrreconstruct is using one of the cyrus.* (I don't know which one, sorry) to know if it is a mailbox and then to create the correct entry into mailboxes.db, then at least verify you have empty cyrus.* files in each mailbox you want to recover.


Once all data is restored, it is strongly recommended to let Cyrus recalculate the quotas (if used), as "manually" added mail is not taken into consideration for quotas. To recalculate the quotas for all users, use

 su - kolab-r -c "/kolab/bin/cyrquota -f"

Unfortunately, there is no easy way to recalculate the quota for just one user.

LDAP data backups (openldap)

Use the "slapcat" utility as follows:

/kolab/sbin/slapcat -l /path/to/file/kolab_ldap_db.ldif

The .ldif file is a normal text file that can be backed up using TAR or any equivalent utility.

This LDIF file will still work even if the database format or software changes.

Also take a copy of the following files:

cp /kolab/var/openldap/openldap-data/DB_CONFIG /path/to/file/
cp /kolab/etc/openldap/slapd.conf /path/to/file/
cp /kolab/etc/openldap/slapd.access /path/to/file/

If you cannot find these files on the system you are restoring on, then use theses copies. But always prefer the files existing on the system you are restoring to. Especially if you are migrating to a newer version. (even if these files are mostly 100% compatible)

To import/export LDAP data, you can use also: phpLDAPadmin , LDAP account manager

Kolab Configuration Files

Make sure you grab a copy of files in /kolab/etc

Postfix backups

If you have edited any of the kolab templates, these will be under /kolab/etc, so a recursive backup of this should be fine (see "Kolab Configuration Files")

Spamassassin backups

The spamassassin bayesian databases (spamassassin's store of learned data - what is spam and what is not) can be backed up quite simply to a text file, just run

/kolab/bin/sa-learn --backup > backup.txt

this will back up all the learned data to a text file. To restore, just run

/kolab/bin/sa-learn --restore backup.txt

You can use this for moving it to a new server as well. This might not be a good idea to "accelerate" the learning of a new spamassassin, there are some differences which may make learning from scratch more effective, especially if there are different domains/ips involved.

Backup script

This tries to condense the above information into short instruction on how to create a generic backup script. There is an example on Kolab_backup.sh.

The archive location

First of all you will need a storage location for your backup. We are going to create one directory with several sub directories that match the different data types here:

mkdir backup
mkdir backup/{etc,imapd,ldap,spam}

Data backup

Stop all services for a short while:

/kolab/bin/openpkg rc all stop

Backup IMAP first:

rsync -avz /kolab/var/imapd/ backup/imapd/
su - kolab-r -c "/kolab/bin/ctl_mboxlist -d" > backup/imapd/mailboxlist.txt 

Backup the configuration data:

rsync -avz /kolab/etc/kolab/ backup/etc/

Now store the LDAP information:

/kolab/sbin/slapcat -l backup/ldap/kolab-ldap.ldif

And if you are using the spamassassin bayes filtering you should also store the corresponding database:

/kolab/bin/sa-learn --backup > backup/spam/sa-bayes-db.txt

Time to restart all services:

/kolab/bin/openpkg rc all start

Playback

Now we will restore the data from our backup. Again all services need to be stopped first:

/kolab/bin/openpkg rc all stop

In order to restore IMAP you simply need to sync the imapd folder back:

rsync -avz backup/imapd/ /kolab/var/imapd/

The same needs to be done with the configuration data:

rsync -avz backup/etc/ /kolab/etc/kolab/

(If you are restoring to a new version of Kolab, however, you should use diff on any files you have changed and manually update the files in etc instead of using rsync.)

LDAP recovery should be simple too:

cp /kolab/var/openldap/openldap-data/DB_CONFIG /tmp/
rm -rf /kolab/var/openldap/openldap-data/*
cp /tmp/DB_CONFIG /kolab/var/openldap/openldap-data/
/kolab/sbin/slapadd -l backup/ldap/kolab-ldap.ldif
chown -R kolab:kolab /kolab/var/openldap/openldap-data

If you are restoring to a new version of Kolab, you will need to set the php_pw in /kolab/etc/kolab/kolab.conf to match that which is in the backed-up file.

If you use spam bayes filtering you restore the database running:

/kolab/bin/sa-learn --restore backup/spam/sa-bayes-db.txt

Time to restart all services:

/kolab/bin/openpkg rc all start
Personal tools