Backing up a Cyrus server

What is the best way to back up a Cyrus server, anyway? I can't claim to be authorative on the matter, but can give a few tips based on discussion on the mailing list and on my own experience as admin of a (small) Cyrus server.

What to back up?

Occasionally, someone posts to the mailing list asking for help recovering a server from backups. Usually they have backed up the mail spools, thinking that was sufficient, but not backed up the configdirectory. This means that the mailboxes db, subscription lists, etc were not backed up. Usually recovery from this involves a manual reconstruction of the mailboxes.db (mailbox list and ACL information) and the loss of all user subscription information.

All information to reconstruct the critical mailboxes.db file can be obtained from the mail spools, and reconstruct will update the mailbox list file if the data stored in cyrus.header does not match that which is stored in the mailbox list.

There is no way to recover the subscription or seen state information from the mail spools -- if those databases are lost, the data is lost (however Cyrus will recreate the databases when they are next used, if this is the situation).

So: make sure you back up your config directory as well as your mail partition(s). The locations of these are defined in your /etc/imapd.conf with the partition-default and configdirectory directives. In many installs, these will be located in /var/imap (config) and /var/spool/imap (default partition).

It's also worth making sure that any databases you have in Berkeley DB format, are converted to plain text before you run your backup. If you have to restore to a new server with different software, you may find that the Berkley DB environment is different and your mailboxes.db etc are unreadable. This is not fun. Similar issues can turn up if you restore backups from a Cyrus 2.1 server that uses Berkley DB to a Cyrus 2.2 server that defaults to skiplist databases. Making plaintext copies of your databases also helps if the db somehow becomes corrupted. In general, it's just a good idea to keep plaintext copies of the mailbox list, if nothing else.

To convert your mailboxes.db to a text file, you can use ctl_mboxlist -d. As an example of how to use this in a backup script:

# make the text copy of the mailboxes DB
su - cyrus -c "ctl_mboxlist -d" > $CONFIGDIRECTORY/mailboxlist.txt

There may also be other databases that you'd like to make plaintext copies of. mailboxes.db is especially important, though - having a plain text copy of it when you need it can save you a world of trouble.

It may also be a good idea to include the sources of the version of Cyrus you're using, just in case you can't get hold of them when you need them most. I also include an installed Cyrus tree in my backups, to save time if I'm restoring to a binary-compatable machine.

It's probably not a bad idea to keep copies of all the Cyrus config files in your Cyrus backups. Again, this is to save time and stress when restoring. Even if you can't restore them directly, you can use them as a reference to see what might need changing in the new configs.

It may be desirable to explicitly exclude certain things from your backup. For example, the duplicate deliveries database is noncritical, and can be left out to save time and space. It may also be worth excluding Squatter indexes from your backup, as they can easily be recreated from the mail spools. If you're using find|cpio style backups, it's as easy as doing egrep -v '\/cyrus.squat$' to strip squatter files out of the list of files to back up. You might also want to exclude some mailboxes from the backups (such as Junk and Trash mailboxes).

How to actually make the backup

Cyrus doesn't require any really special backup techniques. You should be able to stop the master (thus ensuring that the spool stops changing) then take a copy of your spools, configdirectory, config files, etc as you would when backing up anything else. Some choices for backup methods include rsync, tar to a local tape, find|cpio|gzip, piping tar or cpio output over ssh to a remote host, etc. These are all standard backup methods, and whatever you currently use for everything else should be fine for the Cyrus spools as well. It'd be wise to make sure that your backup solution preserves the permissions on the mail spools, but even that isn't especially critical because the Cyrus spools are all owned by the user that Cyrus runs as.

One issue you may run into, however, is downtime. Stopping the master for the duration of the backup may simply not be acceptable.

reduce downtime

A shorter downtime can be accomplished with a little trick. If you have enough space to keep a copy of your /var/spool/imap directory then you can use rsync to shorten the downtime. First make an offsite copy of your imap folders. You can do this with rsync too:

rsync -vaR /var/lib/imap /var/spool/imap /opt/backup

1. stop the cyrus server
2. do a rsync between /var/spool/imap and offsite copy
3. start the cyrus server
4. do a regular backup of the offsite copy

This should keep the downtime much shorter then for a normal backup.

How to back up a Cyrus server without downtime

There doesn't seem to be a really ideal solution for this but several exist that are 'good enough'. To get `ideal' it'd have to be possible to ask the cyrus server to quiesce all disk activity - ie finish all in progress work, but block when it'd need to commence new disk activity, so the spool and config directory was in a consistent state for a snapshot. Unfortunately this is not presently supported, but in practice it's possible to get away without it.

Most people running a Cyrus IMAP server seem to use some form of dynamic/logical volume management scheme, such as LVM for Linux or others. Those that do not, should. One of the common advantages of LVM systems is the ability to quickly take read-only snapshots of filesystems. At least with the Linux LVM, it's possible to snapshot a mounted filesystem, mount the snapshot somewhere else, and back up the contents of the (static) snapshot while the original volume is still mounted r/w and in use. The snapshot takes up very little space, takes almost no time to create, and can simply be unmounted and discarded when you're finished with it. Using snapshots makes it easy to back up a server even when it's under significant ongoing load and can't be bought down - you can potentially have a couple of inconsistent mailboxes in the snapshot (because a message had been written but the index not yet updated), but you won't have major consistency issues even if your backup takes a long time. A mailbox with a damaged header cache or something like that is easily fixed with `recover' at restore time.

Without the use of snapshots, you would need to stop the master (locking out all clients and blocking mail delivery) while your backup runs. This is because it is not currently possible to tell the Cyrus system to 'complete all business and pause' - without killing the master, anyway. A 'SIGSTOP' sent to the master and/or all IMAPd processes won't really help, as there may be I/O in progress - and it'll still disrupt clients for the duration of your backup. If you don't want to stop the master for backups, the alternative is backing up the live mail store - risking significantly inconsistent backups. Consider what happens if a user moves a message from their sortme to their important folder between when you back up their important folder and when you later back up their sortme folder. You will entirely fail to back up the message. I don't know in detail the problems that are likely if you do a backup on a live config dir and mail spool - it might not be too serious. I'd be reluctant to do it, though.

Given the issues described above, I strongly recommend the use of some form of snapshot if your platform has the ability, or stopping the master for the duration of the backup if you can't take a snapshot.

LVM Resources:

LVM HOWTO

LVM HOWTO - Using snapshots

Linux LVM

One approach is to use `find' and `star' to handle the actual backup, after doing an LVM snapshot of the mail spool logical volume and mounting that read-only.

Something like the untested example:

    # NFS/other network FS mount to write backup tarballs to
    # Can also be a staging directory for scp, or you can just
    # adapt this code to pipe the tar data straight into an ssh
    # connection to a remote host.
    BACKUP_PREFIX=/var/backups/mail

    # Store the date so if we trip over midnight things don't go
    # horribly wrong
    ISO_DATE=`date -I`

    # A timestamp file that we use to record backup dates,
    # and use for differentials.
    TIMESTAMP=/var/lib/cyrus/backup_timestamp

    # set newer to something like "newer=/path/to/timestamp/file"
    # if you want to do a differential backup of files new/changed since
    # the timestamp file. You probably want to make this conditional,
    # so that you (eg) run a full backup every week, then differentials
    # against the full backup daily. 
    NEWER=""

    # Mail spool LVM volume group and logical volume name
    MAILSPOOL_VGROUP="MAIN"
    MAILSPOOL_LVNAME="mail"

    su - cyrus -c "/usr/sbin/ctl_mboxlist -d \
        > /var/lib/cyrus/mailboxes.txt"

    # Snapshot the logical volume the mail spool is on
    lvm lvcreate -s -n mail_snapshot_${ISO_DATE} -L 2G "/dev/${MAILSPOOL_VGROUP}/${MAILSPOOL_LVNAME}"
    MAILSPOOL="/mnt/mail_snapshot_${ISO_DATE}"
    mkdir -p "${MAILSPOOL}"
    mount -o ro /dev/${MAILSPOOL_VGROUP}/mail_snapshot_${ISO_DATE} "${MAILSPOOL}"

    # Generate a list of files to back up and pipe the list to star for adding to an archive.
    # we use `find' to generate a list because it's highly flexible and lets us filter out
    # undesired files. For example, adding a line like:
    #         -path "${MAILSPOOL}/user/*/Sent" -type f -size +1024k -mtime +7 -prune -o \
    # can be used to omit sent messages older than one week and larger than 1MB from backups.
    # Similarly:
    #     -path "${MAILSPOOL}/user/*/*." -type f \
    #          -not -path "${MAILSPOOL}/user/*/*/*" \
    #          -size +1024k -mtime +30 -prune -o \
    # is another size-and-age related control.
    #
    find /var/lib/cyrus "$MAILSPOOL" \
        -path "${MAILSPOOL}/user/*/Junk*" -prune -o \
        -path "${MAILSPOOL}/public/quarantine" -o \
        -path "${MAILSPOOL}/public/shared_junk" -o \
        -name cyrus.squat -prune -o \
        -type s -prune -o \
        -type p -prune -o \
        -print \
        | star -czl -list=- -f=${BACKUP_PREFIX}-${ISO_DATE}.xstar.gz $NEWER

    # Get rid of the logical volume snapshot
    umount /mnt/mail_snapshot_${ISO_DATE}
    rmdir /mnt/mail_snapshot_${ISO_DATE}
    lvm lvremove /dev/MAIN/mail_snapshot_${ISO_DATE}

    # touch the timestamp file and record the backup
    date >> "$TIMESTAMP"

I've omitted quite a bit of error checking and so on in the above - it is an example, not a ready to use backup script.

You can just pipe the output of star straight through an ssh connection (using a passwordless private key that only has access to a very limited unpriveleged backup user account on the target host) using something like:

   star [blah blah] -f=- | \
      ssh -i /path/to/key user@host \
      "cat > /remote/backup/file/path.star"

XFS: reliable snapshots

According to comments on the Cyrus mailing list by Simon Matter, it is possible to instruct an XFS filesystem to freeze before taking a snapshot. From the xfs_freeze man page:

The -f flag requests the specified XFS filesystem to be frozen from new modifications. 
When this is selected, all ongoing transactions in the filesystem are allowed to 
complete, new write system calls are halted, other calls which modify the filesystem 
are halted, and all dirty data, metadata, and log information are written to disk. 
Any process attempting to write to the frozen filesystem will block waiting for the 
filesystem to be unfrozen.

This helps avoid the issue described above with journalling filesystems and snapshots.

Tru64 UNIX AdvFS

Tru64 UNIX has AdvFS (Advanced File System), which supports clones. A "clone" is a read-only snapshot of a file system. It should be noted that this command requires a special license (which was a bit steep - HP, for God's sake lower those prices!). Here is a sample script for performing a backup:

#!/sbin/sh

#
# I keep Cyrus in */opt/* directory structures, as per DEC's recomendation
# Here, I'm using "vdump", but you can use anything you like
#
VERSION=2.2.12
HOME_DIR=Cyrus-IMAP4-${VERSION}
ROOT_HOME=/opt/${HOME_DIR}
USR_HOME=/usr/opt/${HOME_DIR}
VAR_HOME=/var/opt/${HOME_DIR}
#
# I sincerely hope you have setup Cyrus so it starts on boot
#
/sbin/init.d/cyrus stop
su cyrus -c "/usr/sbin/ctl_mboxlist -d" > ${VAR_HOME}/config/mailboxlist.txt
clonefset var_domain var var_clone
/sbin/init.d/cyrus start
#
# Now we can do backup as much as we like, cyrus is running and clone is keeping the snapshot
#
mount -t advfs -o ro var_domain#var_clone /clones/var
vdump -DC /clones/${VAR_HOME}
umount /clones/var
rmfset -f var_domain var_clone

Solaris 9 Snapshot Backup w Tivoli Example

Solaris 9 has filesystem snap ability. I am currently using it in production with Tivoli Enterprise Backup.

FSSNAP=`fssnap -o bs=/MOUNTEDFS,unlink /SNAPWHAT`
mount -F ufs -o ro FSSNAP /whereyouwantit


#!/bin/sh
#
# This creates a snap using the backingstore /snap, which is a mounted, local disk
# it is told to snap the /apps file system.  The output of fssnap gets passed into mount which
# mounts the newly created snap on /backup/apps
#
# We do not stop cyrus before our daily backup as downtime is not permitted.
#   We take an hourly backup of mailboxes.db, and a monthly safe backup of the 
#
mount -F ufs -o ro `fssnap -o bs=/snap,unlink /apps` /backup/apps

# Backup /mail through tivoli.  a dump would work here, or something else.
dsmc inc /mail -snapshotroot=/backup/mail &> /logs/mail.backup/mail.$DATE
umount /backup/mail
fssnap -d /mail

-- Article originally by Craig Ringer - 06 Dec 2003, 2005, 2007 - craig at postnewspapers dot com dot au