Cyrus IMAP Server: Overview and Concepts
This document gives an overview of the Cyrus IMAP server. The Cyrus IMAP (Internet Message Access Protocol) server provides access to personal mail and system-wide bulletin boards through the IMAP protocol. The Cyrus IMAP server is a scalable enterprise mail system designed for use from small to large enterprise environments using standards-based technologies.
A full Cyrus IMAP implementation allows a seamless mail and bulletin board environment to be set up across multiple servers. It differs from other IMAP server implementations in that it is run on "sealed" servers, where users are not normally permitted to log in. The mailbox database is stored in parts of the filesystem that are private to the Cyrus IMAP system. All user access to mail is through software using the IMAP, IMAPS, POP3, POP3S, or KPOP protocols.
The private mailbox database design gives the server large advantages in efficiency, scalability, and administratability. Multiple concurrent read/write connections to the same mailbox are permitted. The server supports access control lists on mailboxes and storage quotas on mailbox hierarchies.
This document is organized into the following areas:
- Mailbox Namespace
- Access Control Lists
- Login Authentication
- New Mail Notification
- POP3 Server
- The syslog facility
- Mail Directory Recovery
- Configuration Directory
- Message Delivery
In this implementation, non-ASCII characters and shell metacharacters are not permitted in mailbox names.
Optionally, the server can present mailboxes using the UNIX hierarchy convention.
All personal mailboxes for user "bovik" begin with the string "user.bovik.". For example, if user "bovik" had a personal "work" mailbox, it would be called "user.bovik.work". To user "bovik", however, the prefix "user.bovik." normally appears as "INBOX.". The mailbox "user.bovik.work" would therefore appear as "INBOX.work". If the access control list of the mailbox permitted other users to see that mailbox, it would appear to them as "user.bovik.work".
The mailbox "user.bovik" is where the user "bovik" normally receives new mail, and normally appears to user "bovik" as "INBOX". The mailbox "user.bovik" is referred to in this document as user "bovik"'s INBOX.
Administrators create and delete users by creating and deleting the users' INBOX. If a user has an INBOX, then they are allowed to subscribe to mailboxes. Only users without dots in their userid are permitted to have an INBOX. (A user with a dot in their userid would be able to login but would not be able to receive mail. Note that when using the unix hierarchy seperator, this is not the case, and any user may have a dot in their userid.)
When an administrator deletes a user's INBOX, all of the user's personal mailboxes are deleted as well.
With the one notable exception of INBOX, all mailbox names are system-wide--they refer to the same mailbox regardless of the user. Access control lists determine which users can access or see which mailboxes. Using
In contexts which permit relative mailbox names, the mailbox namespace works as follows:
- Names that do not start with "." are fully qualified.
- Names that start with "." are relative to the current context.
- lookup- The user may see that the mailbox exists.
- read- The user may read the mailbox. The user may select the mailbox, fetch data, perform searches, and copy messages from the mailbox.
- seen- Keep per-user seen state. The "Seen" and "Recent" flags are preserved for the user.
- write- The user may modify flags and keywords other than "Seen" and "Deleted" (which are controlled by other sets of rights).
- insert- The user may insert new messages into the mailbox.
- post- The user may send mail to the submission address for the mailbox. This right differs from the "i" right in that the delivery system inserts trace information into submitted messages.
- create- The user may create new sub-mailboxes of the mailbox, or delete or rename the current mailbox.
- delete- The user may store the "Deleted" flag, and perform expunges.
- administer- The user may change the ACL on the mailbox.
- The user can read the mailbox.
- The user can read the mailbox, and can post to it through the delivery system. Most delivery systems do not provide authentication, so the "p" right usually has meaning only for the "anonymous" user.
- The user can see the mailbox and can read it, but the server does not preserve the "Seen" and "Recent" flags. This set of rights is primarily useful for "anonymous IMAP."
- The user can read the mailbox and the server preserves the "Seen" and "Recent" flags, but the mailbox is not visible to the user through the various mailbox listing commands. The user must know the name of the mailbox to be able to access it.
- The user can read and append to the mailbox, either through IMAP, or through the delivery system.
There are two special identifiers, "anonymous", and "anyone", which are explained below. The meaning of other identifiers usually depends on the authorization mechanism being used (selected by --with-auth at compile time, defaulting to Unix).
Note that authorization is not authentication. Authentication is the act of proving who you are. Authorization is the act of determining what rights you have. Authentication is discussed in the Login Authentication part of this document.
In the Unix authorization mechanism, identifiers are either a valid userid or the string "group": followed by a group listed in "/etc/group". Thus:
root Refers to the user root group:staff Refers to the group staff
It is also possible to use unix groups with users authenticated through a non-/etc/passwd backend. Note that using unix groups in this way (without associated /etc/passwd entries) is not recommended.
Using the Kerberos authorization mechanism, identifiers are of the form:
principal.instance@realmIf ".instance" is omitted, it defaults to the null string. If "@realm" is omitted, it defaults to the local realm.
The file "/etc/krb.equiv" contains mappings between Kerberos principals. The file contains zero or more lines, each containing two fields. Any identity matching the first field of a line is changed to the second identity during canonicalization. For example, a line in "/etc/krb.equiv" of:
bovik@REMOTE.COM bovikwill cause the identity "bovik@REMOTE.COM" to be treated as if it were the local identity "bovik".
A site may wish to write their own authorization mechanism, perhaps to implement a local group mechanism. If it does so (by implementing an auth_[whatever] module), it will dictate its own form and meaning of identifiers.
For example, given the ACL:
anyone lrsp fred lwi -anonymous sThe user "fred" will be granted the rights "lrswip" and the anonymous user will be granted the rights "lrp".
Note that some rights are available implicitly, for example 'anonymous' always has 'p' on user INBOXes, and users always have rights on mailboxes within their INBOX hierarchy.
The Cyrus IMAP server uses the Cyrus SASL library for authentication. This section describes how to configure SASL with use with Cyrus imapd. Please consult the Cyrus SASL System Administrator's Guide for more detailed, up-to-date information.
- Kerberos v4 Plaintext passwords are verified by obtaining a ticket for the server's Kerberos identity, to protect against Kerberos server spoofing attacks.
The method of plaintext password verification is always through the SASL library, even in the case of the internal LOGIN command. This is to allow the SASL library to be the only source of authentication information. You'll want to look at the sasl_pwcheck_method option in the SASL documentation to understand how to configure a plaintext password verifier for your system.
To disallow the use of plaintext passwords for authentication, you can set allowplaintext: no in imapd.conf. This will still allow PLAIN under TLS, but IMAP LOGIN commands will now fail.
The server will permit logins by identities in the local realm and identities in the realms listed in the "loginrealms" option in "/etc/imapd.conf".
The file "/etc/krb.equiv" contains mappings between Kerberos principals. The file contains zero or more lines, each containing two fields. Any identity matching the first field of a line is permitted to log in as the identity in the second field.
If the "loginuseacl" configuration option is turned on, than any Kerberos identity that is granted the "a" right on the user's INBOX is permitted to log in as that user.
The Cyrus IMAP server supports quotas on storage, which is defined as the number of bytes of the relevant RFC-822 messages, in kilobytes. Each copy of a message is counted independently, even when the server can conserve disk space use by making hard links to message files. The additional disk space overhead used by mailbox index and cache files is not charged against a quota.
Quotas on a quota root apply to the sum of the usage of any mailbox at that level and any sub-mailboxes of that level that are not underneath a quota root on a sub-hierarchy. This means that each mailbox is limited by at most one quota root.
For example, if the mailboxes
user.bovik user.bovik.list.imap user.bovik.list.info-cyrus user.bovik.saved user.bovik.todo
exist and the quota roots
user.bovik user.bovik.list user.bovik.savedexist, then the quota root "user.bovik" applies to the mailboxes "user.bovik" and "user.bovik.todo"; the quota root "user.bovik.list" applies to the mailboxes "user.bovik.list.imap" and "user.bovik.list.info-cyrus"; and the quota root "user.bovik.saved" applies to the mailbox "user.bovik.saved".
Quota roots are created automatically when they are mentioned in the "setquota" command. Quota roots may not be deleted through the protocol, see Removing Quota Roots for instructions on how to delete them.
Mail delivery is a special case. In order for a message to be delivered to a mailbox, the quota root for the mailbox must not have usage that is over the limit. If the usage is not over the limit, then one message may be delivered regardless of its size. This puts the mailbox's usage over the quota, causing a user to be informed of the problem and permitting them to correct it. If delivery were not permitted in this case, the user would have no practical way of knowing that there was mail that could not be delivered.
If the usage is over the limit, then the mail delivery will fail with a temporary error. This will cause the delivery system to re-attempt delivery for a couple of days (permitting the user time to notice and correct the problem) and then return the mail to the sender.
The server only issues warnings when the user has "d" rights because only users with "d" rights are capable of correcting the problem.partitions. A single quota root can apply to mailboxes in different partitions.
The Cyrus IMAP server comes with a notification daemon which supports multiple mechanisms for notifying users of new mail. Notifications can be configured to be sent upon normal delivery ("MAIL" class) and/or sent as requested by a Sieve script ("SIEVE" class).
By default, both types of notifications are disabled. Notifications are enabled by using one or both of the following configuration options:
- the "mailnotifier" option selects the notifyd method to use for "MAIL" class notifications
- the "sievenotifier" option selects the notifyd method to use for "SIEVE" class notifications (when no method is specified by the Sieve action)
The optional second argument to the "create" command can usually be given only when using a specialized Cyrus-aware administrative client such as cyradm.install-netnews.php.
When Kerberos login authentication is being used, the POP3 server uses the server identity "pop.host@realm" instead of "imap.host@realm", where "host" is the first component of the server's host name and "realm" is the server's Kerberos realm. When the POP3 server is invoked with the "-k" switch, the server exports MIT's KPOP protocol instead of generic POP3.
- CRIT - Critical errors which probably require prompt administrator action
- ERR - I/O errors, including failure to update quota usage. The syslog message includes the specific file and Unix error.
- WARNING - Protection mechanism failures, client inactivity timeouts
- NOTICE - Authentications, both successful and unsuccessful
- INFO - Mailbox openings, duplicate delivery suppression
- message files
- There is one file per message, containing the
message in RFC 822 format. Lines in the message are separated by
CRLF, not just LF. The file name of each message is the message's
UID followed by a dot (.).
In netnews newsgroups, the message files instead follow the format and naming conventions imposed by the netnews software.
- This file contains a magic number and variable-length
information about the mailbox itself.
- This file contains fixed-length information about the
mailbox itself and each message in the mailbox.
- This file contans variable-length information about
each message in the mailbox.
- This file contains variable-length state information about each reader of the mailbox who has "s" permissions.
An administrator may recover from a damaged disk by restoring message files from a backup and then running reconstruct to regenerate what it can of the other files.
The "reconstruct" program does not adjust the quota usage recorded in any quota root files. After running reconstruct, it is advisable to run "quota -f" (described below) in order to fix the quota root files.
The mailboxes file in the configuration directory is the most critical file in the entire Cyrus IMAP system. It contains a sorted list of each mailbox on the server, along with the mailboxes quota root and ACL. To reconstruct a corrupted mailboxes file, run the "reconstruct -m" command. The "reconstruct" program, when invoked with the "-m" switch, scavenges and corrects whatever data it can find in the existing mailboxes file. It then scans all partitions listed in the imapd.conf file for additional mailbox directories to put in the mailboxes file.
The cyrus.header file in each mailbox directory stores a redundant copy of the mailbox ACL, to be used as a backup when rebuilding the mailboxes file.
The "quota" program, when invoked with the "-f" switch, recalculates the quota root of each mailbox and the quota usage of each quota root.
There is no program to recover from damaged subscription files. A site may recover from lost subscription files by restoring from backups.
If a subdirectory of "log" exists with the same name as a user, the IMAP and POP3 servers will keep a telemetry log of protocol sessions authenticating as that user. The telemetry log is stored in the subdirectory with a filename of the server process-id and starts with the first command following authentication.
- hostname of the client
- login name of the user, if logged in
- selected mailbox, if a mailbox is selected
The "proc" subdirectory is normally be cleaned out on server reboot.
Mail transport agents such as Sendmail, Postfix, or Exim communicate with the Cyrus server via LMTP (the Local Mail Transport Protocol) implemented by the LMTP daemon. This can be done either directly by the MTA (prefered, for performance reasons) or via the deliver LMTP client.
Local Mail Transfer Protocol
LMTP, the Local Mail Transfer Protocol, is a variant of SMTP design for transferring mail to the final message store. LMTP allows MTAs to deliver "local" mail over a network. This is an easy optimization so that the IMAP server doesn't need to maintain a queue of messages or run an MTA.
The Cyrus server implements LMTP via the lmtpd daemon. LMTP can either be used over a network via TCP or local via a UNIX domain socket. There are security differnces between these two alternatives; read more below
For final delivery via LMTP over a TCP socket, it is necessary to use LMTP AUTH. This is accomplished using SASL to authenticate the delivering user. If your mail server is performing delivery via LMTP AUTH (that is, using a SASL mechanism), you will want their authentication id to be an LMTP admins (either via the admins imapd.conf option or via the <service>_admins option, typically lmtp_admins).
Alternatively you may deliver via LMTP to a unix domain socket, and the connection will be preauthenticated as an administrative user (and access control is accomplished by controlling access to the socket).
Note that if a user has a sieve script, the sieve script runs authorized as *that* user, and the rights of the posting user are ignored for the purposes of determining the outcome of the sieve script.
Single Instance Store
If a delivery attempt mentions several recipients (only possible if the MTA is speaking LMTP to lmtpd), the server attempts to store as few copies of a message as possible. It will store one copy of the message per partition, and create hard links for all other recipients of the message.
Single instance store can be turned off by using the "singleinstancestore" flag in the configuration file.
Duplicate Delivery SuppressionA message is considered a duplicate if two copies of a message with the same message-id and the same envelope receipient are received. Cyrus uses the duplicate delivery database to hold this information, and it looks approximately 3 days back in the default install.
Duplicate delivery suppression can be turned off by using the "duplicatesuppression" flag in the configuration file.
Sieve, a Mail Filtering LanguageSieve is a mail filtering language that can filter mail into an appropriate IMAP mailbox as it is delivered via lmtp. For more information, look here.
Cyrus Murder, the IMAP AggregatorCyrus now supports the distribution of mailboxes across a number of IMAP servers to allow for horizontal scalability. For information on setting up this configuration, see here.
Return to the Cyrus IMAP Server Home Page