Exim is designed to work efficiently on systems that are permanently connected to the Internet and are handling a general mix of mail. In such circumstances, most messages can be delivered immediately. Consequently, Exim does not maintain independent queues of messages for specific domains or hosts, though it does try to send several messages in a single SMTP connection after a host has been down, and it also maintains per-host retry information.
When Exim receives a message, it writes two files in its spool directory. The first contains the envelope information, the current status of the message, and the headers, while the second contains the body of the message.
The envelope information consists of the address of the message's sender and the address(es) of the recipient(s). This information is entirely separate from any addresses contained in the headers. The status of the message includes a list of recipients who have already received the message. The format of the first spool file is described in chapter "Format of spool files".
Any header rewriting that is specified in the configuration (see chapter "Address rewriting") is done once and for all as the message is received. It is also possible to specify the addition or removal of certain headers at the time the message is written out (see chapters "Generic options for transports" and "Common generic options for directors and routers").
Every message handled by Exim is given a message id which is sixteen characters long. It is divided into three parts, separated by hyphens. Each part is a number, expressed in base 62:
The names of the two spool files consist of the message id, followed by -H for the file containing the envelope and headers, and -D for the data file.
By default all these spool files are held in a single directory called "input" inside the general Exim spool directory. Some operating systems do not perform very well if the number of files in a directory gets very large; to improve performance in such cases there is an option that causes Exim to split up the input files into 62 sub-directories whose names are single letters or digits.
Exim can be configured not to start a delivery process automatically when a message is received; this can be unconditional, or depend on the number of incoming SMTP connections or the system load (where available). In these situations, new messages wait on the queue until a queue-runner process picks them up.
A message remains in the spool directory until it is completely delivered to its recipients or to an error address, or until it is deleted by an administrator or by the user who originally created it. In cases when delivery cannot proceed -- for example, when a message can neither be delivered to its recipients nor returned to its sender, the message is marked `frozen' on the spool, and no more deliveries are attempted.
An administrator can `thaw' such messages when the problem has been corrected, and can also freeze individual messages by hand if necessary. In addition, an administrator can force a delivery error, causing an error message to be sent.
As delivery proceeds, Exim writes timestamped information about each address to a per-message log file; this includes any delivery error messages. This log is solely for the benefit of the administrator, and is normally deleted with the spool files when processing of a message is complete. However, Exim can be configured to retain it (a dangerous option, as the files can accumulate rapidly on a busy system). Exim also writes delivery messages to its main log file, whose contents are described in chapter "Log files".
All the information Exim itself needs to set up a delivery is kept in the first spool file with the headers. When a successful delivery occurs, the address is immediately written at the end of a journal file, whose name is the message id followed by -J. At the end of a delivery run, if there are some addresses left to be tried again later, the first spool file is updated to indicate which these are, and the journal file is then deleted. Updating the spool file is done by writing a new file and renaming it, to minimize the possibility of data loss.
Should the system or the program crash after a successful delivery but before the spool file has been updated, the journal is left lying around. The next time Exim attempts to deliver the message it reads the journal file and updates the spool file before proceeding. This minimizes the chances of double deliveries caused by crashes.
The main delivery processing elements of Exim are called directors, routers, and transports, and collectively these are known as drivers. Code for a number of them is provided, compile-time options specify which ones are included in the binary, and runtime options specify which ones are actually used.
A transport is a driver that transmits a copy of the message from Exim's spool to some destination. There are two kinds of transport: for a local transport, the destination is a file or a pipe on the local host, while for a remote transport the destination is some other host. A message is passed to a specific transport as a result of successful directing or routing. If a message contains several addresses, it may be passed to a number of different transports.
A director is a driver that operates on a local address, either determining how its delivery should happen, or converting the address into one or more new addresses (for example, via an alias file). A local address is one whose domain matches an entry in the list given in the `local_domains' option, or has been determined to be local by a router -- see below. The fact that an address is local does not imply that the message has to be delivered locally; it can be directed either to a local or to a remote transport.
A router is a driver that operates on an apparently remote address, that is an address whose domain does not match anything in the list given in `local_domains'. When a router succeeds it can route an address either to a local or to a remote transport, or it can change the domain, and pass the address on to subsequent routers.
In exceptional cases, a router may determine that an address is local after all, and cause it to be passed to the directors. This happens automatically if a host lookup expands an abbreviated domain into one that is local; it can also be made to happen if an MX record points to the local host. This is the only case in which the directors are used to process an address that may not match anything in `local_domains'. The diagram below illustrates the relationship between the three kinds of driver.
address
|
|<---------- new address --------
V |
----------------- |
| matches | |
------- no ----| local_domains |--- yes ------- |
| | option? | | |
V ----------------- V |
----------- ------------ |
| routers |--- local after all ------------->| directors |---
----------- -------------
| -------------- |
---------------->| transports |<---------------
--------------
As new features have been added to Exim, the distinction between routers and directors has become less clear-cut than it once was. However, since a typical configuration has a number of directors and routers, checking the domain against `local_domains' once at the start does use fewer resources than checking it for each of them.
When a message is to be delivered, the sequence of events is roughly as follows:
Delivery is said to be deferred when the message remains on the queue for a subsequent delivery attempt after a temporary failure. Such messages get processed again by queue-runner processes that are periodically started, either by an Exim daemon or via `cron'.
Temporary failures may be detected during routing and directing as well as during the transport stage. Exim uses a set of configured rules to determine when next to process the failing address (see chapter "Retry configuration"). These rules also specify when Exim should give up trying to deliver to the address, at which point it generates a failure report.
When a delivery is not part of a queue run (typically an immediate delivery on receipt of a message), the directors are always run for local addresses, and local deliveries are always attempted, even if retry times are set for them. This makes for better behaviour if one particular message is causing problems (for example, causing quota overflow, or provoking an error in a filter file). If such a delivery suffers a termporary failure, the retry data gets updated as usual, for use by the next queue-runner process.
When a message cannot be delivered to some or all of its intended recipients, a delivery failure report is generated. All the addresses that failed in a given delivery attempt are listed in a single failure report. If a message has many recipients, it is possible for some addresses to fail in one delivery attempt and others to fail subsequently, giving rise to more than one failure report for a single message. The wording of delivery failure reports can be customized by the administrator. See chapter "Customizing error and warning messages" for details.
Delivery failure messages contain an `X-Failed-Recipients:' header, listing all failed addresses, for the benefit of programs that try to analyse such messages automatically.
A failure report is normally sent to the sender of the original message, as obtained from the message's envelope. For incoming SMTP messages, this is the address given in the MAIL FROM command. However, when an address is expanded via a forward or alias file, an alternative address can be specified for delivery failures of the generated addresses. For a mailing list expansion (see chapter "Using Exim to handle mailing lists") it is common to direct failure reports to the manager of the list.
If a failure report (either locally generated or received from a remote host) itself suffers a delivery failure, the message is left on the queue, but is `frozen', awaiting the attention of an administrator. However, there are options which can be used to make Exim discard such failure reports, or to keep them for only a short time.
There are many reasons why a message may not be immediately deliverable to a particular address. Failure to connect to a remote machine (because it, or the connection to it, is down) is one of the most common. Local deliveries may also be delayed if NFS files are unavailable, or if a mailbox is on a file system where the user is over quota. Exim can be configured to impose its own quotas on local mailboxes; where system quotas are set they will also apply.
A machine that is connected to the Internet can normally deliver most mail straight away (the usual figure for Cambridge is 98%). In its default configuration, Exim starts a delivery process whenever it receives a message, and usually this completes the entire delivery. This is a lightweight approach, avoiding the need for any centralized queue managing software. There are those who argue that a central message manager would be able to batch up messages for the same host and send them in a single SMTP call. I do not myself believe this would occur much in general, unless messages were significantly delayed in order to create a batch.
However, if a host is unreachable for a period of time, a number of messages may be waiting for it by the time it recovers, and sending them in a single SMTP connection is clearly beneficial. Whenever a delivery to a remote host is deferred, Exim makes a note in its hints database, and whenever a successful SMTP delivery has happened, it looks to see if any other messages are waiting for the same host. If any are found, they are sent over the same SMTP connection, subject to a configuration limit as to the maximum number in any one connection.
Go to the first, previous, next, last section, table of contents.