Understanding the Exchange Message Transfer Agent

How the MTA works and what to do if it falters

Lands' End's warehouse needs United Parcel Service, a post office needs mail carriers, the lungs need the circulatory system. And in Microsoft Exchange, the Information Store needs the Message Transfer Agent (MTA). A storehouse of anything—clothing, letters, oxygen, or messages—isn't useful without a delivery system. Although most people talk about Exchange's store, they give little attention to the MTA. Let's look at the MTA's role and how you can take full advantage of the service.

The MTA Service
Exchange Server uses a set of Windows NT services. The core services are Directory Services, the Information Store, the System Attendant, and the MTA. The MTA's function is to send messages between Exchange servers and external messaging systems. (The Information Store takes care of delivering messages to recipients on the same server.)

The MTA performs several major functions for Exchange:

The MTA usually runs all the time to perform these functions. Let's look at how it works with messages.

The MTA Database
All computer systems need a place to hold information while the system is processing it. The MTA does not use a relational database to store messages en route to their final destination. Instead, the MTA uses a set of files (db*.dat—db000001.dat, db000002.dat, and so on) in the MTA work directory (usually EXCHSRVRMTADATA) to form a logical database. The number of db*.dat files depends on the number of messages passing through the MTA. Lightly trafficked systems can have between 50 and 100 db*.dat files; systems coping with heavy traffic loads have many more.

I have unpleasant memories of situations where more than 50,000 db*.dat files accumulated in the MTADATA directory. Imagine how long NT Explorer takes to display the directory contents of 50,000 files! Why would the MTA create so many db*.dat files?

Microsoft designed the MTA database to hold transient files. The design expects the MTA to retrieve messages from the store and then route them onward as fast as possible. The MTA, therefore, uses db*.dat files only to store messages that the MTA is currently processing. You can consider the db*.dat files the set of messages that the MTA has accepted responsibility for but has not yet delivered.

When the MTA accepts a message from the store, it writes the message's content into a db*.dat file. However, the MTA attempts to reuse a set of db*.dat files and doesn't necessarily create a new db*.dat file for each message. After the MTA has successfully dispatched a message, the db*.dat file that had held the message's content is available for reuse. Thus, any build-up of db*.dat files in the MTA work directory generally signals that all is not well in the messaging network.

Unlike classic messaging systems (e.g., IBM PROFS and Digital ALL-IN-1), Exchange sends many system-generated messages in the background. These messages include public folder hierarchy and content replication, directory replication, and messages the System Attendant generates. Without replication, you wouldn't get the advantages of automatic directory updates or public folder population that Exchange provides.

All messages going to external systems pass through the MTA. If you don't pay attention to the amount of traffic that replication activities generate, any network outage that prevents messages from flowing across connectors will result in a build-up of messages in the MTA work directory. Messages will also build up when you take down Exchange servers for any reason.

To control the number of db*.dat files and the threshold for the size of temporary files, you can make Registry settings in

HKEY_LOCAL_MACHINESystemServicesMSExchangeMTAParameters

The relevant Registry keys here are DB File Handles, which determines the number of db*.dat files used for caching, and DB File Size Delete Threshold, which determines the maximum size of a file that the MTA will retain. When the MTA is finished with a file, it automatically deletes any file that exceeds the threshold.

If everything works smoothly, using NTFS to host the MTA database avoids the overhead accrued by relational databases. Going directly to the file system will always be faster than interacting with a relational database where Exchange must access the contents of a message as a binary object, usually spread over multiple pages in the database. According to Microsoft engineers, accessing discrete files from the disk is the fastest way to provide data to the MTA. NTFS is better than FAT because NTFS is more secure, and you don't want to offer a way for people to interfere with messages as they pass through the MTA.

However, when problems occur, the db*.dat files pile up, much like lost souls seeking a way out of a maze. The queue illustrated in Screen 1 is the result of a server in a remote site losing network connectivity for two days. Note that the messages listed to go to the remote site are all replication traffic, not interpersonal messages.

Because the MTA work directory can create and delete many files over time, the MTA represents a major source of I/O transactions on an Exchange server. If possible, isolate the MTA work directory from the other major sources of I/O (the transaction logs and the Information Store). You can isolate the MTA work directory by moving it away from the disks that hold the Information Store and transaction logs.

Exchange 5.5 focuses on performance and scalability. (For more information about Exchange 5.5, see the sidebar, "MTA Improvements in Exchange 5.5," and my article "Microsoft's Exchange 5.5 Debuts," December 1997.) In Exchange 5.5, the MTA improves the use of db*.dat files by intelligently caching files for deleted objects to reduce the overhead in creating and deleting items as the MTA processes them. Exchange 5.5 splits the Messaging API (MAPI) into seven queues to let the MTA process messages faster. Also, Exchange 5.5 uses fewer database objects (files) to handle each message. Although users have not yet seen the effects of these changes, the increased speed of all aspects of Exchange, including the MTA, is a good sign. Exchange lets you build massive systems and places no limit on the size of the Information Store. But introducing a massive Information Store won't help if another part of the system becomes a limiting factor; therefore, Microsoft is doing the right thing by seeking extra performance throughout Exchange.

An MTA Health Check
Although the MTA usually works inconspicuously in the background, the systems administrator can't just ignore the service. Let's look at how you can monitor the MTA queues and the set of files that lie behind the queues. This monitoring keeps you aware of network outages and consequent queue backups and interruptions in message delivery.

To analyze and fix problems in the MTA database, you can use mtacheck.exe, a utility in the EXCHSRVRBIN directory. If the MTA closes down abnormally, perhaps as a result of a system crash, the MTA automatically verifies its database when the MTA service starts up. You can run mtacheck if you suspect that the database is corrupt and the MTA has written errors into the event log.

The mtacheck utility has no GUI; you must run it from the NT command line. You can refine mtacheck by specifying the parameters listed in Table 1.

If you configure many Exchange servers to replicate public folder content too frequently (e.g., by setting a lot of public folders to replicate Always rather than every hour), many replication messages can build up in the MTA work directory. The same situation can occur if you create too many replicas. The more db*.dat files in the directory, the longer the integrity check will take, and this process can slow down overall Exchange startup. To speed up the integrity check, Microsoft has greatly improved MTA close-down and restart times in Exchange 5.5. Much of the improvement comes from a complete overhaul of the internal structures that control the set of files and objects that make up the MTA database.

Running mtacheck with the /rd and /rp switches can clean out a lot of messages (/rd removes directory replication messages; /rp removes public folder replication messages). For example, my company has built its Exchange network in a series of hubs and spokes. One site recently experienced a communications failure. Approximately 11,500 files had built up in the MTA work directory on the closest hub server before the site restored communications. Most of these files were replication messages, so I ran mtacheck with the /rp and /rd switches to clean out the work directory. After 20 minutes of intense mtacheck activity (75 percent to 80 percent CPU utilization on a Pentium 100), the site reduced the number of files to a more acceptable 200.

An extract of the type of information reported in the mtacheck log file appears in Listing 1. The site ran mtacheck on June 9, 1997 (970609 in X.400 Universal Time Coordinate—UTC—time). The mtacheck utility dates each message, so you can see that the MTA held some of these messages (those dated 970602) in the work directory for a week. The MTA usually discards such old messages after a week. (Reporting that 103 percent of all objects were damaged is, I hope, just a quirk of mtacheck—or a bug!)

Run mtacheck with the /rp and /rd switches only if you know what you're doing. In the case I described, I checked all the MTA queues first to assess the types of messages there. After I learned that replication messages were clogging the MTA work directory, I didn't hesitate to run mtacheck. Remember, Exchange always backfills missing data from public folders or the directory, so if mtacheck removes any useful data in messages, Exchange will soon replace it.

Clearly, you must check the log file carefully after any mtacheck run. If you find more replication messages than expected, review the replication schedule for folders (or public stores as a whole) and the number of replicas, to make sure that you're not introducing too much replication traffic into your network. The mtacheck utility will clear up a mess, but failure to check why the problem occurred will result in another mess in the future.

Diagnostic Settings
As with other major components of Exchange, you can instruct the MTA to report diagnostic information to the NT event log. Diagnostics range from None (the default setting, which means that the MTA will log only critical events or service startup/close-down events) to Maximum. Always use the default setting, unless you must track down the cause of a problem. Crank MTA diagnostics to Maximum to gather data before you report a problem to Microsoft support.

In addition, Microsoft publishes several Performance Monitor (Perfmon) counters for the MTA. You can use these counters to monitor the flow of messages into and out of the MTA and to detect conditions such as an unexpected buildup of messages for a connector.

The GWART and How It Works
When the MTA receives a message, it must decide how to route it. The GWART (Gateway Address Routing Table—the text representation of the routing table) contains information about the paths available to route messages to their final destinations. In the EXCHSRVRMTADATA directory, gwart0.mta contains current routing information. The somewhat obscure detail available in gwart0.mta is hard to read; a graphical representation of routing paths would make it easier for systems administrators to identify likely problems (e.g., SMTP mail for a particular domain not being routed out through the optimal IMS connector). As long as I'm telling Microsoft my wishes, I'd also like the ability to input network details (e.g., bandwidth and type of link) and have the Exchange Administrator program display them with the routing paths. Such a tool would be invaluable for planning the ebb and flow of data within an Exchange organization, especially for administrators of large, complex networks with many sites and connectors.

The EXCHSRVRMTADATA directory holds gwart1.mta, which stores a copy of previous routing information. The MTA recalculates the table every time you add or update configuration information for a connector. Systems administrators can force an update by selecting the MTA object from any server in the site and clicking Recalculate Routing, as shown in Screen 2.

Exchange automatically replicates GWART information to all other servers in an organization. But sometimes you don't want to make details of a new connector available to everyone. For example, if you install a new fax connector in a site in Australia, you don't want American users to be able to route faxes via Australia. You can create a LOCAL routing entry for the connector. LOCAL routing entries let local servers know about a connector but restrict the publication of connector details outside the host site. In Exchange 5.5, you'll be able to restrict access to a connector within a site (this functionality depends on subsites, another new feature in Exchange 5.5). The new version of Exchange provides some additional user interfaces to manage link localization, the term used to describe restricted connectors.

Both gwart0.mta and gwart1.mta are simple text files that you can open and examine with the Notepad utility or any other text editor. Let's use some of my organization's data to see what you might find in GWART.

Interpreting the Signposts
The GWART organizes the routing table into sections. Each section tells Exchange how it can route messages sent to a particular type of address. The table lists native Exchange-style addresses first; each site in the organization has a separate entry. Screen 3 shows details of the addresses for six sites.

The entry for each site begins with the organization and site name. Note that the routing table stores directory names, which are not necessarily the same as the display names for objects that you see when you use the Exchange administration program to browse in the Exchange directory.

The first entry in Screen 3 shows how the MTA routes messages to users in the Belfast site within the Digital Equipment organization. Routing requires two steps. First, the MTA must route messages across the site connector that links this site (Stillorgan) to the Dublin site. When messages reach Dublin, Exchange routes them across an Internet Mail Connector (IMC) to Belfast. Fortunately, the descriptive names allocated to the sites and connectors make this GWART reasonably easy to read.

For Sites 3, 5, and 6 in Screen 3, Exchange must route messages in three steps. For example, Exchange routes messages to the Malmoe (a city in Sweden) site across the site connector to Dublin, then across a connector between Dublin and a server called COPSI1 (which is in the Copenhagen site), and then across another connector to Malmoe. You can obtain more explicit details of the routing from Dublin to Copenhagen and Copenhagen to Malmoe by examining the X.400 or SMTP connector information listed later in the routing table.

Understanding the Routing of X.400 Addresses
Microsoft has based all the MTA's underlying structures and workings on the world standard organizations' 1988 recommendations for the X.400 messaging protocol. In X.400 terminology, the Exchange MTA is a fully compliant message transfer agent.

In theory (although not always in practice), the MTA can connect Exchange to different X.400-based messaging systems, as long as both sides have the same understanding of the recommendations.

Exchange automatically generates several types of addresses when you add new recipients (mailboxes or custom recipients) to the directory. The typical set of addresses includes

You can generate only the address types you need (e.g., you don't need to generate cc:Mail addresses if you never use the cc:Mail connector). Some address types, such as SMTP, let you maintain multiple addresses of that type for a recipient. Removing unwanted addresses is a good idea because the less directory data that Exchange must replicate around the organization, the better. You need an X.400 address, however, because it is fundamental to the way Exchange routes messages within an organization.

Screen 4, page 176, illustrates some of the routing instructions for X.400 addresses. An X.400 address is a set of attribute pairs. When the MTA checks an address on a message, it looks for just enough attributes to discover how to route the message. The scan begins at the left side of the address and moves to the right until the MTA has built up enough of the address to determine the route by referring to the routing table.

The first attribute is the country code, followed by the administrative domain, private domain, organization name, organization units, and on to an individual's given name and surname. Usually, the first four or five attributes are enough to route a message. For example, in Screen 4 you can see that Exchange defines two routes with a GB (Great Britain) country code and two with IE (Ireland). Under each country code, you can see the other attributes stated in descending order until the address becomes unique. Thus, the four attributes

C=GB; A=DIGITAL; P=READING; O=SOUTH UK IST 

are enough to tell the MTA to route any address beginning with these attributes across the site connector to Dublin and then via the X.400 connector to the South UK site. And the attributes

C=GB; A=DIGITAL; P=WARRINGTON; O=NORTH UK

are sufficient for the MTA to route messages beginning with these attributes across the site connector to Dublin and then via a different X.400 connector to the North UK site. To complete the picture, an else clause tells the MTA to route any other X.400-style address that begins with C=GB to Dublin, then to Copenhagen, and then on to another connector back to the UK.

Although routing from Ireland to Denmark and then to the UK doesn't appear to make sense, someone—probably in the Copenhagen site—entered this route in the site-routing table. Because the MTA automatically replicates routing information between all the servers in an organization­and any strange routing instructions one site enters will go to everyone else­review the routing table occasionally to uncover anomalies such as this example.

Although X.400 is the best connector for linking sites across low-bandwidth connections, you will use other connectors within an Exchange organization. Screen 5 shows the instructions in the GWART that tell the MTA how to process SMTP and SNADS addresses. The SMTP=* instruction tells the MTA to route any SMTP-style address sent from this site across the connector to Dublin and out via the IMS from there. But what are all the fully qualified domains (e.g., BVOJEN.BVO.DEC.COM and UTO-EXCHANGEIST.UTO.DEC.COM) specified here? When you connect a new site using the IMS, you typically provide a routing address to tell the connector how to route mail to the site. At some point, organizations used these addresses to connect sites with the IMS. The data will stay in the routing table unless you remove it, and, as I mentioned before, removing spurious routing information from the table is a good practice.

A SNADS connector entry completes the tour through the routing table. Reaching the connector requires four steps—not unusual for connectors that link to mainframe messaging systems.

Silent Partner
An MTA is an important component of any messaging system, especially those systems that have to deal with a heterogeneous world. Understanding the Exchange MTA will help you keep Exchange running smoothly.