The Exchange Server Troubleshooter - 01 Nov 1998

I want to devote most of my first column as the Exchange Server troubleshooter to answering some readers' questions about message routing. Troubleshooting nondelivery receipts (NDRs) and mail failures is easier if you understand how routing works.

How does Exchange Server route messages?

Exchange Server routes messages to a destination in three cases: when the sender and recipient are on the same server, when they're on different servers within the same site, and when they're on servers in different sites. The last case includes messages that you send via the Internet Mail Service (IMS) and X.400 connectors, because Exchange treats those connectors as if they were servers in a foreign site.

How does Exchange Server route messages when the sender and the recipient are on the same server?

Routing messages when the sender and the recipient are on the same server is the simplest case. The client sends the message to the server, where the Information Store service processes it. Exchange delivers non-Messaging API (MAPI) clients' messages directly to the Information Store service via protocol stacks that handle Post Office Protocol (POP) 3, Internet Message Access Protocol (IMAP) 4, and HTTP clients. The Information Store then puts one copy of the message into the private Information Store, and each recipient mailbox gets a pointer to the message. This single-instance storage means that Exchange has to store only one copy of the message in the private Information Store on each server, which results in significant space savings over storing a copy of the message in each mailbox. Finally, if any recipients are connected with MAPI clients, the Information Store service notifies them that new mail is available.

How does Exchange Server route messages to recipients on different servers in a site?

Intrasite delivery involves several steps. Let's say Alice, whose mailbox is on a server named HQ, sends mail to three users: one on HQ, one on the ENGINEERING server, and one on the FINANCE server. When the HQ Information Store service gets the message from Alice's Outlook client, it resolves each recipient address in the directory. Finding that one user is on HQ, the service delivers the message according to the steps that the previous question outlines. For each recipient whose mailbox is not on HQ, the following happens:

What is the GWART?

The GWART is the Gateway Address Routing Table. To understand the GWART, though, you need a couple of fundamentals. (For more information about the GWART and how it works, see Tony Redmond, "Understanding the Exchange Message Transfer Agent," Windows NT Magazine, January 1998.) Let's start with address spaces. Each connector can have one or more address spaces attached to it; these spaces define a range or set of addresses that the connector can talk to. For example, the IMS usually has an address space of *, which means that it can handle any Simple Mail Transfer Protocol (SMTP) address. You can just as easily define the address spaces *.com and *.org to control mail delivery to individual domains.

Each address space has a routing cost associated with it. For more information about routing costs, see Mark Ott, "Intrasite and Intersite Directory Replication," May 1998. This cost is a number between 0 and 100; higher values are more expensive. When Exchange gets ready to route a message, it always chooses the lowest-cost path that will get the message to its destination. Because each connector in a site can have cost parameters, you can set routing costs on each address space to control message flow. Assign lower costs to the connectors you want Exchange to use most frequently, and higher costs to connectors you want Exchange to use less often. Because Exchange always picks the least-cost path, messages flow where you want them to, unless the cheapest connector is unavailable. In that case, messages automatically begin flowing over the next-cheapest path.

The address space and cost information lives in the GWART. The GWART resides in the Organization's directory, so Exchange automatically replicates it to all servers and sites in the Organization. (Two text copies of the GWART, GWART0.MTA and GWART1.MTA, are stored in the exchsrvrMTADATA directory.) The System Attendant service and the MTA update the GWART whenever you add a new address space or when Exchange Server recalculates the routing cost. You can schedule the task to occur automatically, or you can perform it manually. The Routing Calculation Schedule tab of the Site Addressing object lets you choose Recalculate Routing to force a GWART recalculation. When an MTA wants to find the best route for a message, it consults the GWART.

What about recipients who don't have Exchange Server addresses?

If the recipient address's DN is an SMTP, Lotus Notes, Microsoft Mail, or other non-Exchange address, the MTA uses the address space and routing information in the GWART to find a connector that can process the message.

How does the MTA decide which route to use?

Routing is a two-step process. In the first step, routing, the MTA lists every connector that has an address space that matches the recipient address. For example, if the MTA is routing a message with a cc:Mail address, it builds a list of all connectors that have cc:Mail address spaces. The result of the routing phase is a list of all the places that Exchange could send a message.

The second step, selection, uses the list of all possible routes and their associated costs to choose the least-cost route. The MTA adds the costs required for each message path, and the one with the lowest total cost wins­that's the path the MTA will use to route the message.

Let's say you're sending an X.400 message and you have three X.400 connectors in your organization. One connects directly to the backbone and has a cost of 25 with an address space of *. One connects to XYZ, a company you do business with. Its address space restricts its use to messages sent to the path of country (c) =US and organization (o) =XYZ Inc., and its cost is 5. The third is a connector to MCI; it has an address space of * and a cost of 50. If you send a message to someone at XYZ, the least-cost route to get it there is directly over the XYZ connector, so the message will follow that path. If you send a message to someone else, the least-cost route is through the backbone connector. If that connector is unavailable, the MTA will use the next available path—through MCI—instead.

Can you give me a more complex example?

You've already seen how messages flow within a site. The picture expands when you want to send mail outside your site, either through a connector or to another Exchange site. Let's say you're forwarding a great joke to your mother, your boss, your cubicle mate, two coworkers, and your email administrator. To make things interesting, let's say that Mom is a custom recipient with an SMTP address; you, your cube mate, and your boss are on the same Exchange Server server; your email admin is at your company's main site in Atlanta; and your two coworkers have Lotus cc:Mail. You compose your message and send it. What happens next?

How can I troubleshoot routing problems?

Two indicators signal routing problems. The most obvious indicator is an NDR. The other indicator is mail not reaching its destination. Here are some basic steps you can use when you suspect a routing problem.

What does ESEUTIL do?

ESEUTIL, the Exchange Server 5.5 database-maintenance utility, is the descendant of EDBUTIL, the maintenance tool that shipped with Exchange Server 4.0 and 5.0. ESEUTIL has six primary functions:

Why is defragmentation necessary?

As new data arrives, Exchange Server has to expand the Information Store database files to accommodate it. This expansion causes the database files to grow on disk, in the same way that a balloon inflates when you blow air into it. However, let's say that in a typical day your store receives 100MB of mail and users delete 80MB of it. You might guess that the total change in Information Store size is only 20MB, but the actual change in the size of the database on disk could range from zero to 100MB. When users delete items, Exchange Server doesn't return the newly freed space to the operating system (OS); instead, it keeps the space and recycles it as users add new data to the Information Store.

Over time, the amount of white, or unused, space in the database fluctuates as users add and delete items. For best performance, Information Store items need to be in contiguous chunks in the database file, but because Exchange Server allocates space on a first-come, first-served basis, a contiguous arrangement isn't always possible. The result is a fragmented database, which slows down database performance. In some cases, your Information Store files might also have white space that the OS or another application could use.

When is running the Exchange Server database-maintenance utilities appropriate?

This question provokes a lot of discussion, but Microsoft has a particular position: Don't run ESEUTIL or EDBUTIL unless it's absolutely necessary! This edict might seem draconian, but Microsoft has a reason for it. Any time you take Exchange Server offline and change the database, you're taking a chance that it won't come back up. Why put your messaging system at needless risk?

What maintenance is absolutely necessary?

The obvious do-or-die time to use ESEUTIL or EDBUTIL is when you're trying to restore a failed server. Administrators also typically use these tools after they've discovered a Directory or Information Store corruption error. (Ideally, they discovered the corruption early and not after they backed up the corrupt database onto the entire backup rotation.) Although you can fire up the tool as soon as you notice an error, you're usually better off calling Microsoft Product Support before you do anything. Product Support specialists have more experience isolating database problems than most administrators, and their advice can save you a lot of trouble.

For Exchange Server 4.0 and 5.0, you might benefit from using EDBUTIL to defragment your database from time to time because these versions don't have good online defrag capabilities. Exchange Server 5.5 is another story: Its online defragmentation is much more efficient, so you have little reason to take down an Exchange Server 5.5 machine just to do an offline defrag.

Another reason you might want to run ESEUTIL or EDBUTIL is to defragment your disk if you've removed large amounts of data from your store. For example, you might have moved many mailboxes to another server or removed a set of Usenet public folder replicas. In this case, running an offline defrag can recover a significant amount of database white space. However, as new data arrives, the store's size will grow again. So unless you really need the space, recovering the space won't help you for long.

If you decide to take the plunge, make sure you have enough disk space. Defragmenting a store requires 110 percent of the store file's size in free disk space; for example, if your store is 2GB you'll need 2.2GB free. Check Microsoft Support Online articles Q183888 (http://support.microsoft.com/support/kb/articles/q183/8/88.asp) and Q182903 (http://support.microsoft.com/support/kb/articles/q182/9/03.asp) for more details about online and offline defragmentation.

We have to occasionally remove mailboxes for people who have left the company. How can we send back a message to the sender explaining that the mailbox owner has left the company instead of sending an NDR?

Most humans will figure out the situation when they get an NDR that says that an address isn't valid. However, spelling it out might be more helpful, and it's fairly simple to do. Start by creating a new account, and give it an inbox rule that answers all mail with a short note; the note can say whatever you want. When someone leaves the company, reassign the mailbox to the new account by using the "Primary Windows NT Account..." button on the General tab of the properties dialog in Exchange Administrator. When people send mail to the former employee, the rule will return them a short "Alice doesn't live here any more" message. After a month or two, you can safely delete the mailbox, after everyone's gotten the word.