What to Do When Your IMS Fails

The Internet Mail Service (IMS) is a crucial piece of Compaq's internal messaging infrastructure. Compaq has at least three Exchange Server organizations. Two organizations support large user populations, and the third organization includes Compaq Services consulting groups' servers and sites worldwide. The IMS ties the organizations together and connects Exchange Server to other external and internal Simple Mail Transfer Protocol (SMTP) messaging systems.

Any IMS outage affects the flow of messages and causes queues to build up quickly. To help you handle IMS malfunctions, I'll explain how the IMS works with the Information Store (IS) to move and store messages, and why problems can occur. Then, I'll show you how to troubleshoot a problem, drawing—as I frequently do—on a situation I dealt with recently.

The IMS and the IS
In Exchange Server 4.0, the Internet Mail Connector (IMC) supported SMTP. In Exchange Server 5.0 and subsequent releases, Microsoft renamed the connector to the IMS, for two reasons. First, the IMS does more than a simple connector does. For example, it reroutes messages for Post Office Protocol 3 (POP3) clients. Second, Microsoft has integrated the IMS more closely with the IS. The IMC uses a set of subdirectories under the imcdata root to store messages and queue data, whereas the IMS uses a set of hidden folders in the IS to store queued messages. The result is that you see an IMS mailbox when you view mailbox resources for a server through the Microsoft Exchange Administrator program. (By the way, in Exchange Server 5.5 Service Pack 1—SP1—Microsoft fixed the bug that wildly inflates the resources the IMS mailbox uses. Now, you don't have to worry about the IMS using 50MB.)

Even with its closer relationship to the IS, the IMS uses the imcdata subdirectories for some tasks. For example, if you turn on diagnostic logging, you can capture a copy of every incoming and outgoing SMTP message on the server. Exchange places copies of these messages in the inarchive and outarchive subdirectories. However, you probably don't want Exchange to fill your disk with copies of messages. Message journaling, a new feature in Exchange Server 5.5 SP1, can capture copies of messages from all sources and archive them to a mailbox or public folder. However, making copies of every message generated on a busy server will make finding a specific message difficult. Products such as Compaq's Enterprise Vault are more efficient at archiving messages.

Using the IS for queued messages integrates the IMS more tightly into the heart of Exchange, but this strategy has side effects. The most important effect is that a problem with the IMS can cause the IS to terminate abruptly. This problem requires immediate attention, because an IS failure disconnects all client activity, causing the phones to ring at your Help desk.

A Bad Message
Most IMS problems stem from messages that the IMS regards as bad, that is, messages that don't obey the rules for the internal format of SMTP messages. The Internet Engineering Task Force (IETF) has published the rules in several Request for Comments (RFCs). RFCs 821 and 822 laid down the original structure for SMTP messages, but subsequent RFCs have incorporated many changes. For example, RFCs 2045 through 2049 define how to implement MIME encapsulation to let SMTP messages carry attachments, such as Microsoft Word documents or Microsoft Excel worksheets.

Because no central certifying body verifies that all SMTP implementers comply with the standards in the RFCs, minor disagreements in interpretation inevitably occur. The result is that an SMTP mail server can't understand the structure or format of a message that another server presents. Software logic that attempts to understand what the originating server is sending and accepts the incoming message can handle minor differences in messages' structure or content. However, no software can understand everything, so sometimes an SMTP server can't deal with an incoming message.

In my case, the problem started when the IMS crashed after a bad SMTP message arrived at the server. The IMS can process most messages that arrive on a server; perhaps only one message in tens of thousands causes a problem. When a malformed message arrives, the IMS captures a copy of the message in a folder called Bad in the Exchange mailbox. In addition, the IMS issues event number 4116 to the Windows NT application event log, as Screen 1 shows, explaining that the IMS hasn't delivered a message.

Sometimes the IMS issues event 4117 instead of event 4116. Event 4117 means that an error occurred when the IMS was accepting a message from a remote SMTP server. The IMS saves a copy of the complete message in a spool file in the imcdatainarchive directory. The Microsoft article "XADM: Bad Message Causes Dr. Watson in Store.exe" (http://support.microsoft.com/support/kb/articles/q169/2/23.asp) documents this problem.

Based on the events in the log, incoming messages apparently were causing the IS a heap of problems. Events 4116 and 4117 were appearing with ominous regularity. Dr. Watson reported that exceptions had occurred in store.exe that forced the IS to shut down. An exception means that the code in a program has encountered a condition that it can't handle. In other words, an error has occurred, and the program doesn't have a GOTO followed by some code and a RESUME.

The event 4116 and 4117 messages tell you to examine the contents of the spool files in the imcdatain directory (which the IMS creates to accept incoming messages) to see if anything is obviously wrong. You can use an editor such as Notepad or WordPad to open the spool files and view everything except the MIME-encoded body parts (e.g., attachments).

However, most systems administrators don't know the relevant RFCs well enough to analyze SMTP message contents to see if anything's wrong. In this situation, you have two options. First, you can restart the IMS and hope that reinitializing the IMS will fix the problem. This approach is a variation on the old turn-off-and-on solution that systems administrators have used for years to solve hardware problems. If you're successful, the IMS will detect the files in the imcdatain directory and process them after the IMS has restarted.

The second option, which you usually try if the first option fails, is to remove all the files from imcdatain. The rationale for removing the files is that a problem exists in one of these files and that removing them all will let you get the IMS up and running again. Later, when things settle down, you can review the content of the bad files and advise people on the recipient list that a message addressed to them didn't get through and the sender needs to resend it.

In my situation, the restart-and-hope solution didn't work. The IS restarted successfully, but each time I tried to restart the IMS, it failed just after initialization. The option to remove the files from imcdatain didn't help either, because the IMS had accepted the incoming message and added the message to its internal queue within its IS mailbox. Exchange Server stores the queue for incoming messages in the MTS-IN folder in the IMS mailbox. Each time a thread in the IMS attempted to process the offending message, store.exe had an exception and shut down.

I gained no insight from the entries in the application event log, which displayed the same entries as event 4116 just before the IS terminated. As a side effect, each time I restarted the IMS, the service retransmitted some messages, which precipitated many recipient complaints. To eliminate the offending message, I had to look more closely inside the IMS.

The Mail Database Viewer
Usually, the correct response to event 4116 is to use mdbvu32.exe, the Mail Database Viewer utility from the supportutils folder on the Exchange Server CD-ROM, to review the content of the message and decide what to do with it. In my case, I wanted to use MDBVU32 to access the contents of the MTS-IN folder in the IMS mailbox.

To access a mailbox, you first create a Messaging API (MAPI) profile to use with MDBVU32. You create the profile on the server where the IMS resides and specify the name of the IMS mailbox. However, Exchange hides the IMS mailbox from view, and you can't access it in the same way you access a user mailbox. The Microsoft article "XFOR: How to View/Delete Messages in MTS-IN and MTS-OUT Queues" (http://support.microsoft.com/support/kb/articles/q165/5/05.asp) explains how you can create a special profile to access the IMS mailbox, either by using the PROFINST utility or by creating the profile directly in the system Registry. You can obtain PROFINST from the Microsoft BackOffice Software Development Kit (SDK) or from http://support.microsoft.com/support/downloads/LNP244.asp.

Because you don't always have access to PROFINST, knowing how to amend the Registry is useful. If you use regedt32.exe to examine the key HKEY_CURRENT_USERSoftwareMicrosoftWindows NTCurrentVersionWindows Messaging SubsystemProfiles, you'll find a list of all the MAPI profiles defined on the system, as Screen 2 shows. The IMS uses the profiles with an MSExchangeIMC prefix; several profiles exist to allow multithreaded access to IMS structures.

You can use an existing IMS profile with MDBVU32, but you might affect a system profile by working with it. Therefore, select any of the MSExchangeIMC profiles, and save the contents of the key with the Registry, Save Key command. Then, use Edit, Add Key to create a new key called IMC Profile (or a similar name), and use Registry, Restore to restore the data in the saved key to this key. The result will be similar to what Screen 3 shows. These steps duplicate a profile that can access the IMS mailbox.

MDBVU32 is a test utility, so its user interface is a little esoteric. When the utility starts, it usually presents the same dialog as any other MAPI client to let you select the profile to use. If the dialog doesn't appear, reboot the server. A workaround is to start up another MAPI client (Outlook or Windows Messaging) and use it to connect to the right profile. Then, start MDBVU32, select the MAPI_Explicit_Profile check box, and click Return. MDBVU32 will use the profile that Outlook selects, and you'll connect to the IMS mailbox.

Select the OpenMessageStore option from the MDB menu to open Mailbox - Internet Mail Service. Then, choose MDB, Open Root Folder. All the folders in the mailbox appear in the Child Folders window. Double-click any of the folders to display the messages within. Screen 4 shows the contents of the Bad folder. As you can see, Exchange has moved several messages into the folder.

Before you decide what to do next, you need more information. If you double-click a message, MDBVU32 displays its MAPI properties, as Screen 5, page 4, shows. MAPI properties include the recipient list, the author, when the user sent the message, and the presence of attachments. These properties give you enough information to tell a user to resend the message.

In my situation, a message in the Bad folder didn't cause the problem; through trial and error, I found that a message in the MTS-IN queue was the culprit. Viewing the contents of the MTS-IN folder, noting the details of the messages, and then deleting the bad messages from the folder (and the queue) solved the problem. In Screen 5, note the Operations available option. The drop-down menu contains a set of MAPI operations, including Delete Folder and Move Messages. In this case, I used the entry

lpFld->DeleteMessages()

which instructs MDBVU32 to delete all selected messages from the folder when you click Call Function. Another way to delete all the messages from the queue is to use Exchange Administrator to temporarily set the IMS Transfer Mode property to None (Flush Queues), as Screen 6 shows. However, this operation flushes both the MTS-IN and the MTS-OUT queues, and you might lose more messages than you expected to. After I removed the messages from the queue, the IMS and the IS restarted successfully.

Another Queue Cleaner
You can also use the GWCLEAN utility to remove messages from the MTS-IN or MTS-OUT queues. The code for GWCLEAN is in the Microsoft Platform SDK. GWCLEAN moves the messages to a .pst file. Like MDBVU32, GWCLEAN uses a profile to connect to the IMS mailbox and access the messages. GWCLEAN is a good option if you want to move all the messages out of the queue in one operation. However, MDBVU32 shows you more of the internal workings of the IMS mailbox. The syntax for GWCLEAN is

GWCLEAN Profile Name [/Days= /File= /Folder= /HELP]

where Profile Name is the profile that you created to access the IMS mailbox, /Days= is the number of days remaining before the utility will move all messages, /File= is the name of the destination .pst file for the messages, and /Folder= is the name of the folder where the messages currently reside (e.g., MTS-IN).

Be Prepared
The Exchange Server Resource Kit and the support directory of the server CD-ROM contain many interesting tools. However, you usually don't examine them until you have a problem, and the IMS—especially in Exchange Server 5.5 or later—seldom experiences problems. So, take a look at the tools you have and be prepared for the day you meet a malformed SMTP message.