Reap the Power of MPS_Reports Data

Years ago, Microsoft support engineers had no single data-gathering tool to use in resolving customer issues. The process of collecting system data was time-consuming, laborious, and unreliable. Depending on the problem, customers would run various data-gathering utilities, manually extract the event logs, and copy log files from the machine having the problem, then email the data back to us. A group of support engineers decided there was a better way and created the Microsoft Product Support Reporting tool (MPS_Reports) to aid in the data-gathering process and help resolve customer issues faster. As I'll demonstrate, MPS_Reports is a tool that you, too, can use in your Windows troubleshooting. Let's explore the tool itself, the information it provides, and how you can use it when troubleshooting common system problems.

About MPS_Reports
You can find a description of MPS_Reports and links to download the tool here. As you'll see on the MPS_Reports web page, there are several different versions of the tool. This is because the specialty groups within the Microsoft support organization wanted to collect information for their particular areas of responsibility. Thus, each group has its own specialized version of MPS_Reports.

Under the covers, the tool comprises several utilities bundled into an IExpress package, which executes the utilities through a .cmd file. IExpress is a utility that lets you create a self-extracting setup program via a wizard interface. IExpress also comes with Windows 2000 and later OS versions. You can learn more about IExpress technology here.

The data from MPS_Reports comes in easy-to-read formats compressed into a .cab file, so you can conveniently transport the data from one system to another. When MPS_Reports runs, you'll notice a DOS-style window pop-up that shows you the progress of the tool in terms of which data is currently being gathered.

When MPS_Reports has finished collecting data, the DOS window will disappear and you'll see the .cab file's directory folder containing the data collected from your system. You can also view the extracted reports under the Reports folder located in the same WindowsMPS_Reports directory.

Note that a new version of MPS_Reports is scheduled for release in April 2009. This new version will be compatible with Windows XP and later, including Windows 7, and available at http://www.microsoft.com/downloads. Included on that page will be an email address and community forum to which you can submit questions about how to customize the data-gathering functionality for your environment.

Using MPS_Reports Data
Using MPS_Reports can save you time on a call to Microsoft support. On a support call, the first thing a support engineer typically asks you to do is to run MPS_Reports, so if you run the tool before calling the support team, you'll save time by having the .cab file ready to send to us. Another benefit of MPS_Reports is that it provides data you can use in solving the following types of problems on your own.

Problem 1: Comparing File Versions on Two Cluster NodesUnderstanding which files differ between a system having a problem and another system that isn't having a problem is often an essential step in solving a variety of different issues. With a little work, you can use MPS_Reports data to create an easy-to-read Microsoft Excel PivotTable that shows you the differences between the systems' files.

To do so, first run MPS_Reports on the two systems you want to compare and extract each .cab file to a folder named for the source computer the data was collected from. In this case, the .cab files come from a two-node cluster; the systems are called NODE001 and NODE002. For this example, I extracted the two .cab files into the directory D:MPS_Reports_example.

Once you have the .cab files, you can run the Findstr command to merge data from multiple .csv files contained in the .cab file. In this example, I want to compare the drivers loaded on both nodes, so I run the following command:

D:MPS_Reports_example>findstr /s  *_DRIVERS.CSV > DRIVERS.CSV

This Findstr command searches the current folder and all subfolders for the string "" in files named *_drivers.csv. This results in the data from multiple MPS_Reports .cab files being merged into one file. Figure 1 shows some sample output from the Findstr command.

Figure 1: Sample Findstr output showing merged .csv files

NODE001fileversNODE001_DRIVERS.CSV:,,,C:WINDOWSSYSTEM32NTKRNLPA.EXE,SYMBOLS_PDB,2358909,1173096985,"March 05, 2007 04:16:25", 2449408,,ntkrpamp.pdb,{52A81696-FB5B-432A-B0B1-B46297F104D3}, 1,5.2.3790.4035,5.2.3790.4035 (srv03_sp2_gdr.070304-2240),Microsoft Corporation,NT Kernel &System,2321408,29843232,613059328,"March 5, 2007 5:16:46 AM",

Notice that the output from Findstr shows the folder path for the drivers.csv file. From this information, we can obtain the computer name. To do so, load the resulting .csv file into an Excel spreadsheet, as I've done in the Drivers.xlsx spreadsheet in Figure 2.

Figure 2: File differences between two nodes in a cluster

Use the Text to Columns feature to extract only the computer name from the first column of data in the Findstr output. The remaining columns are loaded into cells because of the comma delimiter. Now each line of data can be associated with the computer from which it was collected.

Next, insert a blank row at row 1 of the spreadsheet and fill in the column headers. Then, using Excel's PivotTable feature, you can do a side-by-side comparison of the drivers loaded on each node. (For more information about using Excel PivotTables, see articles listed in the Learning Path box.) In Figure 2, you can see that NODE001 is running older versions of Ntfs.sys and ntkrnlpa.exe. For a cluster, it's important that the same version of drivers run on all nodes.

Problem 2: Finding Examples of Malware
Malware can cause various problems on your systems, among them sluggish behavior, networking issues, and application failures. One of the pieces of data that MPS_Reports provides is a system report file called computername_msinfo32.nfo. You can use this .nfo file to scan through the Startup Programs section under the Software Environment heading, which lists all executables that launch at system startup. As Figure 3 shows, two executables listed were unfamiliar to me: *mcwave and wtoolsa.

Figure 3: Msinfo32 report showing two malware programs

I searched on the program names and found links to two sites that confirmed that these programs were malware: http://www.liutilities.com/products/wintaskspro/processlibrary/wtoolsa and http://www.spywaredata.com/spyware/malware/mcwave.exe.php.

Problem 3: A cluster node fails to join an existing cluster due to an RPC error.
A customer recently reported a critical issue with a cluster node being evicted from an existing cluster. This problem occurred on a newly installed system during installation of the cluster service. The customer-reported error was The remote procedure call failed and did not execute. This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.

Upon receiving the MPS_Reports .cab file from the customer, we immediately went to the Cluster log file because it contains verbose error descriptions about problems occurring during setup. The cluster logs revealed the following additional information: 00000ab0.000009ec::2009/01/03-13:42:45.328 INFO Network Name : Name MYCLUSTER failed IsAlive/LooksAlive check, error 1450. By running the following Net helpmsg command, from the command's output we determined that the error 1450 was caused by insufficient resources:

C:et helpmsg 1450Insufficient system resources exist to complete the requested service.

Using this information and our knowledge that the customer experienced the problem shortly after rebooting the system, we started investigating potential causes of a memory constraint on a system immediately after a reboot. Such conditions would include any memory parameters passed into the boot.ini file, such as the /3GB switch, which allocates less memory for kernel-mode drivers and more memory for user-mode applications such as database or email.

Since MPS_Reports collects the boot.ini file included in the .cab file, we quickly noticed that the customer was running with the /3GB switch. We recommended that the customer follow the guidance in the Microsoft article "How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB", which suggests modifying boot.ini to allow more precise tuning of user-mode memory for applications that require more than 2GB (the default without using the /3GB switch). After we added the /userva switch with the recommended 3030 value, the node could join the existing cluster and the critical problem was quickly resolved.

Dig into the Data
Knowing what to look for and in what files takes some time and practice. However, once you start digging into the data, you'll find a treasure chest full of useful information that may prevent a support call or at least help expedite the resolution process.

Special thanks to Gary Braughton, a Microsoft senior escalation engineer, and Sean Wheeler, a Microsoft senior premier field engineer, who contributed to this article.