Queuing Data with Microsoft Message Queue Server

Get in line to improve your business-critical data delivery

When developers create transaction-processing (TP) applications (e.g., for exchange of stock, banking transactions, or shop-floor control), they need to make sure that such applications not only process transactions accurately but also transfer data from one process to another safely and in an orderly way. Data that is lost or disordered when it traverses a network defeats a TP application's purpose. Therefore, developers often use message-queuing technology to guarantee reliable data delivery in TP applications. A message-queuing system increases the reliability of interprocess communication by using a sending process to place data in a queue, and a receiving process to retrieve the data from the queue.

Traditionally, developers have either developed their own message-queuing systems or purchased such systems from a vendor. Developing a sophisticated message-queuing system requires developers to be versed in underlying network communication such as transport protocols and message routing, necessitating long development cycles.

Although third-party technologies offer immediate solutions, buying and maintaining such a technology can be expensive. Microsoft has built its message-queuing technology, Microsoft Message Queue Server (MSMQ), into Windows NT Server 4.0, Enterprise Edition (NTS/E) and Windows NT Server 4.0 Option Pack. With MSMQ, you can design and implement an enterprisewide message-queuing system to support all message-queuing applications in your NT network. MSMQ's software development kit (SDK) lets you develop custom message-queuing applications without the need to program direct interprocess communication or to know underlying network communication. MSMQ is a crucial vehicle for message exchange in your NT network. In this article, I'll help you understand message-queuing technology and MSMQ's unique application of that technology.

What Message-Queuing Does
When application data traverses a network, the data is often referred to as messages. A queue or message queue is a holding place in the network that temporarily stores traveling messages. A queue can reside in a computer sending a message, in a computer receiving a message, or in an intermediary computer between the sending and the receiving computers. A message-queuing system mainly manages queues, messages, and message-queuing servers and clients that create, delete, locate, store, forward, and route queued data.

The message-queuing system guarantees safe and orderly delivery of messages. With this system, a TP application can transfer data from one process to another reliably and process transactional requests accurately. For example, a stockbroker can place customer trade requests in a queue using a stock-trading application with message-queuing enabled. A trade server takes the orders from the queue and processes them without losing or mishandling an order.

Applications that can use asynchronous communication, such as email and Help desk applications, can also use the message-queuing system. After these applications send data to a queue, they don't need to wait for a response before carrying out other tasks. Message queuing also supports mobile applications. For example, when a mobile salesperson enters an order on a laptop at a customer site, the ordering application sends the order to a local queue in the laptop. When the salesperson returns to the office and connects the laptop to the network, the order-receiving application on the server will automatically pick up and process the order from the queue in the laptop.

The MSMQ System
MSMQ is Microsoft's implementation of message-queuing technology for NT networks. An MSMQ system consists of an MSMQ enterprise, which contains one or more sites. An enterprise site contains one or more MSMQ computers, both servers and clients. MSMQ servers run on NTS/E, or on NT Server 4.0 if you install MSMQ from the Option Pack. MSMQ clients run on NT 4.0 or Windows 95 computers. Each MSMQ computer (except dependent clients) can contain message queues. In Screen 1, the left panel shows an MSMQ enterprise, Big Company. Using Big Company as an example, let's take a close look at the MSMQ system.

Enterprise. An MSMQ enterprise comprises the computers in your network that run MSMQ. All MSMQ computers in an enterprise exchange messages directly or through MSMQ servers. Microsoft recommends that one company establish only one enterprise. MSMQ provides an administration tool called MSMQ Explorer, which lets administrators manage the enterprise from the single console Screen 1 shows.

Sites. An MSMQ site is a collection of MSMQ computers that communicate with one another through fast links. A site often is a LAN or a physical location. Links between sites are slow, using, for example, WAN connections. Big Company has three offices: one each in Chicago, Los Angeles, and New York. Thus, Big Company has three sites--Chicago, LA, and NY--that MSMQ defines by location. If you have deployed Exchange Server in your network, your MSMQ sites can be the Exchange Server sites. You connect two MSMQ sites through a site-link, which has an associated cost you define according to the link's speed, with numbers from 0 to 999,999. The lower the cost figure, the faster the link. A designation of 0 means that no direct link exists between two sites.

Connected networks. An MSMQ connected network (CN) is a collection of MSMQ computers that communicate with one another directly, which means that the computers in one CN use an identical network protocol. MSMQ supports only IP and IPX. You can have multiple CNs in one site or one CN that crosses multiple sites. One computer can belong to multiple CNs if it is multihomed, runs both IP and IPX, or has multiple IP addresses on the same NIC. In a small IP network, such as Big Company's, MSMQ lets you use only one CN in the entire enterprise.

Servers. MSMQ uses four kinds of servers to manage message queuing: Primary Enterprise Controller (PEC), Primary Site Controller (PSC), Backup Site Controller (BSC), and Routing Server (RS). An MSMQ enterprise can contain only one PEC, which holds the enterprise topology information, such as the sites, site-links, and CNs in the enterprise. The first MSMQ server you install in your network is the PEC. Each site can contain only one PSC to hold the site information, such as servers, clients, and queues in the site. The PEC is also the PSC of a company's first site.

For fault tolerance and load balancing, an enterprise needs at least one BSC per site. Each BSC holds a read-only copy of its site's information. RSs belong to specific sites and support message routing between sites, CNs, and MSMQ computers. A site can support any number of BSCs and RSs. RSs don't hold a local copy of enterprise and site information but can obtain that information from a PSC or BSC. All controllers support message routing. All MSMQ servers can contain local queues to store and forward messages.

MSMQ stores enterprise and site information in the MSMQ Information Store (MQIS). The PEC and PSCs replicate their maintained information to one another throughout the enterprise, and each site's PSC replicates its information to all BSCs in the site. This system of replication guarantees that every MSMQ controller in an enterprise contains a complete copy of the MQIS. MQIS is a Microsoft SQL Server database, and you must install SQL Server 6.5 (SP3 or later) on all your MSMQ controllers. However, MSMQ in NT 5.0 uses Active Directory (AD) as a repository to store MQIS, eliminating reliance on SQL Server.

Clients. MSMQ includes two types of clients: dependent and independent. Dependent clients rely on a designated MSMQ server to send and receive messages. MSMQ refers to such designated servers as supporting servers. Any MSMQ server in any site can be a supporting server. Each supporting server can support up to 15 dependent clients. Without a constant connection to its supporting server, a dependent client can't function.

In contrast to dependent clients, independent clients can function without an MSMQ server. With an independent client you create queues in the local machine and store messages in the local queues. Dependent clients work well for desktop users with a fast and stable network connection, and independent clients work well for mobile users or remote office users without access to a local MSMQ server. The independent client includes the MSMQ SDK, so you need to install the independent client for your application developers.

Queues. MSMQ offers two major queue types: public and private. MSMQ publishes public queues in the MQIS. MSMQ servers and independent clients have three special preinstalled public queues: journal, dead-letter, and transaction dead-letter (Xact Dead Letter in Screen 1). MSMQ logs copies of messages that a computer receives and sends in the computer's journal queue, stores expired or undeliverable messages in the dead-letter queue, and stores expired or undeliverable transactional messages in the transaction dead-letter queue. Microsoft Distributed Transaction Coordinator (MS DTC), a component of MSMQ, ensures that a transactional message operation either succeeds or fails.

To create a public queue on an MSMQ server and independent client, you use MSMQ Explorer or MSMQ SDK. For example, Big Company created a trading public queue on its MSMQ server bigserver2. The right panel of Screen 1 displays some messages from a stock-trading application that bigserver2 contains. Because private queue information is not part of the MQIS, you can't use MSMQ Explorer to create a private queue. You must use MSMQ SDK to create a private queue.

MSMQ Message Exchange
The MSMQ system provides a platform for message-queuing applications, whereby an application uses MSMQ message queuing to reliably send and receive data. All message-queuing applications share the same logical procedure for message exchange, which Figure 1 shows.

When a sending application needs to send a message, it either creates a new queue or locates an existing queue to send the message to. The application locates a public queue by running a query on the queue information registered in the MQIS. Queue queries consist of a queue name, type, or specific attribute. Because the MQIS doesn't store private queue information, an application has to specify a private queue's full path (in the form of machinenamequeuename) to find a private queue. When an application finds a queue, it opens the queue, sends one or more messages to the queue, and then closes the queue.

When a receiving application needs to receive a message, it uses the same queue-locating method as the sending application used to find a queue. Receiving applications create a queue if no queue exists. After the receiving application locates or creates a queue, it opens the queue, receives messages, and closes the queue. You can initialize receiving applications any time your business allows. If no message is in the queue the receiving application opens, the application waits until messages arrive.

MSMQ messages contain a message body and control information. You can include any text or binary application data, such as "Buy 10 shares of stock ABC," in the message body. The control information defines how MSMQ needs to handle the message during delivery, and includes such information as message priority and expiration time.

The MSMQ SDK includes a set of C/C++ and ActiveX API functions you can use to write an application with message queuing enabled. The SDK is easy to use, because you need only five functions to complete the message-exchange procedure: create queue, locate queue, open queue, send message, and receive message.

MSMQ Routing
How MSMQ routes a message from the sender to the queue and from the queue to the receiver is transparent to the message-queuing application. Understanding how MSMQ routing works can help you design an optimal MSMQ system.

When one computer needs to communicate with another in an MSMQ enterprise, MSMQ first tries using the underlying network protocol to establish a direct session between the computers. If MSMQ can't set up such a connection, it will use its routing mechanism. For example, suppose a sending computer runs IP but the destination computer uses IPX, and both computers are in separate CNs. An MSMQ controller or RS that supports both IP and IPX and belongs to both CNs will route the message for the two computers. Another example is when a destination computer is offline or unreachable. The MSMQ takes the message from the source computer and stores it in an MSMQ server for future forwarding. MSMQ always tries to use the shortest routing path for intrasite communication. When MSMQ routes a message from one site to another, it uses the site-link costs you defined to locate the intersite routing path with the lowest cost (i.e., the fastest path).

In addition to native routing, MSMQ provides a routing mechanism called session concentration, which you use to optimize your MSMQ network. When you use session concentration, you can reduce the number of connections, or sessions, that occur within a site or between sites. For example, suppose 10 independent clients in an MSMQ site need to communicate with one another constantly. The site will contain a total of 10 * 9 sessions (each of the 10 clients communicates with 9 other clients). You designate an MSMQ server to serve as the message hub for the 10 computers, so that all incoming and outgoing messages for the 10 computers must go through the server. By routing the 10 computers' messages through the server, you reduce the number of sessions from 90 to 10. To build fault tolerance and enable load balancing, you designate two message hubs for an independent client--one hub for incoming messages and one hub for outgoing messages.

Using the message-hub concept, you can apply session concentration to intersite communication by designating an MSMQ server as the site gate from the local site to a neighboring site. All of the intersite messaging traffic must pass through the site gate. If each of two sites has a site gate, the number of sessions between the sites reduces to one. For fault-tolerance and load balancing, assign two or more site gates to a site.

In Business with MSMQ
MSMQ is well designed for the enterprise. A business-critical application can take advantage of cluster-aware MSMQ and Microsoft Cluster Server (MSCS) for high availability. MSMQ with Microsoft Transaction Server (MTS) can meet the high-reliability requirements of TP applications. MSMQ is a highly secure system in which you can define access control to individual components in the MSMQ system.

MSMQ supports message authentication and encryption through digital certificates. You use this security feature in your message-queuing system and TP applications to protect business data across the network. Although MSMQ runs only in an NT network, it can work with foreign message-queuing systems, such as IBM's MQSeries, through third-party support such as Level-8.

As message-queuing technology increases in importance for business applications, approximately 50 vendors and companies--including AT&T, HP, IBM, Intel, and Microsoft--have formed a Business Quality Messaging (BQM) special interest group to accelerate the adoption of message-queuing technology in business. In March, the group introduced a BQM standard for reliable data delivery between companies over the Internet through a BQM service provider. You can inspect the details of the standard at http://www.bqm.com. MSMQ is obviously an outstanding product for BQM services.