Comparing Windows Azure Queues to Azure AppFabric Durable Message Buffers

An update to the Windows Azure AppFabric Service was announced at Microsoft Professional Developers Conference 2010 (PDC10). The update touted a new feature called Durable Message Buffers in the AppFabric Labs environment. Of course, a queue by any other name is still a queue. And who knows? The Durable Message Buffers name might still change before a production release. But whatever the name, the question is this: How does the Durable Message Buffers feature differ from Windows Azure Queues?

In this article, we will compare the salient features of both to help you make an informed decision. Obviously, if you have a production need for queues now, your choice is simple: Only Windows Azure Queues is currently released to production, and it has been for some time. However, if you are in the planning stages of a project, read on!

Basic Queue Operations
From a high level, the purpose of a queue is to provide you with a FIFO structure that usefully organizes the way in which messages are inserted by producers and processed by consumers. Therefore, for any queue-based system, you will see operations that send messages to the queue (also known as adding, enqueuing, or putting) and that receive messages (also known as removing, dequeuing, or getting). In general, the receiving operation is really two parts: One part gets the next message, the second part is an operation that deletes the message when the operation completes successfully. Or, it returns the message to the queue if processing fails, such as when the consumer crashes or when the consumer is not the right one for the message. These two high-level operations surface as various operations in both Azure Queues and Durable Message Buffers.

To interact with Azure Queues, you can make direct REST requests or use the Windows Azure Storage Client Managed Library to build those REST requests for you. When you receive a message from the queue via the Get operation, the message is made invisible in the queue. Other consumers of the same queue will not be able to request it for a period of time. When the consumer has successfully processed the message, the Delete operation can then be used to remove the message from the queue. To send messages to the queue, producers must perform a Put operation.

As of this writing, Durable Message Buffers can be accessed only by using REST requests. The operation to receive a message from a buffer is peek/lock, which locks the message in the buffer for a period of time, making it effectively invisible to other consumers. When the consumer has finished processing the message, he or she can use the Delete operation to remove the message from the buffer. To send a new message to the buffer, use the Send operation.

Azure Queues and Durable Message Buffers differ in their handling of locked or invisible messages. When a message is not processed within the expected time period, both protocols return the message to the queue for other consumers to process. However, only message buffers let the consumer explicitly unlock the message instead of waiting for the timeout to expire. Moreover, only message buffers let the consumer receive a message from a buffer and delete it from the buffer as an atomic operation. On the other hand, if in the consumer wants to simply see what messages are in the queue, without preventing other consumers from processing (and subsequently deleting) them, a Peek operation is needed. Only Azure Queues offers this functionality.

Queue Management
The queues in Azure Queues are only partially managed through a user interface (UI). The production Azure Management Portal, as shown in Figure 1, lets you create storage accounts (e.g., set the URL and region or affinity group) and configure a custom domain.
 

Figure 1: Windows Azure Management Portal in production

However, it does not provide tools for creating, configuring, inspecting, or deleting queues. For this, you are expected to use the management APIs directly or graphical tools from third-party vendors, such as Cerebrata's excellent Cloud Storage Studio.

For Durable Message Buffers, there is an equivalent basic UI in the AppFabric Labs (see Figure 2) that enables namespace creation and security configuration via the Access Control Service.
 

Figure 2: AppFabric Labs Management Portal

Here, too, you need to create queues via API calls. Note that there is currently no UI to create, view, inspect, reconfigure, or delete queues. Because both Azure Queues and Durable Message Buffers require APIs for management, it's worth taking a look at some of the differences between the two.

In Azure Queues, you again have the option of either making REST requests or using the Storage Client. Durable Message Buffers currently limits you to crafting the REST requests yourself. Both protocols offer operations to create and delete queues, but with some differences. In Azure Queues, you can add up to an additional 8KB of name-value pairs as metadata for the queue via Get Queue Metadata, by using Set Queue Metadata. The Durable Message Buffers API doesn't offer a similar construct. It does, however, let you request a buffer's description via the Get Buffer operation and get a list of buffer descriptions via its List Buffers operation. Currently, only Azure Queues provides a mechanism to retrieve the approximate count of messages (via the Get Queue Metadata operation).

Quotas and Limits
Although basic operations and management help you to understand the functionality of Queues, the useful insights really come from understanding the various quotas and limits imposed by each. Quotas and limits will definitely affect your solution's architecture.

The quota that is most likely to affect you is the maximum size of any message that can be sent to the queue. In Azure Queues, the max size is 8KB; but, in effect, you get less than this because of the Base64 encoding of the message payload. This quota is in sharp contrast to that of Durable Message Buffers, which supports messages up 256KB. Obviously, the extent to which your messages will be larger than the max message size directly correlates to how much work you will have to put into an alternative, such as having the message only point to data stored elsewhere or splitting up large messages into multiple smaller ones.

The quota that is the next most likely to affect your design is the upper limit on the time for which a lock can be held on a message that is being operated on. For Azure Queues, the maximum is two hours, but for Durable Message Buffers it is only five minutes. Neither provides infrastructure for extending the lease on a locked message.

There are other message-oriented quotas to consider as well. While Azure Queues lets you store an unlimited number of messages per queue, with no practical per-queue size limit, any single message has a maximum Time to Live (TTL) of seven days. (All storage for a single account is limited to a 100TB, not likely a problem for queues.) After seven days, the message is automatically deleted from the queue. By contrast, with Durable Message Buffers, you are limited to 10 buffers per account, with each buffer limited to 100MB in size. There is no limit on the number of messages in that buffer as long as they fit within the 100MB limit. In addition, messages in the buffer have an unlimited TTL.

Queue Reliability and Performance
When considering cloud-hosted queues, you want to examine the host nodes, replication, and usage of a persistent storage. Both Azure Queues and Durable Message Buffers are designed to reliably host queue messages, even in the face of a single node failure. The queues in Azure Queues are built on top of Azure Tables and are inherently triple-replicated and maintained in durable storage. Durable Message Buffers are not stored in memory either; instead, they are built on top of SQL Azure.

In addition to reliability, you'll probably also want to consider performance. Because message buffers are still in Community Technology Preview (CTP), there's not much documentation to consult here. But Azure Queues does have quite a bit of documentation, as you can see in the Additional Resources section at the end of this article. In Azure Queues, a single queue has a scalability target of 500 messages per second, with all queues in a storage account scaling collectively to handle 5,000 messages per second. That said, your mileage may vary, depending on whether your Azure resources are throttled. Because the queue name is used to partition load, all messages to a given queue are processed by a single server, with different queues potentially handled by different servers.

Security
It's also important to consider how producers and consumers are authenticated and authorized to use queues. Azure Queues simply uses a shared key approach for access. This key, which appears in the Authorization header of requests, includes the storage account name and a Hash Message Authentication Code (HMAC) signature. All requests to a queue in Azure Queues must include this Authorization header; there is no support for public requests.

Durable Message Buffers does not use storage credentials. Instead, the user presents the Access Control Service (ACS) with a username, password, and queue URI in order to acquire a Simple Web Token (SWT) from ACS along with claims indicating authorization for Send, Listen, or Manage. The Buffer Management API always requires authorization. Runtime operations, however, support anonymous access that is configurable at queue creation time by setting the Authorization Policy to Not Required. The Required, RequiredToReceive, and RequiredToSend values can be used to restrict to authorized access for both queuing and dequeuing operations, dequeuing operations only, or queuing operations only respectively.

In regard to transport security, Azure Queues provides support for access over SSL. However, only message buffers can be configured to require SSL for all runtime operations or, optionally, to allow the client to use SSL for send or receive operations.
 
An Exciting Choice
Outside the aforementioned features, each technology has certain unique capabilities. Azure Queues, for example, supports retrieving or peeking up to 32 messages in a single request. Additionally, only Azure Queues offers an explicit operation to clear all messages from a queue.

Durable Message Buffers proffers the unique capability to support long HTTP polling, whereby control of the request timeout can be made on each receive request. With this, an application can return immediately if no messages are available, or wait up to two minutes for an incoming message before returning a timeout error.

Each of these queuing technologies has its strengths, and it will be interesting to see what other features become available for each in the future. In the meantime, I hope I've helped you get excited about positioning one of these solutions for use in your upcoming projects.