Manage Staging Deployments in Microsoft Windows Azure

If you are using Microsoft Windows Azure to host your applications and services in the cloud, it's easy to figure out how to publish to the cloud for production. You publish to the production instance with the configuration settings pointing to production, and voila, you're done. In this article, I will discuss how to handle publishing to staging. One issue is the "ever-changing URL" problem, when you publish to a staging instance of an Azure service, and another is the issue of handling different configuration settings for production, staging, and development. Let's start by talking about those pesky staging URLs.

Pesky Staging URLs: The Problem

For production instances, Microsoft sets the URL for your service using the service name you selected when you first set it up. For example, if you have a service called a-contoso, the URL for accessing that service is http://a-contoso.cloudapp.net, as shown in Figure 1.

Most people set up a DNS entry to point to their service rather than using and displaying the cloudapp.net address. For example, the Contoso company might set up www.contoso.com to point to contoso.cloudapp.net. This is what people expect. When you go to microsoft.com, you expect to see microsoft.com, not something like microsoftwebsite.cloudapp.net. (I totally made that up, by the way, so don't bother trying it in Internet Explorer.) (OK, I saw you try that. Told you so.)

Another advantage of using DNS entries is flexibility. If Microsoft wants to move the application to another region or host it on Amazon's cloud service (ha ha!), they can just change the DNS entry to point to the new site, and it has no impact on their customers.

This is all fine and dandy for production, because the URL that is assigned to the production instance of a service doesn't change. But what happens if I publish to the staging instance? I have set up three services for Contoso in my Azure portal, as displayed in Figure 2. Let's publish my application to the staging instance of a-Contoso, shown in Figure 3.

Figure 3: Publish to staging instance of a-Contoso

After it finishes, you'll see the service address displayed in the Azure portal, shown in Figure 4. If I publish it to staging again, here's the service address displayed in the Azure portal, shown in Figure 5.

The URL changes every time you publish the application to the staging instance. What if this is a service used by a client application? Or a website accessed by another website? You would need to change those entry points to use the new URL each time. Or if you have a DNS entry for the staging instance, you would have to change it every time you publish the application to the staging instance, then wait for it to propagate before you can try out your new version.

Unfortunately, you can't always schedule this to coincide with lunchtime. If you're like me, this will propagate quickly for everyone except the QA director who is waiting for the next version, and they will be typing "doesn't work for me yet" repeatedly in their IM window because their connection to the network doesn't pick up DNS changes for a long time, for some mysterious reason. This impacts productivity and tends to frustrate and irritate the QA director, and trust me, if you're the head of engineering, you do not want to irritate the head of QA!

This might not seem like a big deal if you only have one service. But what if you have 12? And what if you have interdependencies, like service A calls service B which calls service C, so if you want to test A, you also have to publish B and C. This means you have to publish C, get the URL, and put it in B's configuration, then publish B, get B's URL to put in A's configuration, and then publish A. Are you tired yet?

Pesky Staging URLs: The Solution

There's an easy solution to this problem. Let's say Contoso has three production instances. For each production service, I set up a corresponding "production service" (production to Azure) with the abbreviation "st" in the name to indicate that these are actually the staging version of the services. So now I have my list of services displayed in Figure 6.

I set up DNS entries to point to each of these, and they will never need to be changed. For example, I set up www.contoso-staging.com to point to a-st-contoso.cloudapp.net.

You can decide for yourself what naming convention to use, and whether you want the string that indicates it is staging to appear before or after the service name. You have to think about the sort order. You can basically sort them by service:

a-Contoso
a-Contoso-staging
a-Contoso-WcfService
a-Contoso-WcfService-staging
a-Contoso-Website
a-Contoso-Website-staging

or you can sort them by deployment type:

a-Contoso
a-Contoso-WcfService
a-Contoso-Website
a-st-Contoso
a-st-Contoso-WcfService
a-st-Contoso-Website

I tend to be paranoid about accidentally publishing to production, I don't have services published to staging all the time, and I don't want the staging services mixed in with my production services because it's more clutter, so I sort them by deployment type.

Just because Microsoft calls it "staging" doesn't mean you have to use it for that. And actually, I still use the staging instances of each Azure service when preparing to push a release into production. It currently takes 10 to 30 minutes to publish an Azure instance. 10 minutes is more likely in the middle of the night when I do my testing; 20 to 30 minutes is more likely in the middle of the day when everybody else is trying to publish their services as well.

If I have a release that is going to require publishing several Azure services, I don't want to wait until the entire release team is in the Go Live meeting to do that, because they would die of boredom. So I publish the instances ahead of time (with production configuration settings) to the staging instances of the production services, and during the Go Live meeting, I do a virtual IP (VIP) swap. This is fast, and I can also leave the old instances up for a couple of hours (or days) in case I have a problem, or if I'm particularly paranoid about the release.

One thing to note, though—if you have any worker roles pulling entries off of a queue, you should stop the old services after doing the VIP swap, or both the staging and production instances will process those entries. If you have had breaking changes and the old production instance (now running in the staging instance) picks entries off the queue, it may crash trying to process them. (Trust me, in the middle of a huge deployment with eight or 10 services involved, it's easy to overlook this. It can take longer than you think to figure it out. Don't ask me how I know this.)

You do still pay for services that are stopped or suspended, so don't forget to go back and delete them after you've verified that production is working fine so your bill isn't twice as much as expected at the end of the month.

That takes care of the pesky changing-staging-URLs problem.

What About the Configuration Settings?

The other issue that I came across was how to handle the different configuration settings for production, staging, and development. At first, I tried having multiple sets of configuration values in the XML for the Service Configuration (.cscfg file). I just commented out the section I wasn't using and uncommented the section I was using. This became pretty onerous, and manual changes are always dangerous. Also, it made the build engineer reliant on the programmers setting the configuration values correctly for the last build for production. I don't think I need to tell you how well that worked. Let's just say not all programmers are anal-retentive enough.

Before I relate the solution I came up with, let me say that the Windows Azure team released an August 2011 update to the Windows Azure Tools for Microsoft Visual Studio that provides a different solution. The screenshots used in my solution are for a project created with only Azure Tools 1.3 installed. You can also use this approach when you've created your project and you're using Azure Tools 1.4, but your default configuration will be called ServiceConfiguration.cloud.cscfg instead of ServiceConfiguration.cscfg.

Cloud projects have a reference to one or more web and/or worker roles, and the service configuration belongs to the cloud project. When you publish, you publish a specific cloud project. There is nothing keeping you from having multiple cloud projects referencing the same roles. This means you can have multiple cloud projects that point to the same roles, but use different configuration settings for each cloud project. So you can have staging, production, and development cloud projects that point to the same roles, but have their respective configuration settings.

I've created a solution with just a basic website in it to illustrate how to do this. This has only one cloud project, and it is called Basic Website. My solution is displayed in Figure 7, and my production configuration settings are displayed in Figure 8.

Figure 7: Basic website with one cloud project

Let's add a new Cloud project to the solution in Visual Studio. Right-click on the solution and select Add, then select New Project. On the dialog that comes up, select the cloud project template, and name it the same as production, but append "Staging" on the end, as shown in Figure 9.



Figure 9: New cloud project for staging deployments


Click OK, and the New Windows Azure Project dialog is displayed, shown in Figure 10.



Figure 10: New Windows Azure Project dialog


Do not add any roles. Just click OK. You should now see both projects in Solution Explorer, and BasicWebsiteStaging has no roles defined, as illustrated in Figure 11.



Figure 11: Solution with both cloud projects


Now add a role to the new project, but rather than creating a new role, let's just attach the web role we already have. To do this, right-click on the Roles folder in the BasicWebsiteStaging project, and select Add, then select Web Role Project in solution, as shown in Figure 12.



Figure 12: Add existing web role project


This brings up the Associate with Role Project dialog showing all the available roles in Figure 13.

Select our single web role and click OK. This will add the role to the service configuration file and the service definition file. Now we can copy the configuration settings from the production role and paste them into the staging role and edit them so they apply to staging. Mine is displayed in Figure 14.



You can add another project for development so the developers can set the configuration to whatever values they need without impacting either the staging or the production builds. Plus, staging and production have separate configurations, and no manual changes are needed when publishing each type of deployment. When you want to publish the service to Azure, just right-click on the project you want to publish and select Publish.

What Did Microsoft Add to Tools 1.4?

With the Windows Azure Tools for Microsoft Visual Studio 2010 1.4 (August 2011) release, Microsoft added a feature to address multiple configuration files. Let's take a look at that. I have updated to Tools 1.4 and am now going to add a new cloud project to my solution and see what I get.

So right-click on our solution and add a new project. I'm going to name mine BasicWebsiteTools1.4. When prompted to add a new role, just click OK without adding one. Next, in your project, right-click on Roles and select Add WebRole Project in Solution, then select our WebsiteWebRole.  Now I see multiple configuration files, as displayed in Figure 15.

Figure 15: New cloud project with Azure Tools 1.4

You can see that now Microsoft adds two Service Configuration files, one for deploying to the cloud and the other for running in the local development fabric. Note that you only have one Service Definition file. This means that when you add configuration entries, you must add them to both Service Configuration files and the Service Definition file.

So how do we use this to our advantage? Let's look at how to manage our configurations so we have Staging, Production, and Local. Right-click on the BasicWebsiteTools1.4 project and select Manage Configurations. This brings up the Manage Service Configurations dialog box in Figure 16.



Figure 16: Manage Service Configurations


Select Cloud, then click on Rename to get the dialog box in Figure 17.



Figure 17: Rename Configuration


Change Cloud to Production and click OK. Then with Production selected, click on Create Copy, put in Staging for the name and click OK, then close the dialog. Now you should have three Service Configuration files: Local, Production, and Staging. At this point you can double-click on any of the files and put in your settings.

Another way to update the settings is through the UI, as illustrated in Figure 18. If you go to the properties for the Worker Role and click Settings, you can see there is a drop-down box for which configuration to use. When it says All Configurations, it will show all the settings. If one has the same value for all configurations, it will show the value. Otherwise it will say , and you have to use the drop-down to specify which configuration's values you want to see. You apply these settings when you actually publish your application to the cloud. If you right-click on our new cloud project, you will see the new Publish Windows Azure Application dialog with a drop-down for selecting which configuration to use, displayed in Figure 19. Using this new capability, you can set up as many configurations as you need and just select the one you want when you publish. You must have the same configuration entries for each type of deployment, which removes the possibility that you will add a setting to one configuration but not the others. Using the first method I explained works when you haven't upgraded to Tools 1.4, and it enables you to publish without having to think about which configuration to use—you can just right-click and publish. Also, if you need to have configuration settings specified for staging but not production (or vice versa), you can do that. If you just want to be able to specify different values for your settings, and you don't mind selecting the service configuration at publish time, and you've updated to Azure Tools 1.4, you can use Microsoft's feature. Just use whichever works best for you.What About Web.config Files? What if you can't move all your configuration settings from the web.config to the Service Configuration files? What if your application has different web.config files depending on whether you are publishing to staging or production? Rather than manually copying and pasting the contents of the web.config files, you can set up a build configuration and web.config file for each deployment type, then use a pre-build command to copy the appropriate web.config file to the output directory. Suppose the web.config file in our basic website needs to be different depending on whether it's deploying to staging or production. First, create the web.config files. We'll need web.config.production and web.config.staging. Be sure to put a comment in each one so you can tell them apart. You still need a web.config file to be part of the project and have its Build Action set to Copy so it will be included in the build; the pre-build command will replace it with the right version before the application is built. If you are using source control, be sure to check out the web.config file before doing the build; you must have write access to it. Add the two new web.config files to your web role project (Add/Existing). On each file, set the Build Action to None, and set Copy to Output Directory to Do not copy, as shown in Figure 20.Figure 20: Web.config.production propertiesYou don't want these files to be accidentally included when you build your project. Now let's set up the build configuration for production. Click Build and then ConfigurationManager. You will see the default for the build configuration currently selected. Click the Active Solution Configuration drop-down and select New. Name the new configuration production. This must be the same as the file extension on the web.config file you want to include in order for the pre-build command to work. Set Copy settings from to Release and check the box for Create new project configurations, as shown in Figure 21. Click OK. Now you will see your new build configuration for all the projects, as shown in Figure 22. Click OK to save these settings, then do the same again and add a build configuration for staging. Next, bring up the project properties (still in your web role project) and go to the Build Events tab. In the pre-build event, put the following statement:COPY/Y "$(ProjectDir)web.config.$(ConfigurationName)" "$(ProjectDir)web.config" Let's look at the pre-build command. $(ProjectDir) is the directory your project is in, and where the web.config files reside. $(ConfigurationName) is the build configuration, such as production. When translated, if the build configuration selected is production, the pre-build command will look something like this:COPY/Y E:MySourceMyProjectweb.config.production E:MySourceMyProjectweb.config Before the project is built, it will copy the production web.config to the project folder, replacing the default one, and that web.config file will be included in your deployment as "web.config". If you set the build configuration to staging, it will copy the staging version. You can put this in, select a build configuration, build the project, then go check the project folder to see which one is there by checking the comment you included. This is a pretty simple way to automate the selection of the web.config file included in your deployment. You just have to make sure the file extension on the web.config files matches the build configuration it goes with. I have an application that has different web.config files for staging/production and http/https. So I have four cloud projects, four web.config files, and four build configurations, each with something like "production-http" in the name. I just set the build configuration and publish the corresponding project, and it works like a charm.One Last Recommendation for Staging Deployments If you use Azure Diagnostics, I recommend that you set up a separate storage account for your staging instances, and set the connection strings in the service configurations for the staging cloud projects to point to the staging storage account. If you do this, you won't have staging diagnostics combined with your production diagnostics, and you can delete all the staging diagnostics after you're finished testing.Robin Shahan is director of engineering at Goldmail and has over 20 years of experience in software engineering. She is a Microsoft MVP in Client App Dev, moderates Microsoft's MSDN ClickOnce and Setup & Deployment forum, and frequently speaks at Northern California Microsoft .NET User Group meetings. Contact Robin via robindotnet.wordpress.com.