Working with NoSQL data in Azure and MongoDB

Why NoSQL

In today’s ever demanding world of change, inundation of data from diverse sources, applications face significant challenges in adjusting their backend data model to surface this data whether it is in a standard Web Application, Desktop Application or Mobile Application.  Not only that, but as organizations try to scale their servers that surface this data to devices, there is a limit to what they can scale vertically (i.e. bigger CPU, RAM, Disk Space) sooner than later they have to scale horizontally (i.e. adding more servers to the farm); this presents another challenge when working with databases, your standard joins between data object then becomes increasingly complex when trying to join related schemas across physical devices.  Enter NoSQL, this somewhat schema-less collection of documents provide that flexibility to adjust your data model on the fly and because all the collections (aka in the RDMS world as tables) are self-contained i.e. they maintain all the relationships (parent/child etc) within each document, it provides an indispensable approach not only for scaling out but for performance.  In this blog, we will look at a quick and easy go to market solution using the tools you are provided OOB from Azure that can you get you an extensible solution in no time.

Solution Goal

Beyond learning at a high level about NoSQL, why even bother reading this post you may ask yourself? Well, this is actually a true use case that I had the opportunity to use when I was brought in to solve a particular issue facing a department in one of the U.S. Federal Government agencies.  The challenges they faced was:

Initially the department wanted to create their own API, host and secure it in the cloud. They were already keen on using a NoSQL approach after researching and some due diligence but they thought that it would require significant investment both in time and a level of complexity that would be a limiting force moving forward. I however, knew of a way to use a few seemingly detached Azure workloads/services to architect a solution that not only met the stated solution goal, but exceeded it all out of box and in ½ the time.

As this blog will highlight, we will use the following Azure technologies in support of our goals

First we will create a FREE MongoLab DbaaS instance from the Azure MarketPlace (you can certainly get a bigger instance for a premium), then we will create a WAMS solution. Now you may be asking yourself “the solution never mentioned anything about mobile…” and you would be right, but.. but.. but.. all a WAMS solution is….. is a Web API backend with a Mobile front end, but the backend gives you sooooooo much for free OOB, it’s a shame not to use it and the added benefit, is that you can go mobile real easy as all the heavy lifting is already done.

Create your MongoLab DbaaS                                                            

First you will need an Azure subscription which you can get here. Then you will log into the portal and under “MarketPlace” spin yourself up a new MongoLab instance as you see below.

Setup is quite easy with the wizards they provide you, in fact in 3 easy steps you have a MongoDb instance that is accessible across the Internet. Initially you can be setup for a free sandbox account as you see below for $0.00 (yah, totally free) and I have yet to run out of the ½ Gig space they give me in storage. If that is not enough for you, then you can scale up to the other plans as you see below and it seems they even run specials with promocodes.

When you are done, you will have a MongoDb database and you will be automatically logged in with the account you used in the provisioning process.  You can and should provision additional users for access to the collections and not rely on the super user account for your instance. You may also create additional collections, indexes, etc all from the Web UI console.

While certainly this is one approach you can use and it is very effective and gets you rolling right away, you can also use standard tools to interact with your MongoDb database, or even go with using Terminal (mac or windows). I, on the other hand, prefer to use whats getting to be one of my favorite IDE besides Visual Studio… Webstorm. As you will see below, you can get plugins for MongoDB Shell right in the IDE.

In the above example you see me connecting to my FabianAzure MongoDB instance hosted in Azure by MongoLab and in the Mongo Shell in the bottom pane you see me making changes and doing searches to my collections. This for me is a better experience especially because I can save my work to GitHub Gists directly from the result window…. A nifty tool to have.

Create and Configure WAMS Solution

The next thing you have to do now is to create a Windows Azure Mobile Service (WAMS) solution that will use as its ‘Data Source’ the MongoDb database you created in MongoLab. Now, Ive covered WAMS in several of my own Blog Post and also for SharePoint Pro Mag, so I won’t belabor it here. The one difference that you will need to do ‘in the portal’ is create a Connection String pointing to the MongoDb Instance that is hosted in MongoLab. As it relates to the actual Web URI, you will also need to make changes to the Controller to use a NuGet Package for MongoDb and DocumentDb and also update your Web.Config with the same connection string detail. There is a good blog post here on how to do that.

Pause and Evaluate

Now is an excellent time to pause and evaluate what we have so far, because in essence, we now have an exposed URI via WAMS that has CRUD capabilities [if you wired up your controller that way] to a NoSQL MongoDB backend. Surely this is enough to provide the needed Solution?, well perhaps, but not with a degree of sophistication that we can get if we add one more element. Now there are other things that you get FREE and OOB from using WAMS, you get the ability to create Scheduled Jobs to monitor the WAMS solution and take actions on its behalf (think Web Jobs or Web Roles), you get Push Notifications, you get Identity (Authentication and Authorization) not only with Windows Azure but with Social IDPs such as Google, Facebook, Twitter to name a few, and you also get logs and ability to scale out. 

To really put some polish on this solution however, if we add Azure API management, we really get a professional, squared away, extensible, and top tier tools to push that experience both from a Developer, Consumer, Provider, Subscriber, and Administrator feels. Lets see how.

Create and Configure Azure API Management

API Management is yet another featured solution in Windows Azure and this is the money maker in this solution. It adds the necessary flare and polish that makes this something you can take to market. Here are a few highlights you get when you implement this… then I will get into details

Anywho…. To begin you go back to Azure as you normally do in the Portal and…

In the above screenshot, you can see that under API Management in the quick launch, you click Create

It does take a while for this to spin up, its actually doing a lot in the background, besides provisioning two sites (Admin Portal and Developer Portal) its creating all that you see here on the left hand side of the screen shot here above.  You get a “Echo API” for free as a starter with all the possible Methods  you can create CRUD-Q and it also shows you things like how to enable Caching, Policies, Products, Subscriptions etc. you should play around with it once you provision yours.

Probably one of the most important things you can do is set up your Operations (Methods) that you will be calling or having your subscribers consume. As you will see in the screenshot below we have a few GETS and a POST. One thing that I like most here is that you can build on your Web API that you have in WAMS without affecting change to  your WAMS API, that is, you can use OData Queries to create Operations on existing WAMS Methods. Take for instance the GetWhatsOnTap GET Method below, what I did there was simple put a filter query using OData syntax on a Boolean field on the base Get All Beer List Method. This means I can (and so can your Developer Subscribers extend what you give them OOB by using OData.

You can really get into some details in-terms of configurables here see below…

In the example here, we can really lock down our API and Operations by enacting Policies on them. You can even manipulate how the result set is formatted on the return trip. These policies can be configured by individual or all Products, API’s or Operations. Another indispensable one that you can do is enable Caching as you can see below

In the example above you can see that I am setting a 12 hour Cache on the Get All Beer List Method on the JailBreak API, the  reason… This list hardly changes and it isn’t going to change more than once per day (the brewery isn’t cranking out multiple beers per day) so I turn on caching on this one to speed up subsequent calls to this API method.

Extending the API Management to Developers/Subscribers

The next thing you can do here is make our API’s available to us to develop against but moreso to other Developers and Consumers at-large. Azure API management provides a Portal specifically for that where you may have API’s bundled as Products and set it up for

This way you can have API’s for people to use for free with Rate Limits on access you want based on the amount of transactions per day perhaps, you can associate cost by way of providing Subscriber Keys etc, its really a world of possibilities.

Below you can see how extensible and developer friendly this is

You can try out API’s and see what results will come back

In Summary

So, lets look at what we have achieved using all these technologies

As far as how I have used this in my world, I have frontloaded my WAMS Uri with this so that I can have an abstracted layer to make changes to my Methods in the underlying WAMS Web API and offer rate limits and caching that is not otherwise provided via WAMS. I use this in my Mobile Applications also because I can tie in multiple APIs under this one Exposed EndPoint and apply several policies all at once even though my API source may be disparate. This is surely not the only way to do this, but I can make a good bet that it will get you there quicker than most other options.  As you start to consider options for building out solutions that are data centric, I hope you will consider this architecture.