MVC Routing, IIS, SEO, and Custom Errors – Oh My!

For all of its amazing strengths, it’s surprising how tedious it can be to get ASP.NET MVC applications to properly handle custom errors – especially if SEO is a concern. Part of this sadly stems from the fact that ASP.NET applications sometimes end up being subordinate to IIS in terms of error handling. But part of the problem also stems from the MVC routing problem itself. In this post I’ll outline some goals or expectations for error handling, cover some reasons why achieving these goals can sometimes be such a pain with MVC applications, provide some examples of how to get this working, and share a number of links to some great resources that shed additional light on the subject.

Caller Beware, Caller Confuse, Caller Inform

Handling HTTP errors should, ideally, conform to the age old development mantra that dictates that there are effectively three different ways to handle exceptions when they arise. Either you can do nothing and just let things ‘explode’ – potentially leaving things in an unstable state (i.e., Caller Beware), or, ideally, you’ll handle the error and inform the caller (be it an actual user sitting in front of a browser – or a bot) of the problem and some options to try and remedy the situation as per Caller Inform. Then, anything that falls between those two approaches amounts to Caller Confuse – and usually results from developers ‘bungling’ the error handling routine to some degree or another and making things possibly better or possibly worse than Caller Beware.

Out of the box, I’d argue that ASP.NET and IIS both default more or less to Caller Beware – at least when SEO is a consideration and where typical end users are concerned – as ASP.NET’s Yellow Screen of Death (YSOD) and IIS’s native error messages (especially for 403s, 500s, etc.) aren’t exactly what could be considered user friendly. Further, ASP.NET has sadly, had a long tradition of embracing Caller Confuse – by allowing developers to configure which HTTP status codes they’d like to ‘trap’, and then redirecting (HTTP 302) users to another page where a canned (or even detailed) error message is displayed. That might sound like a win, but if this takes what should be an HTTP 404 and redirects it via an HTTP 302 off to an ‘error page’ that subsequently throws an HTTP 200 (OK), then – while typical end-users will be happy – you’ve just created a perfect example of Caller Confuse where SEO and bots are concerned. (Happily, this Caller Confuse approach to redirecting is easy enough to override – as you’ll see below.)

So, with those baseline definitions out of the way, the goal or expectation I’m shooting for with this article is to be able to capture any and all (desired and therefore configured) HTTP ‘error’ codes and provide a Caller Inform solution to each of those. Meaning, specifically, that I want to provide a human-readable error page that describes the problem and gives the user some information about what they might do to correct the issue, and I want to make sure that I’m not decreasing SEO by adding in extraneous HTTP 302s or spitting out HTTP 200s when I should be throwing out 404s, 500s, and the likes. Further, I’d also like a way to ‘intercept’ 404s and match them against previous URL schemes so that any changes I’ve made recently with my app and its URLs doesn’t cause link-rot for anyone following previous links into my site. (This, though is a ‘bonus’ or added feature I’d like to add – and which I have added to my own sites – but which is outside the scope of this post.)

Setting the Stage

You’d be forgiven if you thought that achieving the goals listed above should be pretty easy – unless, of course, you’ve tried this before. Because, while it’s fairly easy and straight-forward to tackle the major aspects of Caller Inform error handling within an MVC application, there are some very tricky and subtle pitfalls – or pain points you need to watch for. Moreover, what might work with one version of ASP.NET MVC (say version 2 or 3) won’t necessarily work with later versions (like versions 4 and 5) – or against different versions of IIS.

What follows, then, is an overview of pain points associated with MVC 5 apps. To start, I’ve confured a sample web.config as follows – with the definitions listed below placed within the node:

You can obviously target more HTTP status codes as you wish, but I’m keeping things in this post simple – and simply re-routing the errors defined above into an ErrorController – with methods for each of the error codes I’ve defined. Note too that I’ve specified the redirectMode as ResponseRewrite – to overcome ASP.NET’s default behavior of using HTTP 302s for a redirect (which is non-ideal from an SEO standpoint). Further, here’s a quick look/overview of my ErrorController – just to put everything in perspective:

public class ErrorController : SiteController{    public ActionResult Index()    {        if (Response.StatusCode == 200)            Response.StatusCode = 500;         return View("Crash500");    }            public ActionResult Forbidden()    {        if(Response.StatusCode == 200)            Response.StatusCode = 403;        return View();    }    public ActionResult Crash()    {        if (Response.StatusCode == 200)            Response.StatusCode = 500;         return View("Crash500");    }    public ActionResult NotFound(string url)    {        return base.ExecuteNotFound(url);    }}

From here, things pretty much work as expected – with two major pitfalls or problems.

Pain Point – IIS

The first issue is that IIS (7 and above), bless its little heart, will override your configuration and simply hijack your custom error pages – meaning that your attempts at Caller Inform will be thrown in the toilet and users will be greeted with Caller Beware responses from IIS. What can be even more infuriating is that this problem likely won’t appear at all if you test against a local IISExpress instance instead of a ‘full blown’ IIS instance. Furthermore (if I’m remembering correctly), you can’t even depend upon IIS to hijack ALL of your customer error handlers – just some of them. As an example, in a snippet from my base Controller’s ExecuteNotFoundMethod, the following code wouldn’t work at all – until I added in the Response.TrySkipIisCustomErrors = true; directive:

string path = url;NotFoundDescriptor descriptor = this.NotFoundDescriptorManager.NotFoundDescriptorByPath(path);if (descriptor != null){    switch (descriptor.Type)    {        case NotFoundDescriptorType.Redirect:            return new RedirectResult(descriptor.RedirectedPath, false);        case NotFoundDescriptorType.RedirectPermanent:            return new RedirectResult(descriptor.RedirectedPath, true);        case NotFoundDescriptorType.Removed:            return View("../Error/Gone410");  // sets 410 in the view itself.         default:            throw new ArgumentOutOfRangeException();    }}// deal with idiotic issues from IIS: Response.TrySkipIisCustomErrors = true;return View("../Error/NotFound404"); // sets 404 in the view itself.

Personally, I’m not quite sure how IIS was ever allowed to hijack or upstage error messages (and statuses) set by an application – but I am glad that ASP.NET at least has the option to try and skip IIS Custom Errors (even if it still seems ridiculous to have to add this to your code).

Pain Point – MVC Routing

The other big issue you’ll have with MVC applications stems from routing. Assume for a second you’ve set up some non-default routes in your RouteConfig.cs file (or whatever you’re using to manage routing) and then that you’ve left the ‘default’ handler and added in a ‘catchall’ to capture 404s – as follows:

routes.MapRoute(    "Default",    "{controller}/{action}/{id}",    new { controller = "Home", action = "Index", id = UrlParameter.Optional },    new { controller = "Home|Services|Contact|Error" });routes.MapRoute(    "catchall",    "{*url}",    new { controller = "Error", action = "NotFound" });

In the example above, you can see that I’ve further constrained the default route so that it only allows controllers of a specific name. Likewise, the obvious intent of the catchall route is to grab anything that hasn’t matched so far, and send it off to the NotFound method of the Error Controller. All in all, pretty straight-forward and fairly logical. Only, there’s sadly a problem – which is that if someone were to hit the site above with a path of “/pigglywiggly” (which doesn’t exist), their request will ‘drop’ past other defined routes, past the default route, and into the catchall – at which point they’ll be sent to a 404 page that implements Caller Inform to let end-users know the content wasn’t found and to let bots know that the content is missing – i.e., HTTP 404.

However, if a path of “/services/pigglywiggly” was sent it, a nice, Caller Inform of 404 isn’t rendered. Instead, a Caller Beware page is thrown by IIS with an ugly 404 message that provides virtually no additional information. If you crack the covers or trap exceptions within your app, you’ll see that the root cause of the problem is that there’s no Method named “pigglywiggly” on the Services Controller. Or, in other words, Routing detected that the path matched a controller and then routed the request of the request into that controller to match up the remainder of the path with an action method.

To address this issue, you’ve got a couple of choices. One is to get rid of the default route – and then explicitly define a route for each and every controller + action method desired. If you’re going to go this route, then I think Routing Attributes would likely make more sense than a RouteConfig ‘table’. But, I’m also going to argue that one of the tenets of MVC is ‘convention over configuration’ – meaning that I like the idea of the default route as it is very MUCH a convention that helps make setting up routes tons easier. It just so happens, however, that it completely falls down with this particular problem. Or at least, it looks like it does until you realize that there’s a void method defined by the ‘base’ MVC Controller class called HandleUnknownAction() – which suits this situation just perfectly. Using this method is pretty straightforward, as you can see below:

protected override void HandleUnknownAction(string actionName){    string requestedUrl = HttpContext.Request.Path;    this.ExecuteNotFound(requestedUrl).ExecuteResult(this.ControllerContext);}

I’ve implemented the method above in my ‘SiteController’ – or the base controller that I derive Controllers from in my own app. The call to ExecuteNotFound() is simply a call to a method that returns an ActionResult. In my app I actually have that method check to see if the requested path matches older URL schemes or content that have either been moved (301) or removed (410) – so that I can update users and bots accordingly. But, all you technically need is an ActionResult – or other form of result – that you can execute. As such, something like the following could easily replace the last line in the method above – and would simply route users to a static 404 page (where you’d want to make sure you’ve set the status code to Http 404 up in the view’s ‘header’):

this.View("../Error/NotFound").ExecuteResult(this.ControllerContext);

Conclusion – and Additional Resources

Once you have a decent handle on the different hurdles you’ll need to jump through to get custom error pages working correctly, getting everything to play nicely in an MVC app isn’t that terrible. On the other hand, it can be a bit of a pain to beat your way through some of these problems on your own though. As such – and if you’re on a different version of MVC or running into additional problems, this StackOverflow question and its answers (i.e., don’t just check the marked answer) can be a valuable resource. Likewise, I found Ben Foster’s blog post on custom error pages in ASP.NET MVC apps to be a great resource and sanity check for setting up core details around error handling. Finally, do be aware too that there are some potential security concerns you might want to watch out for around using the HandleUnknownAction – as David Hayden calls out in his blog.