Patrick Hynds on Disaster Recovery for Developers

Editor's note: Welcome to .NETRocks Conversations, excerpts of conversations from the .NET Rocks! weekly Internet audio talk show. Hosts Richard Campbell and Carl Franklin chat with a wide variety of .NET developer experts. This month's excerpt is from show 658, with .NET security expert Patrick Hynds, co-host with Michele Leroux Bustamante of the LockDown weekly Internet talk show, who dispenses disaster recovery wisdom for developers and other tech professionals.

Carl Franklin: What do we do, not just how do we recover, but how do we prevent data loss in the first place when that catastrophic physical event happens?

Patrick Hynds: So this is disaster recovery pure and simple. It's the... what do you do when the place is flooded, it burns down, the server fails, and there's escalating levels of mayhem that you have to plan for. And really, all you have to do is plan for it, know what the threats are and what you would do with it, and you're pretty much covered. The problem is, most people put this off until after it's too late.

CF: The thing is, nobody really wants to think that such a tragedy could happen to them. Part of it is actually accepting the fact that these things happen.

Richard Campbell: I think the big thing here is the planning part of this. I used to have a book I called "In Case Of," and the big thing was writing it out in a way that I didn't need to be there to execute on it.

PH: Yeah, that's key.

RC: Sometimes the disaster is you. They lose you, or they lose other key people. Like you've got to have documentation that allows people to know where things are... bank account numbers and who owns our telephone service. That kind of information goes a long way toward just staying functional after a disaster.

PH: So, specifically what we wanted to talk about today is, where do smaller companies fit into this? What if I'm a 50-person company. What should I be worried about?

So let's start with the first level of advice. Figure out your threats, think about the levels of threats. Is it that a server fails and doesn't come back up, but the hard drive's still good? OK, what are you going to do in that scenario? What if the server fails, and the hard drive is non-recoverable? OK, what if the server room burns or floods? What if the building is destroyed? What if the building [is destroyed] and all your employees are killed? And that's the question that usually shakes the CEO and the board of directors to their core, because a lot of people have never even thought about that.

Most of the time, the answer is, we're done. Especially with smaller companies, where the company is four other people, so if we lost 80 percent of the employees then we wouldn't try to continue. Though that means that the ones who survive are out of work.

RC: But as long as that's an accepted plan, right... there's sort of a recognition point, there's a point where business can't continue or the data's lost, and that's ok. I think that's setting reasonable guidelines. I think we get rather irrational about this... that we can never lose data, we can never be down. It's not reasonable.

PH: All data's not equal. So home directories are great, but my ISO library is infinitely replaceable. My email server, my SharePoint data, my financial system... that's probably [not]. So when I look at a small company and they talk to me about DR, and want me to talk to them about DR, I bring up the systems that are most likely to be the most important... the top tier, must-have, you-better-back-them-up-daily items.

So there's email. Financial systems. Usually it's the invoicing system or the ERP system or whatever holds the financial data, it's another—we need a daily backup. I would also say that your CRM system, the way you talk to your clients, client lists, or source code if you're a software or consulting company... those are the ones that I find universally are... if we only had most of those back, we could keep going.

CF: People don't usually talk about a hacker attempt or an infiltration as a disaster. But it certainly could lead to a disaster.

PH: If they delete all your data, it's going to test your disaster recovery plan because it's all about restores. It's all about getting back running.

So the cheapest way to get out of this is to identify the must-haves, the things you can't live without. Back them up based on how much data you can lose. I used to get asked by students when they were programming, well, how often should I save? I would say, every three seconds ask yourself, would I be mad if the computer shut off right now? And if the answer is no, don't bother saving. If the answer is yes, then save. And so, it's the same here.

But it's all about a plan, getting that plan. Then you've got to get it offsite, if you care about a real disaster. Like what happened at Christchurch. Having the stuff in a drawer or on a backup server in the same room doesn't help you. Richard, what do you think of the offsite backup? Where do you put the offsite backup?

RC: Well, we used to send tapes offsite. We had a courier that picked them up every day, there was a service for that, stored them in a very safe location. And then we had a server room fire. And we could not buy the tape drive!

CF: So it's not just the tapes. You've got to have a duplicate of the hardware itself...

RC:... and they wear out. The models go out of business, stuff goes away. We had the tape and couldn't read it. These days I think that whole backup mechanism is obsolete. You talk about a cheap, easy, low-friction backup solution, Dropbox. I'm running Dropbox on a server, it's got a directory—that's part of the Dropbox, and my various other apps that have stuff to back up... drop it there, it syncs onto the cloud. But you still need detailed instructions on what supplied that data and how to use it. If I drop a set of DNS records onto that Dropbox, if I know DNS backwards and forwards and I'm the guy who dropped it there, I can use it. But if I'm out of the picture, nobody even knows what that is. It's got to be documented: This is the DNS stuff, and here's how you load it back up into a new DNS server.

CF: Patrick, do you find that companies with a 50-person range are reluctant to spend this kind of cycles, both in thinking about it and getting somebody to do the documentation, to do the plan?

PH: And also, it brings no value. It's a tax, it's a tax on their business because it brings no value unless the world falls apart. And they're hoping the world doesn't fall apart.

RC: I mean, the point is, it is a reasonable hope. Most of the time the world does not fall apart.

PH: But then, you have insurance on your house, I assume, as well?

RC: Yep.

PH: Because just, you never know. So if you get a company that's small, two guys in a garage, I really recommend that they start by doing the simple, you know, 2TB USB drive, we each take a copy of everything that matters, and we go home... once a week. So that you start from the beginning with this disaster recovery. Because eventually you'll say, you know what, I don't need the ISO images, or I don't need the VM machines. All I need is the data. And you start getting an idea of what's important and what's not.

The problem is, most people if they do this, they do it in a rush. And it's the kind of thing you need to think about on the drive, in the shower, when you're just about to drift off to sleep. It's a deep-thought problem, as opposed to, I'm going to spend 15 minutes and come up with our disaster recovery plan. I really agree with Richard, the documentation side of things. You know, what do I do if—the "what if" book. The plan in case this happens, the plan in case that happens. If there's pages and chapters missing, so be it. But at least start filling it in so somebody can do something about it. And the other thing is, don't store that book on top of the servers!

There's much more! Listen to the full interview at dotnetrocks.com/default.aspx?showNum=658.


Richard Campbell and Carl Franklin (both Microsoft Regional Directors) are the voices and brains behind .NET Rocks! (www.dotnetrocks.com). They interview experts to bring you insights into .NET technology and the state of software development. They’re more than dry technical interviewers—they have fun!