Recovering from Active Directory Disasters

Active Directory (AD) is typically one of the key network services in an organization. Without it, everything comes to a grinding halt. With this in mind, it’s important to be prepared for the various disasters that might strike a forest.

When it relates to AD, the scope of the disaster can vary quite a bit. It can be as simple as the failure of single domain controller (DC) or the accidental deletion of a single object. An even worse situation is when an entire organizational unit (OU) hierarchy is accidentally deleted. In the worst case scenario, an entire domain or forest might need to be restored.

The good news is that many of the techniques that apply to recovering from simple disasters also apply to recovering from catastrophic disasters. I’ll discuss how to recover from the two most common calamities: a failed DC and accidentally deleted objects.

Backup Strategy

You first need to make sure that you have something to use for a recovery. At a minimum, you should have valid system state backups of at least two DCs in each domain in your AD forest. Windows Server Backup (Windows Server 2008 and later), NTBackup (Windows Server 2003 and Windows 2000 Server), and most commercially available backup tools can perform valid system state backups. However, it’s always worth testing the backups to make sure everything is in order. One important point regarding backup tools is that you should use a Volume Shadow Copy Service (VSS)–aware backup tool. Backup tools that rely on disk imaging or virtual machine (VM) snapshot technologies are generally incompatible with AD. Restoring a backup made by one of these tools can cause serious replication failures known as update sequence number (USN) rollback.

In many organizations, the responsibility for server backups and restores falls to a different team than the team that runs AD. This leads to a couple of problems. First, you have no direct control over the backup process, which makes validating backups difficult. Second, many backup tools require an agent on each DC being backed up, which indirectly provides elevated access to the DC.

To mitigate these problems, I frequently employ a two-tiered approach to DC backups. I use a script to run Windows Server Backup each night on the DC and keep a week or two of backups locally on the DC. The folder containing the backups is then shared, with access restricted to the backup tool, as many backup tools can back up a file share without an agent. I also sometimes store the backup files on neighboring DCs within a site. So, for example, if you have DC1 and DC2 in a site, the backups of DC1 are stored on a file share on DC2 and vice versa.

The benefits of this two-tiered approach include:

I posted the script I use to run Windows Server Backup as well as directions for setting it up in my article, "Managing Local Backups with Windows Server Backup".

DC Recovery

One of the great things about AD is the mostly stateless nature of the DC. Aside from potentially holding one or more Flexible Single-Master Operation (FSMO) roles, a DC should generally be a matching replica of other DCs in the domain, except for some potential delay in replication depending on your topology. If a failure renders a DC inoperable, this stateless nature is fantastic because it will often remove the need to go through a complicated restore from a backup. Instead, you can simply reinstall Windows and use Dcpromo to promote the server to a DC and replicate all of the data back in—assuming your domain has more than one DC. If you only have one DC in your domain, you can greatly reduce your exposure to failure by deploying a second one.

Before you reinstall and repromote a DC, though, you need to clean up AD, which is a two-step process. The first step is to seize any FSMO roles that the DC might hold for another DC in the domain. If you’re not sure which DCs are hosting FSMO roles in the domain, run

netdom query fsmo

in a command prompt window to find out. You can then seize the FSMO roles using the Ntdsutil utility. Follow the instructions under the “Seize FSMO roles” section in the Microsoft article “Using Ntdsutil.exe to Transfer or Seize FSMO Roles to a Domain Controller”. It’s very important to note that when you seize a FSMO role, best practice dictates that you should never bring the original role-holder back online.

Because it isn’t possible to put the original FSMO role-holder back in service, the second step is performing a metadata cleanup of the failed DC’s configuration in AD. You can use Ntdsutil for this step as well. Follow the instructions in the Microsoft article “How to Remove Data in Active Directory After an Unsuccessful Domain Controller Demotion”. Alternatively, if you’re using the Server 2008 (or later) version of the Active Directory Users and Computers snap-in, you can complete this step by deleting the DC’s computer account in the Domain Controllers OU.

Repromoting a DC over the network might not be feasible when the amount of data to replicate would place an undue amount of strain on the network. In this case, there are a couple of other options. The first option is to restore the DC’s system state from a backup and continue on. The second option is to use the Install from Media (IFM) functionality, which was added in the Windows 2003 release. IFM lets you take a system state backup (created with NTBackup in Windows 2003) or IFM media (created with Ntdsutil in Server 2008 or later) and point Dcpromo to the AD database in the IFM media. IFM media created by Windows 2003 must first be restored to an alternate location on the file system so that Dcpromo can consume it. The DC will make the necessary changes to the database in the media and replicate only the changes since the media was created over the network.

AD Object Life Cycle

When you delete an AD object, a number of things happen behind the scenes. Most important, deleting an object doesn’t directly correlate to a record being removed from the AD database. To maintain consistency in AD’s replication model, objects first transition through a state known as being tombstoned, as Figure 1 shows. Rather than implementing a distributed mechanism to replicate physical deletions from the database, AD replicates a change to an attribute that indicates the object has been deleted.

Figure 1: Default life cycle of an AD object

When you delete an object from AD, the isDeleted attribute is set to True, which means nearly all the object’s attributes are removed. The object is moved to the Deleted Objects container, and its lastKnownParent attribute is stamped with the distinguished name (DN) of the parent object before the object is deleted. After an object has been marked as deleted, it won’t be visible to any tools that query AD, unless you add a special LDAP control to indicate that you want AD to return deleted objects in the search results. Various free LDAP query tools (such as AdFind) will include this LDAP control for you and allow you to easily search for deleted objects.

At this point, the object will remain as a tombstone for a period of time. The default tombstone lifetime for forests is based on the OS of the first DC in the forest. Table 1 shows the default tombstone lifetimes. Upgrading AD doesn’t change the tombstone lifetime for the forest.

Table 1: Default Tombstone Lifetime for New Forests

Periodically, a background process called garbage collection runs on each DC. The garbage collection process (aka garbage collector) scans the database for tombstones that are older than the forest’s tombstone lifetime and purges them from the AD database.

Up until the point when a tombstone is purged by the garbage collector, you can recover the object using a process known as tombstone reanimation. When you reanimate a tombstone, you only get back a handful of attributes that are kept during the tombstoning process. For example, the attributes saved for a user object include the user’s SID, SID history, and username (sAMAccountName). Notice that this list doesn’t include attributes such as the user’s password, group membership, or demographic information (e.g., name, department). You can control the list of attributes that are preserved when an object is tombstoned by modifying the searchFlags attribute of an individual attribute’s definition in the schema. You can add as many attributes as you like. However, you can’t add linked attributes, such as group membership or the mailbox database containing a user’s mailbox. For information about how to modify the searchFlags attribute, see the MSDN web page “Search-Flags Attribute”.

In AD forests operating at the Server 2008 R2 forest functional level (FFL), you can enable a new feature known as the Active Directory Recycle Bin. As Figure 2 shows, the Active Directory Recycle Bin adds an intermediate state between when an object is deleted and when it is tombstoned. When an object is in this new deleted state, it’s hidden from search results but all its attributes (including linked attributes such as group membership) are preserved.

Figure 2: Life cycle of an AD object when the Active Directory Recycle Bin is enabled

An object in the deleted object phase can be recovered to the exact state it was in at the time of deletion using the same process that’s used to reanimate a tombstone. By default, an object stays in the deleted object phase for the same amount of time as the forest’s tombstone lifetime, as outlined in Table 1. You can change this time period by modifying the forest’s msDS-deletedObjectLifetime attribute.

After the deleted object lifetime expires, the garbage collector moves the object into the recycled object phase. A recycled object is the functional equivalent of a tombstone, with one important difference: You can’t reanimate a recycled object or restore it from a backup.

Object Recovery Mechanisms

As AD matured from release to release, the mechanisms to recover a deleted object have evolved significantly. In Windows 2000, the only way to get a deleted object back was to perform an authoritative restore from a backup. Windows 2003 introduced the concept of tombstone reanimation, which lets you get a partial copy of the deleted object back without restoring it from a backup. Server 2008 R2 added the Active Directory Recycle Bin, which allows the complete recovery of a deleted object without a restoration.

It’s important to note that the shelf lifetime of an AD backup (as well as IFM media) is the same as the tombstone lifetime. If you have the Active Directory Recycle Bin enabled, the shelf lifetime is the lesser of the deleted object lifetime or recycled object lifetime. For example, if the deleted object lifetime is 180 days and the recycled object lifetime is 60 days, then the shelf lifetime is 60 days. Thus, it isn’t possible to restore a deleted object from a backup that’s older than either of these values.

Authoritative Restore

When you need to get an object or series of objects back from a backup, the authoritative restore process is often the way to go. If you’ve ever wondered what the Directory Services Restore Mode (DSRM) option on a DC’s F8 boot menu is for, this is the option you choose to perform an authoritative restore. When you boot in DSRM mode, AD is never started and the database is offline. You can restore the AD database from a backup while booted into DSRM mode, then use Ntdsutil to select the objects that need to be restored. Note that it isn’t possible to perform a restore when the AD NTDS service is stopped on Server 2008 and later DCs.

When you perform an authoritative restore, AD increments the internal version number of the objects being restored. This ensures that when the DC is back online, those objects are replicated out into the rest of the domain and the restored version becomes globally effective.

Authoritative restores are often performed to recover OUs that contain a large number of objects (e.g., users, groups, computers, other OUs). Suppose that you accidentally deleted the Executives OU from the contoso.com domain. To get the OU and everything in it back, here are the steps you need to take:

1.     Boot into DSRM mode and log on with the DSRM password you set during Dcpromo.

2.     Restore a system state backup that was created before the accident. Don’t reboot. (This is a common mistake, especially when under pressure.)

3.     Launch a command prompt window and run Ntdsutil.

4.     Run the command

authoritative restore

5.     Run the command

restore subtreeOU=Executives,DC=contoso,DC=com

(Although this command wraps here, you'd enter it all on one line. The same holds true for the other commands that wrap.)

6.     Review and confirm the confirmation safety prompts. You should then receive a message like the one in Figure 3. Make note of the text and LDAP Data Interchange Format (LDIF) files that are generated.

Figure 3: Message noting a successful authoritative restore

7.     Reboot the DC into normal operating mode.

8.     Log on to the DC and open a command prompt window. Import the LDIF file exported during step 6 by running the command

ldifde -i -far_20110221-151131_links_contoso.com.ldf

This will import the linked attribute values (such as group membership) for the objects restored.

If you need to restore only a single object (e.g., a deleted computer object), you can use the restore object command instead of restore subtree command in step 5. If your forest contains multiple domains, you need to use the text file exported in step 6 to restore group membership for domain local groups in other domains.

Tombstone Reanimation

There are a number of tools that you can use to reanimate a tombstone, but they all ultimately perform the same steps. So, as an example, here are the steps you need to take to reanimate a deleted user named John Doe with the AdRestore utility:

1.     Open a command prompt window and search for the user with the command

adrestore Doe

AdRestore will search the deleted objects for anything matching *doe* and return output like that in Figure 4.

Figure 4: Sample output from the AdRestore utility

2.     Make sure the object you want to reanimate is present, then run AdRestore again with the -r switch:

adrestore -r Doe

3.     Confirm the prompt asking if you want reanimate the object. AdRestore will then reanimate the object to the location it was previously found.

As discussed earlier, tombstones lose most of their attributes when they’re deleted. So, you’ll have to repopulate many of the attributes to make the reanimated object useful again. (Generally speaking, if you use an automated identity management tool, the attributes will be automatically repopulated after the tombstone is reanimated.)

Active Directory Recycle Bin Undelete

The Active Directory Recycle Bin is undoubtedly the best recovery option because all attributes are restored, including linked attributes such as group membership. However, as mentioned previously, your forest needs to be operating at the Windows Server 2008 R2 FFL to take advantage of it.

You can use Windows PowerShell to enable the Active Directory Recycle Bin by running a command such as

Enable-ADOptionalFeature -Identity
  'CN=Recycle Bin Feature,
  CN=Optional Features,
  CN=Directory Service,
  CN=Windows NT,CN=Services,
  CN=Configuration,DC=contoso,DC=com'
  -Scope ForestOrConfigurationSet
  -Target 'contoso.com'

Note that enabling the Active Directory Recycle Bin is not a reversible step. In addition, objects that are already tombstoned when you enable the Active Directory Recycle Bin will no longer be recoverable through tombstone reanimation.

After you’ve enabled the Active Directory Recycle Bin, any objects that are subsequently deleted will be recoverable in their entirety for the duration of the forest’s deleted object lifetime. There are a number of ways to undelete objects, but the easiest is to use PowerShell’s Restore-ADObject cmdlet. For example, here are the steps to undelete a user named John Doe:

1.     Launch the Active Directory Module for Windows PowerShell from the Administrative Tools section of the Start menu.

2.     Search for the deleted user by running the command

Get-ADObject -SearchBase
  "CN=Deleted Objects,DC=contoso,DC=com"
  -ldapFilter:"(msDs-lastKnownRDN=John Doe)"
  -IncludeDeletedObjects
  -Properties lastKnownParent

Make sure that it’s the only object returned in the result set

3.     Restore that object with the command

Get-ADObject -SearchBase
  "CN=Deleted Objects,DC=contoso,DC=com"
  -ldapFilter:"(msDs-lastKnownRDN=John Doe)"
  -IncludeDeletedObjects
  -Properties lastKnownParent |
  Restore-ADObject

If you deleted an entire OU, you’ll need to recover objects in the correct order (i.e., such that an object is not recovered before its parent is recovered) so that they can be put back where they belong. Microsoft has posted a tree undelete PowerShell script that you can use to perform this task.

A Complex Task

Planning for an AD disaster is a complex task because of the multitude of things that can go wrong. However, if you know how to recover from a failed DC and the accidental deletion of an object or an entire tree of objects (such as an OU), you’re well on your way to being prepared for a disaster.