ESEUTIL - even more evil for DAGs

All of the comments I included in my post “ESEUTIL – the evil utility” apply to both standalone servers and those who are members of a Database Availability Group (DAG). I bet you wouldn’t be surprised if I told you that there’s even more to consider when you contemplate running ESEUTIL against a database that has copies within a DAG.

Consider what happens when a database has copies within a DAG. Everything is done to make sure that the copies are the same, right down to insisting that all the copies use the same paths for database and transaction logs. The Replication service ships transactions between servers so that the Information Store can keep the contents of all the copies as identical as possible and transaction logs are kept so that a newly activated copy can be brought completely up to date in as little time as possible.

Now consider what happens if you run ESEUTIL to defragment or rebuild a database. You create a new copy of the database that’s hopefully compacted to the nth degree (defragmentation) or bright and sparkly new with no nasty corrupted pages (rebuild). In either case, the important thing is that the result is a new database. More recently, Microsoft changed its support policy to force administrators to move all mailboxes from a repaired database because the fact is that some repairs cause more damage than they fix, which is yet another good reason for not using ESEUTIL.

But let’s assume that a notion comes upon you and you decide to defragment a database. Or that you read a wonderfully informative web article extolling the virtues of database fragmentation as an essential step on the road to world peace. So you locate ESEUTIL.EXE and read up on the command switches before you dismount the database.

Assuming that you are able to quell the protests of users whose electronic discourse has been interrupted by the database now being offline, you run ESEUTIL /D and watch with pleasure as ESEUTIL takes its own fine time to process the database. Page by page by page by page… Until it’s done and you are able to admire the 100 MB or so that you recovered by running ESEUTIL.

You can mount the newly defragmented database and bring it online. Clients will be able to connect and work as normal. Everything seems perfectly OK until you attempt to activate one of the other database copies (see screen shot of EAC error above) - or Active Manager attempts an automatic failover after a problem occurs. At this point we disappear into a world of hurt as Exchange logs copious events to protest the weird stuff you’re trying to make the Store do.

Depending on the day of the week and the direction of the wind, you might see errors like:

You get the idea. It’s not pretty and it’s all because the internal identity (the signature, or Rand) of the database that you have just mounted does not match the identity of the other copies. ESEUTIL creates a new database, one with a new identity, an identity that is created with no reference to the one that previously existed.

You can check the database signature by running ESEUTIL in MH (dump the header) mode. You’ll have to dismount the database because ESEUTIL requires exclusive access, but you’ll only have it offline for a moment or so.

ESEUTIL /MH outputs all of the header information for the database. Look for the line starting “DB Signature.” You’ll see the date and time that the database was created followed by a numeric Rand value (determined randomly to identify the database). Something like 1179680486 as shown in the screenshot. If you run ESEUTIL /D and then check the database signature in the header, you’ll find that the Rand value is changed. Here is the root of the problem.

When the new database is brought online, Active Manager attempts to check the status of the other copies. During this process the details of the database are validated and checked to ensure that everything is consistent. It is at this point that the house of cards collapses. You’ll have a database mounted, but none of its copies are now valid.

To restore equilibrium, you have to reseed the other database copies from the new database using the Update-MailboxDatabaseCopy cmdlet (use the DeleteExistingFiles parameter to remove the now useless MDB and transaction logs). This isn’t a big problem if the database is a couple of gigabytes; it can be a much bigger issue when databases are a couple of hundred gigabytes.  

The root cause is that ESEUTIL was written at a time when things were much simpler and DAGs didn’t exist. They do now and are a big part of on-premises Exchange 2010 and Exchange 2013 deployments and the high availability technology included in the DAG removes the need to go anywhere near ESEUTIL. Avoid using ESEUTIL at all costs or accept the consequences.

Follow Tony @12Knocksinna