Exchange 2007 Lessons from Microsoft IT

Riding the bleeding edge of technology is a typical part of a Microsoft IT administrator's job. Although many Exchange admins will likely migrate gradually to Exchange Server 2007 as early adopters work out the product's kinks, at Microsoft the Exchange 2007 production rollout to 70,000 users is in full swing. The aggressive rollout is backed by more than 18 months of testing by both the Exchange product team and IT, in keeping with Microsoft's "dogfooding" philosophy of deploying its own products in house before releasing them to the public (for more information, see the Web-exclusive article "Eating Its Own Dog Food," March 2005, InstantDoc ID 45597). Derek Ingalls, general manager, and other members of the Exchange IT staff spent all of 2006 and much of 2005 intensively engaged in testing Exchange 2007 on a 5,500-mailbox Exchange server dedicated to dogfooding. I spoke with Derek about how his team made the shift from Exchange Server 2003 to Exchange 2007 and how their dogfooding experiences can guide other IT pros on the path to Exchange 2007.

What overall process did you follow in dogfooding exchange 2007?
This upgrade was much different for us than previous Exchange upgrades. For us, Exchange 2003 was a lot like a service pack upgrade. We upgraded the entire environment in a weekend. But the Exchange 2007 upgrade process was quite a cycle. From the time we built the first server and put the first production mailboxes on it, we had milestones all along the way. As you know, Exchange 2007 consists of server roles, and not all the roles were done initially. Our first milestone was having the first Client Access server, then production mailboxes on a Mailbox server, then the first Edge server, and so on.

What was the most difficult part of the transition to exchange 2007?
We had some significant battles about storage. Because Exchange 2007 isn't as disk-I/O intensive as previous versions, you can use larger, slower, and less-costly disks for storage. And because high-availability features in Exchange 2007, such as Cluster Continuous Replication (CCR), mean you aren't so dependent on a single piece of storage, you can consider using less-reliable storage. When we were on Exchange 2003, we used a clustered SAN and had four nines of absolute availability. The notable exceptions to this availability were always related to storage outages.

The Exchange team took that problem to heart and wanted to reduce [Exchange users']dependency on SANs specifically. So understanding that customers will want to deploy a number of different storage scenarios in Exchange 2007, they needed us to validate the "crazier"scenarios—such as using DAS or Serial Attached SCSI (SAS).

But microsoft isn't saying that Das is the preferred storage medium for exchange 2007—just that it's an option, right?
Right. There was some controversy in IT about switching from a SAN to the cheaper disks because of our users' expectations of high availability. But because we decided to increase mailbox size from 200MB to 2GB, for compliance as well as dogfooding reasons, we needed to back up 10 times the amount of data that we were backing up in Exchange 2003. Ironically, in the middle of this controversy, we lost a SAN array with 8,000 users on it. It completely died, and we had to reinstall Exchange and recover 800 200MB mailboxes. That was an exceptionally painful exercise. And the Exchange development team guys told us, "If you'd been using CCR, it would have been a two-minute outage, not a two-day outage." They were right. That's really what helped us turn the corner and embrace this scenario. It was an unfortunate event but exactly what we needed.

Another challenge we had was that when we switched to 2GB mailboxes, we also told users, "no more PSTs." That decision caused probably the biggest backlash that we've ever seen. But when we offered to let the users who wanted PSTs out of the pilot program, they said, "no way—I love my 2GB mailbox!" We use the new records management features to set limits and actions to take on different folders, depending on the classification of the mail—for example, whether it needs to be archived or deleted after a certain period of time. With 2GB, users have essentially the experience of a bottomless mailbox.

During the course of the dogfooding, what exchange 2007 features did iT staff most appreciate?
The advances with Windows PowerShell (aka Exchange Management Shell) and the ability to do so much via the command line were a huge win for us from an administrative perspective. When we heard about Monad (the prerelease version of PowerShell) initially, everybody in IT thought that this new way of doing administration would be a headache. But after we started using PowerShell, we absolutely loved it.

I think PowerShell represents tremendous potential for the user community because it provides a consistent way of doing things. I think there will be a community of sharing scripts, cmdlets, and knowledge, so that people won't have to reinvent processes for automated ways of moving mailboxes or certain sorts of tasks.

What's an example of a problem that would be easier for you to identify using Powershell?
The state of services and databases is one. Another example: One of the more problematic roles for us, because it was brand new, was the Edge server role. When troubleshooting issues with the Edge server, we found that our traditional methods—using Performance Monitor or Queue Viewer through Exchange System Manager (ESM)—were less accurate than we would have hoped. But we were able to find the answers through PowerShell; it gave us a more granular view of what was happening on the Edge server.

What's your advice to your iT peers in other organizations to help them get over their fear of Powershell?
Actually, all I think they need to do is take a look at it. After less than 30 minutes of playing with PowerShell, our IT folks were saying, "Wow, this is incredibly powerful and extremely beneficial."