Real-World Exchange 2010 Migration: Staging the Move

Here at Windows IT Pro, our goal is to bring IT pros the information that helps you get your jobs done. Sometimes that means telling the stories of others who have already done a particular task. We're fortunate to be able to talk with our own IT department about the tasks they're involved in, such as the recent upgrade to Microsoft Exchange Server 2010. Like many organizations out there, our IT staff has to deal with budget constraints, employee adoption and communication, and justification to management for any changes they want to implement.

In the first article in this series, I began the discussion with the two members of the company's IT team most responsible for handling the transition to Exchange 2010, senior systems architect Brent Mammen and Exchange administrator Sean Cox (see "Real-World Exchange 2010 Migration: Preparing for the Move"). Now, we move on to talking about the actual migration and the various problems the team encountered—and had to overcome—as the move progressed.

As I mentioned in the previous article, Brent and Sean support over 20 locations, with a few international offices. The IT department is based in Overland Park, Kansas, and at the time that the Windows IT Pro office was moved to Exchange 2010, we were located in Loveland, Colorado. (We've since relocate to a much nicer office in Fort Collins, Colorado.)

BKW: How was the actual migration staged? You basically split the migration by office location?

Sean: Moving the offices one at a time seemed the best way to do it.

Brent: And I think we might have started with you guys. We did Overland Park, and then we started with your offices. We were going to do it by office originally, and our plans were changed slightly as we got into it. Service Pack 1 for Exchange had just been released when we were starting to migrate mailboxes, so the decision was made to go ahead and apply Service Pack 1 for Exchange. Well, an issue that arose with Service Pack 1 was ActiveSync policy support had changed. I think we have a little over a hundred Android devices in our organization. Some of those Android devices weren't able to handle those new ActiveSync policies, depending on the version of ActiveSync client that they had. What we ended up doing is breaking the people out that had the Androids and doing those separately as we went. So we pulled those out of each group, and handled those separately. We had to actually remove the ActiveSync policies from a lot of Android devices to get them to sync with Exchange.

Sean: The removal of Android devices, doing them separately in each office, was based on trying to do more communication to those individuals. We would let you know that this might be an issue after the upgrade; if it is, please let us know and we'll address and resolve whatever issues that you have.

Brent: Also, we set up what we called the Exchange Migration Hotline. So we had a phone number that people could call, internally and externally, during the migration, as they were being moved, if they had a problem. That phone line went to both Sean and I.

Sean: The email address that those communications were sent out under both Brent and I had access to as well. So you had two lines of communication to get with us. We did it that way because we were the most familiar with what was going on, instead of trying to flood the Help desk with calls. We decided to take that on individually and deal with any issues that arose—kind of cut out the middle man.

Brent: Right, that took some pressure off the Help desk. I mean, they still did receive calls, but then they transferred those calls to the Exchange Hotline. Sometimes we were migrating accounts at night, and if people had an issue, they could still call the phone number and with mobility, could reach us on our cell phones.

Sean: It takes a day or two to migrate the office, then we know we could still have issues or questions a week later from people who are just coming back into work when someone's out of the office for a week.

BKW: I remember that email right before the move. It mentioned letting you know if you were using an Android device—which means me. I didn't have any problems after the migration.

Brent: They talk about Android fragmentation, and it does exist because you have different vendors with different email clients. It's not like the iPhone where you have one email client. The different implementations didn't support certain policies and new policies, ActiveSync policies that were introduced with Exchange 2010 Service Pack 1. We didn't really know which clients would work and which wouldn't. We had some of them narrowed down, but if they hadn't done an update on their device, it still might not work. Obviously, newer phones tended to be less problematic, newer Android devices. That was the first way we tackled it, was to make sure that everyone was on the latest version of the OS for their device.

Sean: We did weed out some of the initial issues, or found out what some of those would be, just by migrating IT in general. IT's always a good test group because, well, you work with them, you can let them know what's going on. If something goes wrong, they're not really going to complain too loudly to you because they kind of expect issues, being in the industry—nothing ever works perfectly the first time out. We weeded out and found out about certain issues with the Androids before we ever started rolling this out to any of the regular users in the offices.

BKW: Do you use a variety of smartphones in IT?

Sean: Yes, we have Androids, different flavors of Androids, iPhones, BlackBerrys. So it was a pretty wide range of devices. We were able to find out if they'd be problematic or not before we ever touched the regular user base.

Brent: I think that being a media company lends itself to allowing people to be more creative, more flexible. So the company has allowed people to choose pretty much any devices they want. I think that's great. It does present challenges for us in supporting those devices, but I think it's good for the company.

Sean: But even those challenges are always a learning experience, especially when you go out to the larger user base, in different offices, with different devices. We kind of get more of a handle about what could come up because of the openness of our IT department.

BKW: This is the big trend, allowing employees to choose the devices they use. It can certainly be a struggle for IT departments.

Brent: Yes, it is. More than half of our employee population has devices attached to our Exchange. We have more than 700 devices on our Exchange system. Although, looking at it now, it wasn't as problematic as I thought it might be. It was actually pretty smooth. We had a hiccup with our BlackBerry Enterprise Server and our BlackBerry devices, and I believe it turned out to be a patch that we had to apply.

Sean: Initially when Service Pack 1 came out, I had seen that BlackBerry fully supported it. Then we started having issues with the BlackBerrys, and then I found out that BlackBerry said they only partly support Exchange 2010 Service Pack 1. So I went from seeing full support to part support. They did at the time end up having a patch, an update actually, for the BlackBerry Enterprise Servers to resolve the issues.

Brent: That was pretty quick—I think the issues with the BlackBerrys didn't persist for more than a day or two, and they weren't total outages.

Sean: No, no, it was more calendar-based, the issue that people were having.

BKW: You started out talking about moving the offices one by one. How did you manage the back-end changes? You mentioned before about using this upgrade to consolidate the number of servers the system is deployed on. How did you run the actual move from the Exchange 2007 servers to the Exchange 2010 servers on the back end?

Sean: The process itself, I think, started from the beginning, in a lot of the reading that we did. Microsoft would have recommendations on combining the roles into actual physical servers. Before, on Exchange 2007, we had Mailbox servers separate in a CCR cluster. We had two Mailbox servers, and they were clustered together with two other servers. Then, Hub and CAS were on separate servers. Part of the migration was doing a P2V on those Hub/CAS servers to free up some actual physical hardware. So for the course of the migration, all of the roles except for the Mailbox roles were virtual. Then, as we reallocated the physical servers for Exchange 2010, we started installing Exchange, like we said, the CAS/Hub/Mailbox on one because the DAG allowed us to install roles on top of the Mailbox server role and still have the redundancy with the DAG. In 2007, if we clustered, the Mailbox server role was the only one that could be on those boxes.

Brent: I think I mentioned that we were short on physical hardware, so that's one of the reasons that we did this. But we actually needed some physical hardware in order to start the migration. Sean alluded to, we virtualized some of the boxes, and we took that existing hardware that we had just moved the OS off of onto a virtual machine, we took that physical hardware and used it for our mailbox resources. We built out a 2010 environment beside our 2007 environment. We made sure that we had that up the way that we wanted it—the database availability groups online, test mailboxes on there. We basically ran that for about a month with test mailboxes connecting to it, with Outlook and various clients, made sure everything was fine before we set up the routing between the two systems. So we have Exchange 2007, Exchange 2010, both working fine in their own environment, and then we set up routing. The way that routing works is that most of the requests you send through the new Exchange 2010 system, which redirects if it needs to through the 2007 system.

Sean: A lot of the testing that was involved was testing the DAG—the speed at which a database would fail over from one server to another. We wanted to kind of get a handle on the process, the set up, the administration, and then just the sheer speed. You know, is this faster or slower than a CCR cluster mailbox rolling over from one server to another? What issues are there? How is the management of it, before and after the fact? We did a lot of testing with the DAG on 2010 before we started migrating the mass user base. And everything worked out pretty well. We hit a couple glitches with the DAG testing. From 2010 RTM, a lot of those were definitely resolved in SP1, but as a whole it ran pretty well. Speed-wise, it was a lot faster in activating a database copy from one server to another, compared to a cluster server failover.

BKW: DAGs work as advertised? A good edition?

Sean: Yes. The biggest thing I think we like about it is that the databases are no longer tied to a physical server. They're tied to the organization. It doesn't matter what server the databases are actually on.

Brent: It's helping us set up our DR [disaster recovery] site, and beyond that it's helping us recover mailboxes. The next step in our DR configuration is to set up a lagged copy of our databases, so instead of going back to tape to recover mailboxes, we can go back to an Exchange Store that's running a lagged copy to recover those. So we're getting less and less dependent on tape with the expectation to get rid of tape sometime soon.

Sean: That would be the goal, and I think in all honesty that would be the goal of anybody in IT, whether it's Exchange or anything else. The added features of Exchange 2010 have turned the light at the end of the tunnel on in that respect, where we can see a possibility where that could be the case.

[Um, of course at that point we had a discussion about some of the things speakers had to say at the recent Exchange Connections conference with regards to lagged copies, specifically that Microsoft is no longer generally recommending their use. Sean and Brent determined that more investigation on the matter was needed.]

In the final installment of this interview, "Real-World Exchange 2010 Migration: Implementing the New Stuff," we talk about some of the odds and ends of the migration and how the new features of Exchange 2010 are working in the environment.

Follow B. K. Winstead on Twitter at @bkwins
Follow Windows IT Pro on Twitter at @windowsitpro

Related Reading: