Robocopy XP010 FAQ

Robocopy is one of my favorite and most frequently used Microsoft Windows Server 2003 Resource Kit tools. The new version of the utility—Robocopy XP010—has added several features that I've recently been exploring. Based on reader questions that I've received about this new version, I've put together a list of FAQs that might serve as valuable instruction about Robocopy. Let's start with one that addresses migration concerns you might have about moving data onto Network Attached Storage (NAS) devices.

I'm migrating a large amount of data from a Windows 2000 Server system to a new NAS environment. I'm using Robocopy with switches I've used in the past for Windows-to-Windows migrations. However, my copy times don't seem to be improving with subsequent incremental copies. What am I doing wrong?

Typically, after you perform an initial Robocopy copy run of data, you should see copy times drop because you're copying only the changes that happened since the previous copy run. The amount of data turnover, including newly changed or deleted files and folders, will vary depending on your environment.

If the share is relatively inactive and little change is occurring in the data, copy times will drop dramatically after the initial run because Robocopy has few changes to deal with. If the share is very active, with files (e.g., users' .pst mail files) that are constantly changing, you might see only a slight drop in the copy time because the share contents are experiencing a high rate of data turnover. Eventually, you'll get down to a time that represents the minimum time Robocopy needs to scan through the source and destination and check for changed files.

If you aren't seeing a drop in copy times, there are two possible explanations. You could have a highly dynamic data area that's experiencing a high percentage of change, or you could have a comparison problem in which Robocopy believes changes have taken place and recopies files when, in fact, no changes have occurred.

If you're dealing with a Windows-to-NAS copy task, you need to check with the NAS device vendor to determine whether your device supports NTFS file-time standards. Robocopy's /FFT switch is particularly helpful if you have a NAS device that doesn't support NTFS's 100-nanosecond granularity and might use the FAT file system's 2-second granularity. Rounding the NTFS file times might result in the copying of files that are unchanged; Robocopy senses these files as newer or older and copies them.

The /FFT switch forces Robocopy to use the FAT file-time granularity so that the utility uses a 2-second standard for comparing files. As long as files are time-stamped within a 2-second interval, Robocopy considers them identical and doesn't copy them. This switch option can dramatically reduce your copy times and minimize copying of files that haven't actually changed.

My company has a remote location that's connected by a slow link. Users are complaining that accessing the main corporate file server over the link is too slow. They want us to place a read-only copy of the file-server data at the remote site. I need to set up a script to copy data across the slow link. The process would need to be continuous because changes are always in progress. Can I use Robocopy for my task?

A new item in Robocopy XP010 is the Monitor feature, which you control with the /MON:n and /MOT:m switches. The /MON:n switch specifies the minimum number of changes that must occur before Robocopy runs again. The /MOT:m switch specifies the minimum time, in minutes, that must elapse before Robocopy runs again. For more detailed information about the Monitor feature, see the resource kit's Robocopy.doc Help file.

Typically, Robocopy terminates after it runs through a specified source folder location. The Monitor feature puts Robocopy into a continuous mode, causing it to continuously scan the source location for changes. You can use this new feature in two ways. The first way, which addresses your query, is to continuously replicate data to a remote location. The second way is to copy the data to a new location in preparation for a migration rather than perform periodic Robocopy runs.

If you plan to use the Monitor feature, you have one consideration to keep in mind. If you launch Robocopy in Monitor mode from a script, Robocopy might continue running after your script has terminated. To solve that problem, I use Task Scheduler to launch Robocopy in Monitor mode. Here's a sample of the command string I use in Task Scheduler:

C:robocopy.exe"\SourceServer  SourceShare""\DestServer  DestShare" /COPY:DAT /E /NP    /FFT /NS /NDL /MON:1 /MOT:1   /IPG:5 /PURGE/R:1 /W:1   /LOG+:C:RoboLog.txt

(Be sure you type the entire command on one line.)

If you're dealing with slow links and you're receiving Robocopy copy errors, try the new version's Inter-Packet Gap (/IPG:n) switch. I've used this switch with a value of 5 on a slow link, and it significantly reduced copy errors caused by network timeouts. See the Robocopy.doc Help file for details about the /IPG switch.

We're migrating several hundred file shares to another server location. We're concerned that Robocopy might overlook a share that exists on the source server and not create it on the destination server. How can I be sure that good correspondence exists between the source and destination servers?

The TestShareCorrespondence.bat script in Listing 1 uses the resource kit utility Rmtshare to compare shares on the source and destination servers. Rmtshare is actually a Microsoft Windows NT 4.0 Resource Kit utility that more recent resource kits haven't offered. However, the tool works fine and is still available through FTP from Microsoft at ftp://ftp.microsoft.com/bussys/winnt/winnt-public/reskit/nt40/i386/rmtshare.exe.

I tested TextShareCorrespondence.bat and the other scripts discussed in this article on Windows XP Service Pack 1 (SP1) and Windows 2000 Server SP3 PCs. To use the scripts in your environment, configure the input and output file and utility locations as each script's header explains.

Column-width limitations in this publication might introduce line-wrap problems if you type the script code as it appears in the listings. I recommend downloading the actual .bat files from the Windows Scripting Solutions Web site, http://www.windowsitpro.com/windowsscripting, InstantDoc ID 44324.

We experience problems when users create folder structures that exceed the 256-character limitation that Windows applications enforce. When we performed a recent migration, Robocopy produced a lot of errors concerning long pathnames. When I tried Robocopy XP010, I experienced no long-pathname errors. How does Robocopy XP010 handle long pathnames?

By default, Robocopy XP010 can copy the filenames and folder names that exceed the 256-character limit—to a theoretical 32,000 characters—without logging any errors. If you want to force Robocopy to produce such errors so that you can see where your long-pathname problems originate, use the /256 switch. This switch causes Robocopy to produce errors that are similar to those with which you're familiar. Remember, though, that some user applications might not be able to access paths whose names exceed the 256-character length, and long pathnames might complicate verification of a successful copy.

If you plan to use the resource kit's Diruse directory-size comparison tool or another utility to verify your copy success, the tool might have long-pathname limitations that give you different results between the source and destination. Suppose you have a share named FileServer1ShareA that contains only one file. This 10MB file, called LongFile.txt, is many folders deep in the file structure and has a 253-character path length. You use Robocopy to copy this file to another server, thereby adding one more level to the path: FileServer2ShareBOldShareA. Without realizing it, you've increased the path length by 10 characters. Robocopy will copy the file successfully (unless you use the /256 option to force an error).

If you then use the Diruse tool to compare the results, the 10MB LongFile.txt file isn't visible to the Diruse utility on the new shared location because the pathname now exceeds the 256-character limit. Instead, the comparison shows that you're missing 10MB of data and one file.

Imagine if you had thousands of files that were near the threshold and a small change took hundreds of them over the 256-character limit. Even though everything is fine, you would doubt the integrity of your copy task. At times like this, the Robocopy log file is an invaluable indicator of copy successes and failures.

I'm searching some large Robocopy log files for errors, but the size of the logs is complicating my efforts. What should I be looking for in the logs, and what causes most failures?

The report summary section at the bottom of the Robocopy log file shows failures, but I also typically run a search on the words Exceeded and ERROR in the log files to see where the copy failed. The majority of Robocopy failures I see are the result of insufficient permissions on the source or destination location. If you're copying security settings, ensure that the account you're using has Full control on the destination server.

We're migrating several hundred shares from one server to another. Can we use a script to run Robocopy on all the shares without writing individual Robocopy scripts for each share?

Here are a couple of options for you. If the servers are identical in terms of storage and have the same drive-letter configuration and drive contents, you could copy from the root of each drive instead of from the shares. The downside of this approach is that your log files would be organized by drive and not by shares.

Alternatively, you could use a script that takes input or share names from a list and runs Robocopy against each share. The RoboCopyMigration.bat script, which Web Listing 1 (http://www.windowsitpro.com/windowsscripting, Instant Doc ID 44324) provides, modifies the command-prompt window title so that you can view copy progress at a glance and see which share you're on in the share list. RoboCopyMigration.bat creates logfile names in a format that combines the share name and a timestamp of when the copy run started. Note the use of the new Robocopy /TEE switch, which sends output to both the log file and the console.

I don't fully trust the log files on copy utilities. I want to independently verify that correspondence is 100 percent accurate between the source and destination locations after my final copy is finished. What's the easiest way to do that?

You're wise to use an independent tool to verify your copy operations. You can use several GUI tools—such as TreeSize (http://www.jam-software.com/freeware)—to perform comparisons. If you prefer to stick with a scripted approach, the resource kit's Diruse tool works well. I like to check on the number of files and the total amount of data in the source and destination folders. If these numbers agree, I feel confident in the success of the copy job. The MigrationCompare.bat script in Web Listing 2 creates an output file that gives you file-count and data-size information for each share in your share list.

I've introduced you to Robocopy XP010 and a few of its powerful new switch options. I also covered some tips for performing migrations to NAS devices, using Robocopy for replicating over slow links, and verifying that your copy operations and share creations are accurate. Armed with these scripts, strategies, and techniques, you should be ready to take on any migration tasks from a few hundred megabytes to several terabytes.