Robocopy FAQ

Robocopy is a powerful Microsoft Windows 2000 Resource Kit and Microsoft Windows NT 4.0 Resource Kit tool. This command-line tool can accomplish a variety of scripted copying tasks, including large data migrations and server consolidations. The 22-page robocopy.doc document that's installed with the resource kit tools has a complete explanation of Robocopy's more than 40 switches (more switches than in any other tool I've seen).

I covered the Robocopy tool in "Real-World Scripting: Data Migration with Robocopy, Part 1," March 2001, http://www.winscriptingsolutions.com, InstantDoc ID 19826, and "Real-World Scripting: Data Migration with Robocopy, Part 2," April 2001, InstantDoc ID 20134, and these are good reference documents for you to review before you try to use Robocopy for a large project. In addition, here are answers to some FAQs about effectively using Robocopy for major data migrations and server consolidations. These questions and answers should complement the information in my earlier articles and address queries I've received about using Robocopy.

How dependable is Robocopy? Can it fail to copy a file or group of files without logging an error?

Yes, on isolated occasions, Robocopy can fail without logging an error. When I say isolated, I mean it—I've had this happen only twice in 5 years. To catch any failures, always compare the source and destination locations to ensure that the same number of files, folders, and bytes of data exist in both locations. In addition, I recommend that you log all Robocopy operations and review the logs carefully. The /Log switch syntax is

/LOG:C:logfile.txt

which creates a new file or overwrites an existing file, or

/LOG+:C:logfile.txt

which creates a new file or appends to a previous file.

I often send the entire log file or a summary of the copy operation results from the log file to the responsible manager or file owner to confirm the success of the copy operation. This step helps eliminate any user questions about lost or missing files.

If Robocopy fails to copy a file or folder, how does the log file display the failure?

The log file summarizes both successes and failures in a table at the end of the file. You can also search for the word EXCEEDED in the log file to locate the specific file that failed.

What can cause copies to fail?

One of three main problems can cause Robocopy to fail to copy a file or group of files. The most common situation is that you lack Read permissions on the source directory. If you aren't absolutely sure that you have Read permissions on all areas inside the source directory, be sure to carefully flow down Read permissions throughout that structure for yourself or for the account under which you're running the script.

The second most common problem is pathnames that are longer than the 255-character limit that Windows supports. If file or folder paths are 255 characters or longer, they're often difficult to copy, modify, or delete. If you're unsure whether you have 255-character paths, see "Real-World Scripting: Locating Long Paths in a Directory," January 2000, http://www.winscriptingsolutions.com, InstantDoc ID 7828. This article includes a Perl script that detects any long paths in your directory structure.

The third problem you could encounter is a filename or folder name that contains characters that Windows doesn't recognize or that's also a name reserved for a device. To see an example of this problem, try to create a file on your desktop called com1.txt. Device names such as Com1 and Com2 and printer ports such as Lpt1 and Lpt2 are reserved names and can't be used for filenames. Occasionally, one of these improperly named files mysteriously gets into a folder structure. Robocopy can't copy these files properly and logs improper file handle errors.

If, after a copy, I compare the number of bytes at the source and destination locations and I notice that Robocopy failed to copy a file and hasn't logged the failure, how can I locate the file?

If you have a large folder structure, finding the uncopied file or files can be like finding a needle in a haystack. If only one file wasn't copied, you can use the Search (Windows XP or Win2K) or Find (NT 4.0) utility to search for the file by size. In Win2K or NT 4.0, Search or Find locates a range of files because the utility can search only for files that are bigger or smaller than a specified size. XP improves the Search utility with searches for an exact number of kilobytes.

Alternatively, you can use the Dir command on the source location to find the number of bytes you're missing from the destination location. For example, if a 5402-byte file were missing from the destination location, you could use the syntax

dir /s D:source | find "5,402"

to discover which file might be the culprit on the source location. Another approach is to run the Dir /S command on both the source and destination areas and redirect the output to two log files. Compare these two files and see which entries in the source file are missing in the destination file.

You can also use the WinDiff utility in the Win2K Support Tools or the NT 4.0 resource kit to make file and folder comparisons. Third-party tools such as TimeAcct Information Systems' Disk Triage and JAM Software's TreeSize can also show where two folders vary in size.

I've been running incremental copies every night to prepare for a big data migration and have seen really long copy times and thousands of files that are new in the log files each night along with thousands of Extra files that have been deleted. The size of the area isn't in creasing much, so what might be causing this activity?

This phenomenon could have several causes. Your shares might be undergoing significant data churn. Users might be replacing old data with new data much more frequently than you realized. Or, a user might be changing the name of an upper-level folder. When such a change occurs, Robocopy considers all the files and subfolders in that folder to be new and recopies them. A user changing a top-level folder that affects the location of thousands of files will void your prediction of how long your final copy operation might take. If you use Robocopy's Purge option, the utility takes time to delete the old file structure that has changed. These situations don't signal problems; they just mean longer copy runtimes. Other possible explanations for a slow copy operation might be network or CPU usage problems caused by scheduled tape backups running simultaneously with the copy operation or overhead created by your antivirus software scanning outgoing or incoming files. Overhead is involved when Robocopy traverses the folder structure and compares the source and destination areas, even if it doesn't copy any files.

If you're copying a few hundred or a few thousand files and you don't have 24 x 7 file usage, minimizing the copy time probably isn't that crucial. If you're dealing with half a million files, you really want to minimize the final copy time. Plan to run the final copy operation with the files either offline to users or in read-only mode. If you leave the share online and let users modify files, you might end up with an earlier version of the file or missing files on the new location. You'll also have difficulty getting the sizes of the source and destination areas to match because file sizes will be changing during the copy.

If you're running copies in a tight time period, as you get closer to the final cutover to the new area, you'll want to run more frequent copies. This practice ensures that the final cutover copy makes as few changes as possible and, therefore, that user offline time is minimized. The practice also minimizes surprises such as permissions problems, path length problems, or illegal characters during the crucial cutover time. Discovering and resolving these problems is easier when you aren't under the time pressures of the cutover.

I'm running Robocopy on a server that has one processor and isn't very powerful. When I launch Robocopy, the server hits very high CPU usage. How can I keep the CPU usage down?

If you must run Robocopy on that server, use the Start command with the /Low switch to start your Robocopy script. Alternatively, you could run Robocopy on another server, if you have a fast network connection.

I've heard that copying can cause fragmentation. How can I reduce
fragmentation?

Robocopy supports a /Create switch option that creates a folder structure of 0-byte files. Run an initial copy with this switch, then run a copy without this switch to actually copy the data. The robocopy.doc Help document that's installed with the resource kit explains the rationale for using /Create. Refer to the heading "Minimizing Directory Fragmentation."

When we performed an incremental Robocopy run, some recent permission changes didn't seem to come over properly. What happened?

Robocopy does copy security settings when you use the /Sec switch. However, the utility doesn't update the permissions each time you run it unless you use the /Secfix option. If you have a folder area in which permissions are frequently changed, always use the /Secfix option. Don't use the /Sec or /Secfix option when you've applied different permissions to the destination area and don't want Robocopy to overwrite these permissions. Remember that only NTFS volumes support file and folder permissions, so when you copy from an NTFS volume to a FAT or FAT32 volume, permissions will be lost.

We have standalone servers that use local groups for permissions. When we ran Robocopy with the /Sec switch to copy data from one server to another, it copied the data but not the security settings. What went wrong?

The local groups on the two standalone servers have different SIDs. The destination server didn't recognize the permissions because its local groups didn't have the right SIDs. You can use Small Wonders Software's Secure Copy if you need to create matching local groups on the destination server. The final answer at the end of this article has more information about Secure Copy.

Alternatively, you could use the Subinacl tool to replace the lost permissions on the destination server. The Microsoft article "Cannot Resolve Local Groups When You Migrate Files Between Windows NT 4.0 Member Servers" (http://www.support.microsoft.com/?kbid=250267) explains how. Keep in mind that Subinacl is a complicated utility. Preventing the problem with a product such as Secure Copy is easier than fixing it with Subinacl.

We have a standalone server that uses local groups for permissions and has a locally connected storage shelf full of older 9GB drives. We're going to upgrade to 36GB drives and must remove and replace all the drives at once. No other storage is available on this server. We have a second server that has the capacity to handle the data temporarily, but we're afraid we'll lose our elaborate permissions structure when we offload the data to the other server because it doesn't have the same local groups. What procedure can we use to accomplish this data migration and get the data back to the new storage with the permissions intact?

I have a procedure that I've used with good success, but I would advise you to test the procedure carefully on a small subset of folders to ensure that it works in your environment before you use it for your migration. Also, have a full backup available in case of problems.

Use the standard Robocopy syntax with the /Secfix switch to copy the data and permissions to the temporary server location. Robocopy will bring over the files with their permissions to the interim destination.

When you view the Security tab on a file or folder's Properties dialog box, you'll be unable to see the group name, but you will see the SID that represents the group on the source server. The group name is unknown on the interim server because the local group doesn't exist on this server.

After you've replaced the drives on the original server and created and formatted the new partition, run another Robocopy with /Secfix back to the original server. The data and the associated SIDs are now back on the original server on which the local groups reside and are again visible. Remember that you need to be extremely careful to verify both the source-to-interim and interim-to-destination copy operations.

I used Robocopy to copy some data to an area, then used Robocopy to copy data from an older area to fill in the gaps in the folder structure. A number of earlier files overwrote their later versions with the same filename. What happened?

By default, Robocopy lets an earlier file overwrite its later version. Be sure to use the /XO (excludes older files) option if you want to prevent later files from being overwritten.

Our management team wants me to divide a folder structure and move it to 50 servers and folder locations. Do I need to write a separate script for each of these 50 moves?

Writing a separate script for each folder would not only be time-consuming but would probably result in typos or other path errors that could cause a copy operation to fail or destroy some data. The RobocopyFromList.bat script that Listing 1 shows lets you use a comma-separated value (CSV) input file that contains the source and destination paths. Just copy or enter the source and destination locations into the file with one source and destination per line. Separate the source and destination with a comma, and put a comma after the destination. Figure 1 shows a sample input file.

Is there any way that Robocopy could destroy data?

Almost any tool that automates processes poses risks. Robocopy used with the /Purge option could delete data from the source location if you confused the source and destination locations in the command syntax. If you mistakenly named an empty or nearly empty folder as the source location and one of your large production file shares as the destination, Robocopy could delete your production file share by trying to make it match the empty folder. Fortunately, I've never had this happen, but I've exercised extreme caution. Always make a test run with the /L (list) switch to see whether the results match what you intend. An extra precautionary step would be to use a user account that has only Read permissions on the source location.

I ran two Robocopy operations to one top-level folder, but only the second one seemed to work. What happened?

I'm going to guess that you copied two folders to the same top-level folder with the /Purge switch on. For example, you copied the contents of E:SalesData1 to R:NewSales, then copied the contents of F:SalesData2 to R:NewSales. Actually, both of your Robocopy operations succeeded, but the /Purge switch on the F:SalesData2 to R:NewSales copy deleted all the files that you copied earlier from E:SalesData1 to R:NewSales. You need to either put two subfolders in the R:NewSales folder or remove the /Purge switch from the second copy operation.

I have a Robocopy script on a server. While opening the script remotely to edit it, I accidentally launched it. How can I prevent this mistake in the future?

Because Robocopy can create a folder structure rather quickly, you want to ensure that it can't be accidentally launched on the wrong computer. RobocopyFromList.bat includes a line of code that checks for the computer name.

Another approach is to force the user of the Robocopy script to pass a parameter to the script to make it launch. Use an If statement that looks for a text string, such as

If Not "%1"=="YesIAmSure" EXIT

If the user doesn't run the script with the YesIAmSure argument, the script will exit.

Are there alternatives to Robocopy for large data migrations and server consolidations?

Secure Copy is a GUI tool that has a full command-line interface. As I mentioned earlier, if you use local groups to grant permissions on a server, copying to another server and getting the groups and their permissions to match can be difficult. Secure Copy can actually create matching local groups on the new server and streamline this procedure. Secure Copy can also copy simultaneously to multiple destinations. If you've become proficient with Robocopy and want to make the next logical jump to a tool with more functionality, Secure Copy is a good choice.