DiskXtender 4.2

Manage your media with robust Hierarchical Storage Management capabilities

As your business accumulates data and you move historical data to electronic media, you need efficient storage strategies. However, these strategies can be costly to administer and complex to access. Hierarchical Storage Management (HSM) products can help simplify your storage process by providing one management interface for online and nearline storage (i.e., local and remote storage) and presenting a common virtual entity that presents multiple volumes as one continuous library. OTG Software's DiskXtender (DX) 4.2 is a mature HSM product that can manage many storage devices in a Windows NT HSM scheme. The software's Server, Agent, and Administrator components and additional utilities offer flexible file migration, file caching, powerful yet simple administration, and automated diagnostics and scheduling.

DX Server resides on a server with a physically attached DX-supported storage device. (DX supports a huge array of storage devices, including optical drives and jukeboxes.) DX Server manages nearline storage hardware and configuration information for each DX Agent system and that system's extended drives (i.e., NTFS volumes with migration services that let you move data to an attached nearline-storage device). DX Agent functions as an NT service that permits communication between the DX Agent system and DX Server. You can install DX Agent on the same system as DX Server or on remote computers throughout the network. After you install DX Agent, you can use DX Administrator to extend any NTFS volume on the DX Agent system. DX Administrator provides a single point of administration for hardware, software, media, schedules, and reports. DX's Windows Explorer add-ons give users access to HSM tasks such as enabling or disabling direct read for a file and viewing DX-specific file attributes.

Getting Started
I installed DX Server on a Gateway ALR 9200 server with four 400MHz Pentium II Xeon processors and 2GB of RAM running NT Server 4.0 Service Pack 6 (SP6). I attached a Hewlett-Packard (HP) SureStore 320ex optical jukebox with four drives and an Advanced Digital Information Corporation (ADIC) Scalar 218 tape library with two DLT 7000 drives. I verified that my nearline-storage devices were on the OTG Web site's extensive list of supported erasable optical, CD-ROM, DVD, tape, and write once, read many (WORM) devices, then I installed the product. The wizard-guided installation was straightforward and gave me the option to install DX Server (this option also automatically installs DX Agent and DX Administrator), DX Administrator, and DX Windows Explorer add-ons. (I could also choose to upgrade from an earlier product version.) Installation took about 2 minutes; the wizard then prompted me to press Enter to initialize DX Server and DX Administrator. DX Server started, then DX Administrator opened and automatically attached to DX Server.

Configuring Storage Devices
The left pane of the DX Administrator window lists icons for extended drives, media, and hardware. To add my optical jukebox, I right-clicked the Hardware icon and let the Hardware Wizard guide me through detecting, naming, and configuring my hardware. The options let me control how DX managed the disks and media. I accepted the defaults, and the wizard automatically configured the jukebox's disks. After a required server reboot, DX Server inventoried the jukebox platters, and DX Administrator reported my media's status as foreign. (Because I previously used my media in a UNIX environment, DX didn't recognize the media's file-system format.) I used DX Administrator's Media Prepare Manager tool to format and label my optical media. (If I had wanted to keep the data that was on my optical platters, I would have needed to copy the platters' contents to an intermediate storage location, then migrate the data back to the platters after I configured DX.) DX seamlessly formatted and labeled 22 double-sided optical disks and moved them to the unassigned media category (i.e., the disks were formatted and ready to use but weren't yet assigned to an extended drive). I followed the same procedure to set up my tape library. The only difference was DX's requirement to add the proper NT tape drivers for the DLT drives; DX Server generated an error message that notified me of the incorrect driver configuration.

Creating an Extended Drive
After I configured my nearline-storage devices, I was ready to create an extended drive and migrate data from my hard disk to my optical media and tape library. I installed DX Administrator on my desktop; I could then remotely control all DX operations. I consulted the manual, which OTG provides as a hard copy and as a Portable Document Format (PDF) file on the DX CD-ROM, for instructions about getting started. The documentation was very thorough but somewhat repetitive. I chose to set up an extended drive for my server's 13GB E drive. DX Administrator presented, in order, icons that represented the extended-volume setup steps.

First, I selected E: as the drive for DX to extend. Next, I selected 18 blank optical platters and two blank tapes as the media that DX would make available to the extended drive. I then created two new move groups for the drive. A move group defines the nearline media device to which files will migrate. I configured the first group, which I named Music Files From ALR, to use random media fill (rather than the default sequential fill) and write to the nearline optical media. I configured the second group, which I named Historical Data From ALR, to use random media fill and write to the nearline tapes. (Sequential fill writes contiguous data; I chose random media fill to leverage the speed advantage that I gained by concurrently writing to multiple drives in both my libraries. Data read and write patterns dictate whether a random or sequential media fill method will be more efficient in a particular situation.)

Next, I created move rules, which control how and when DX migrates files to nearline media. You can specify filename (using wildcards), size, last access, last write, creation date, attribute, or location as migration criteria; you can select multiple criteria for each move group. I created two move rules: one for multimedia files and one for historical files. I defined the rules so that DX would move all files under E:multimedia to the Music Files From ALR move group and all files under E:historical to the Historical Data From ALR move group. Then, I set up purge rules, which control how DX deletes files from the hard disk. My rules made all files candidates for purging but directed DX to purge files from E:historical before purging files from E:multimedia. The default purge watermark settings told the program to start purging when the disk reached 95 percent capacity and to stop when the disk reached 90 percent capacity. I decided not to set up any delete rules, which control the permanent removal of files from nearline storage.

Many performance and functionality tuning options were available during extended-volume configuration. I accepted default values unless I needed to tweak something for testing purposes.

Migrating Data
After putting my rules in place, I forced a scan of my extended drive. The scan lasted about 30 seconds, then DX immediately began writing data simultaneously to all my nearline storage devices. As the program migrated files to my optical and tape media, I copied more data to the server's hard disk. I quickly surpassed the 95 percent watermark, and a purge cycle began to delete hard disk files that the program had already written to the nearline media. DX Administrator continuously updated my extended drive's statistics, as Figure 1, page 124, shows.

When the migration finished, I had placed more than 30GB on my extended drive; about 12GB of that also remained cached on the hard disk. The write throughput was reasonable for both media types: Concurrently using multiple drives, I averaged about 1.5MBps to the four optical drives and about 3.3MBps to the two tape drives. I attained these throughputs with write verify on. Theoretically, turning off write verify will result in faster speeds.

With my data now physically stored on three devices, I used DX Administrator's Prefetch Manager to set up a prefetch request. Prefetching lets you schedule copy operations (e.g., to copy frequently accessed files to more easily accessible media during off-peak hours). I created a direct prefetch request by browsing for and selecting files to prefetch; an indirect prefetch request can parse a file list from a formatted text document. I chose the ASAP option to schedule the prefetch to occur immediately after I finished setting up the request. Then, I selected the E:historical subfolder, and the program added to my prefetch list all the files in that subfolder (43 files—165MB in all). I clicked Finish, and the prefetch request's status changed to in progress. The status changed to complete after about 5 minutes. I clicked Log to open the Prefetch log, which confirmed that the program had copied the selected files from tape to the hard disk.

Next, I tested the program's ability to open files that I had stored on nearline media. On extended drives, DX employs what OTG calls a spy driver (i.e., a driver that sits on top of NTFS and handles incoming file requests). On migrated files, you can enable a direct read attribute that tells the spy driver to read the files directly from the media on which the files reside. If you don't set this attribute, the spy driver simply copies the files to the hard disk and leaves the rest of the transaction to NTFS. DX permits fairly granular control over requests at the individual-file level. I could access the files, but reaching a purged file stored on an unmounted tape took a few minutes. The average access time to individually retrieve a random sample of six files from unmounted tapes was 2 minutes and 44 seconds per access; retrieving the same types of files from unmounted optical platters averaged 10 seconds per access. This disparity illustrates the need to consider throughput and access time when you implement an HSM system.

Additional Utilities
DX comes with some useful and intuitive utilities, including Report Generator, Repair Disk, Jukebox Manager, SCSI Manager, Copy Media Manager, and Windows Explorer add-ons. The manual explained how to use the tools, but I had to jump between sections to get a picture of each tool's functionality.

Report Generator's nine canned reports provided accurate, easy-to-read statistics regarding my media, hardware, and extended drive. The Extended Drives Report, which Figure 2 shows, illustrates DX's simple yet useful report format. To facilitate disaster recovery of an extended drive, I used Repair Disk to copy my extended drive's configuration information to another drive. Jukebox Manager and SCSI Manager helped me configure and manage my nearline devices. (Figure 3 shows Jukebox Manager's view of my tape library.) I used the Copy Media Manager to schedule a copy of one of my optical disks. The Windows Explorer add-ons, which you can install on any DX Agent machine, let me view an extended drive, select files on that drive, and perform HSM-specific end-user tasks (e.g., enable or disable direct read, purge, view a file's migration statistics) from Windows Explorer.

DX also includes a media compaction tool that maximizes nearline-storage utilization. When you delete files from tape or optical media, the gaps left behind are no longer writeable. I deleted about half my data from the extended drive, then I ran the media compaction utility on 6 of my 12 optical disks. The utility moved data in a contiguous format from those disks to other optical media and returned the selected disks to the unassigned media category.

A Great Tool for Advanced Needs
Implementing DX is easy if you understand storage devices and data requirements. The manual seemed disorganized to me, but I was impressed with the control that the product gave me over file systems and nearline-storage devices.

DX's price ranges from $599 to $60,000, depending on the type and amount of storage you plan to connect to DX Server. (The $599 price covers a server with one attached optical drive; the $60,000 price covers an enterprise storage library with several thousand pieces of media. The price for the system that I used was $8999. DX Agent licenses to extend drives on remote servers are available for $1299.) These prices might compel you to look at alternative storage solutions, but if you need effective HSM at any cost, I recommend DX 4.2.

If you're an NT user considering a move to Windows 2000, you might wonder how DX will work with Win2K's Remote Storage Service (RSS—for more information about RSS, see Douglas Toombs, "Unlimited Storage," June 2000). OTG designed DX 4.2 for NT, but the company claims that you can port DX 4.2 to Win2K. (OTG also offers a Win2K-specific version—DiskXtender 2000.) Although RSS is a welcome feature, it falls short of providing the robust enterprise-ready functionality (e.g., remote installation, extensive migration criteria, optical-media support, automated file deletion, nearline media content reports, media compaction) available in DX 4.2.