Dynameasure by Bluecurve: Born to Measure

Dynameasure software by Bluecurve lets you test and troubleshoot performancein a client/ server environment. Although this simple statement of purposeimplies that Dynameasure is a modest tool for limited applications, Dynameasureis highly flexible for a variety of applications such as server benchmarking,capacity planning, stress testing, and network analysis.

Dynameasure is different from most other performance-testing softwarebecause it uses actual client systems to simulate the client load. When youdeploy Dynameasure, "motors" running in a client system generate theworkload. You fire up the motors from a central control point, and they ridethrough your network like a band of Hell's Angels looking for trouble. Eachphysical client system can run from one to 20 motors, so you can, for example,use only 100 client systems to simulate the load of 2000 clients. The motors cancoexist with other client-side applications, so you can test while end users areworking--this capability can be pretty handy during capacity planning if youwant to consider your existing client workload.

Bluecurve specifically geared the current version of Dynameasure towardtesting Microsoft's SQL Server environment. The client motors connect to the SQLserver via Open Database Connectivity (ODBC) links and use a mix of transactionsto interact with a test database. Dynameasure then stores the transactionresults in a separate results database. Bluecurve recommends one SQL server tosponsor the test database and a separate one to handle the test results. Thatway, the I/O load of updating the result database does not affect the I/O loadof the test database. You have control over the size of the database and thetransaction mix (e.g., the frequency of read, write, and mixed read/writeoperations). In fact, the amount of control you have over the test environmentand transaction behavior is one of the most impressive aspects of Dynameasure.

Future versions of Dynameasure will support other databases such as Oracleand Sybase, file services, messaging engines such as Exchange, and Web servers.Bluecurve is clearly trying to position Dynameasure as the single tool you needto handle enterprise-level benchmarking and performance testing for all yourclient/ server applications.

How does Dynameasure measure up as an analytical tool? To answer thisquestion, Windows NT Magazine brought Dynameasure into the Lab.

In this article, I'll look at the installation, configuration, andoperational aspects of Dynameasure. Subsequent articles will show you theresults of tests that use Dynameasure to assess the scaleability of SQL Serveron a variety of server platforms.

The Dynameasure Operating Environment
To install Dynameasure, you need to start with at least one operational SQLServer system (again, Bluecurve recommends two), one client system that youdesignate as the test manager (the control point), and at least one clientsystem to run motors (if necessary, you can concurrently use the test managersystem as a client system and run motors on it). The more client systems themerrier, but starting small and progressing upward is probably good.

The SQL server can be Intel-based or a Digital Alpha and can run SQL Server6.0 or 6.5 under Windows NT Server 3.51 or 4.0. The client systems must beIntel-based and can run Windows 95, NT Workstation 3.51 or 4.0, or NT Server3.51 or 4.0.

I initially tested one SQL Server system with one test manager system andone client system. The specific hardware configurations for these tests were asfollows:

A 10 Mbits per second (Mbps) Ethernet LAN interconnected all these systems.

Install and configure your SQL Server system before you installDynameasure. In particular, be sure to establish your network and userpermissions so the test manager and client systems can access the SQL server.Also be sure to install ODBC support for SQL Server. If you are not familiarwith SQL Server, I strongly recommend that you have access to someone with SQLServer experience during the installation process--a little humility here willprobably save you a lot of grief as you progress through the installation steps.

Once your SQL Server system is in place and you are confident in yourclient/server network connections, you can install Dynameasure. The installationprocess involves four phases: manually creating empty Dynameasure databases onthe SQL Server, setting up the test manager system, loading test data on the SQLServer, and setting up the client systems (the motors). After completing thesefour phases, you're ready to begin testing with Dynameasure. (Before you jumpinto the testing aspect, you can look at the sidebar, "Installing andConfiguring Dynameasure," to get a better idea about the installationsteps, which I found to be somewhat quirky.)

Dynameasure in Action
Once you successfully navigate all the twists and turns of Dynameasure'sinstallation and configuration, the product's operation is clean and crisp. Youbegin the test process at the test manager system by invoking the Managerprogram, which then prompts you for the Dynameasure access password. After youenter the correct password, the Manager program automatically launches theDispatcher program so you can define or start a test. Screen 1 shows theDynameasure Dispatcher running in the foreground with the Dynameasure Managerrunning in the background.

When the Dispatcher starts, it establishes contact with the SQL Serversystem you are using in your test environment. The Dispatcher also checks to seewhat client-side operator programs are available--which leads to a veryimportant point: To run a test, you must have the Operator program running ineach client system you want to use. Each Operator program is responsible for allthe motors running on its host system, so the Operator is a key player in thetest process.

The interaction among the Dispatcher and the Operator programs is channeledthrough Dynameasure's control database. Each Operator program checks in when itis started and connected, and the Dispatcher sees check-in messages as theyarrive. Available operators then appear in the Clients area of the Dispatcherdisplay. For example, Screen 1 shows that the Dispatcher has detectedone Operator, so one client icon appears in the Clients display area.

The Operator also provides useful information about the hosting client. Forexample, the Operator reports the processor type, operating system, memory size,and the hosting client's CPU utilization. You can view this information throughthe Dispatcher by selecting the client in the Clients display area and choosingthe Machine detail option from the Resources menu. Screen 2 shows asample Machine Information display. This information is valuable because it letsyou judge what kind of load (i.e., how many motors) that specific client canreasonably generate.

Once you have the Dispatcher and the Operators running, your next step isto select the test to run. As Screen 3 shows, Dynameasure includesseveral OnLine Transaction Processing (OLTP) tests that can simulate a varietyof read/write conditions. In fact, Bluecurve designed Dynameasure's OLTP testenvironment to simulate a mainstream order-entry business application.

To select the tests to run, click Add Test in the Dispatcherwindow. A list of test specifications will appear. Select the tests you want,and press Add to Queue to enter them into the Dispatcher. Once you placethe test in the Dispatcher queue, you can edit the test's characteristics bydouble-clicking the test entry in the queue. Editing the test specifications isoptional, but this option is where you can control the length of the test, thenumber of motors, the number of steps to perform, and even how to weight thevarious I/O operations. Screens 4 and 5 show an example of thetest information you can edit. After you edit the test, you can save it backinto the queue or save it to disk for subsequent use.

When you have placed all your tests in the Dispatcher queue, click StartQueue to begin the test. At that point, the Dispatcher determines what Operatorsare available and how many motors each Operator will handle. Based on thosefindings, the Dispatcher will send messages out to the Operator programs andhave them start their motors. This phase of test preparation can take some time,depending on how many motors you need to start. From the client perspective, themotors appear on screen as they start. Screen 6 shows a typicalclient-side view of an Operator and its motors.

The Dispatcher display also changes to reflect the state of the operatorand motor environment and the state of the test. For example, Screen 7shows a test in progress that uses six motors running under one client operator.As the test progresses, the motor indicators change colors--blue for ready,yellow for wait, gray for not-quite-ready, green for in-progress, and red fortrouble. The Progress tab on the Dispatcher also changes to reflect the overallstatus of the test in progress.

You can divide each test into steps. Each step can bring into play aspecific number of motors. So, for example, you can define a six-stage test thatstarts with one motor and proceeds to five, 10, 15, 20, and then 25 motors. Thisstepwise approach lets you vary the load over time to reflect your userenvironment. Each step is then further broken down into phases.

A step begins with the Warm-Up phase, which includes a start-up limit togive the motors enough time to get started. The process proceeds through theremainder of the Warm-Up phase, in which motors complete their initializationand get ready for the test. After Warm Up comes the Measure phase, which iswhere the motors generate the transaction load. On completion of the Measurephase, the motors go into the Cool-Down phase, where they complete any pendingoperations and then proceed to the Report and Sync phases to report theirfindings to the control database. The total time you assign to the steps usuallydetermines how much time to allocate to these phases, but you can edit theindividual phases.

On completion of the test, you can use the Dynameasure Analyzer program toview the results, as shown in Screen 8. The analyzer lets you view theresults of a specific test, or you can overlap test results within a test run tosee how different trends look in relationship to one another. For example, youcan look at how read-only performance compares to read-write performance. Don'tbe fooled by how simple the Analyzer looks at first glance--it is a powerfulpart of Dynameasure that lets you draw meaningful conclusions from your testruns. We'll spend more time on the topic of using the Analyzer in future issuesof Windows NT Magazine. Note that you can greatly augment theinformation you get from Dynameasure tests if you run the NT Performance Monitor(Perfmon) on the SQL Server during the Dynameasure tests. When you use themtogether, Perfmon gives you an inside-NT view of what is happening to yourserver, while Dynameasure gives you the outside view of your client/serverenvironment.

Does Dynameasure Measure Up?
Aside from the quirky installation and configuration problems I encountered,I found Dynameasure to be a sophisticated instrument that you can customize toaddress a broad range of tests. This sophistication is, of course, not without aprice: A "foundation" license for Dynameasure costs $29,995 andsupports up to 50 motors. Support for additional motors costs $2495 per set of25 motors. So to create a test environment that emulates 300 users, you'relooking at a price tag of $54,935.

Is Dynameasure worth that kind of investment? That depends on how importantaccurate performance data is to you. To give you a better view of howDynameasure can pay off for you, the Windows NT Magazine Lab will beusing Dynameasure to conduct some key benchmarks for publication in the nextseveral issues of the magazine. (For information about why we chose Dynameasureas our Lab's benchmarking tool, see Joel Sloss, "Birth of a Benchmark.")We will use Dynameasure to compare NT 3.51 and 4.0 performance, to assess SQLscaleability, and to measure the relative performance of a variety of serverplatforms currently on the market. We hope our Lab experience with Dynameasurewill give you insights into the potential usefulness of the product in yourenvironment. See you next month--and remember, keep those motors running.