Software Testing Product Review: Gurock TestRail 2.1

Rating: 3 out of 5 stars
Website: www.gurock.com/testrail
Price: $235 per user

We often relegate the test-management phase of an application's development life cycle to a shared spreadsheet, a wiki, or some other home-grown solution. However, these options fail to keep everyone on the same page, ensuring that all testers know the established criteria for testing. And it can be a pain to cull regular status reports from multiple sources. Large enterprises can afford full application life-cycle–management suites, but smaller organizations and virtual teams generally can’t. Fortunately, Gurock Software has a web-based software testing solution that can scale from a small five-person team to a large, multi-project enterprise deployment.

Setup Hassles

Because Gurock's TestRail is based on PHP, not on ASP.NET, you do not install the web application on a Windows-based server by running a setup.exe file. Instead, you must first use the Microsoft Web Platform Installer to configure dependencies for PHP, SQL Server or MySQL, and FastCGI. Second, you retrieve and install several PHP prerequisites, such as the cURL and mbstring PHP extensions. Third, you manually add these extensions to the Php.ini configuration file. Fourth, you manually create an empty TestRail database and set file permissions, and then you can finally open the web-based TestRail Installation Wizard, shown in Figure 1.

Figure 1: Installing TestRail

This whole procedure took nearly an hour on a Windows-based server. It actually took me less time to finish the installation on an Ubuntu LAMP virtual machine than on its Windows-based counterpart. Clearly, this application was written with Linux-based preconfigured LAMP servers in mind. So if your needs on the server side are platform-agnostic, it's easier to set up TestRail in Linux than in Windows.

Getting Acclimated

After the product is up and running, you’ll find the UI to be responsive and intuitive. The tabbed UI design that Gurock uses in TestRail, shown in Figure 2, resembles those in some popular Ruby on Rails–based project management web tools, such as Basecamp or Redmine. However, those similarities end when you start using TestRail's test metrics. The graphs that track project activity and test status are attractively presented through hyperlink-rich report screens.

Figure 2: Creating a new test project in TestRail

The TestRail UI is organized into distinct sections that let you quickly access project status, test cases, plans, pass/fail percentages, and overall progress. Pages have minimal graphics, so they refresh quickly and have a clean look. Fields have easily identifiable labels, drop-down menus are useful and well-placed, and links are relevant. Overall, the presentation is optimized for test engineers who prefer substance over style.

Using the Application

After examining its UI, I put TestRail to the test. I was constantly impressed by the thoughtful design and organization of the application. Although I still had to perform and assign the tests myself, TestRail kept everything neat, concise, and visually on target.

TestRail includes most of the management features that testers have come to expect from similar types of web-based tools, such as milestone tracking, detailed activity reports, individualized to-do lists, and email notifications. But the fact that TestRail can integrate with so many bug-tracking tools (BugTracker.net, FogBugz, Bugzilla, MantisBT, OnTime, Trac) makes it convenient for anyone who uses multiple tools for application life cycle management.

As for the efficiency factor, it can take less than a minute to create a new test project. Of course, it takes a lot longer to populate the test suites and cases, depending on how granular the test cases need to be. Fortunately, you can import (and export) test suites and cases to help speed up the process. Result histories let you include attachments with the test runs, so you can make sure that all your requirements documents, references to screen captures, and other related items are available in one location. Test assignments and personal to-do lists help maintain standard formatting across individually assigned tasks.

In addition, you can print reports that have a variety of layouts and levels of detail, which means that you can customize them either for internal use or to share with customers. It’s also easy to add project milestones, although TestRail has a limited capability to seamlessly integrate with other project-management suites. TestRail does support Active Directory and the ability to customize defect URLs (URLs that let you link TestRail to bug trackers and project management tools), so you can make it work with many web-based issue-tracking and bug-reporting applications.

The Verdict

Overall, TestRail performs admirably and delivers most of the features that test teams require to keep their test results organized and on track. Although it doesn't include all the bells and whistles of its significantly more expensive competitors, it does meet the threshold of the 80/20 rule of software testing (e.g., allowing you to detect and fix 80 percent of the bugs in your application, found in 20 percent of the application's code). However, the setup process could stand improvement because it puts an unnecessary burden on the system administrator. And, because TestRail is based on PHP, some organizations may reject it outright because of security concerns or because of the overhead that needs to be added to a stock Windows-based server in order to get the PHP runtime and related dependencies installed and running.

Fortunately, everything else flows nicely after the one-time effort of installation is complete. The program has just the right feel and number of features for test team members to get comfortable and productive right away.

Mike Riley ([email protected]) is an advanced computing professional specializing in emerging technologies and new development trends. He is also a contributing editor for DevProConnections. Follow Mike on Twitter: @mriley.