How Docker Containers Help, Hinder Software Testing and QA

By now, most of us have heard about how Docker containers make software delivery faster, easier and more stable. However, there has been considerably less conversation about what containers mean for software testing and quality assurance. That’s surprising, because there is a lot to talk about on this point.

Let’s explore this topic by considering all of the ways in which containers change the way we test software, for better and worse.

Containers and Software Delivery

Before jumping into the details of containers and software testing, let’s recap what containers mean for software delivery in general.

In most ways, containers make software delivery simpler and more predictable because they provide a consistent deployment environment that can be used at all stages of the delivery pipeline. In other words, whether you’re building your software, testing your software or deploying software in production, you can use the same environment--a container--to host the software.

This means that you don’t have to worry nearly as much about how differences between your development environment and your production environment will impact your application. This advantage, combined with Docker’s ability to consume resources more efficiently than virtual machines, is the major reason why containers have become so popular during the past five years (despite the fact that containers are a much older concept).

How Containers Make Software Testing Easier

If you are responsible for software testing or quality assurance, containers can make your job easier and harder at the same time. To understand how, let’s start by examining the ways in which containers simplify software testing.

1. Fewer environment variables

Perhaps most obviously, containers reduce the number of environment variables that you have to worry about testing for. You can package most of an application’s dependencies and configuration data inside a container, so that external variables won’t affect the way an application runs. As a result, you don’t need to test an application under as wide a range of environment configurations as you typically would. Instead, you can simply run your tests based on the variables defined within the container that hosts the application.

2. What you test is what you get (in production)

Similarly, because containers (as noted above) provide a consistent application environment that you can use during both the testing and deployment stages of the delivery pipeline, you can have a higher degree of confidence that the way an application behaves during testing will accurately reflect the way it will run in production. There are fewer chances for surprises because--in theory, at least--the test environment is identical to the production environment, at least as far as your application is concerned. In practice, as noted below, there can be some subtle differences between a containerized testing environment and a containerized deployment environment, but those differences are generally not as great as they would be if you didn’t use containers.

3. Simpler test branches

It’s common for software testers to have to test multiple versions of an application. For example, you might have both a Windows and a Linux version of your application that descend from the same codebase but must be tested and deployed separately. You can certainly test each version of the application without using containers. However, containers can make it easier to maintain multiple test branches because they allow you to set up an isolated containerized environment for testing each version of the application. You can reuse the testing container whenever the application code changes and you need to run tests again. This approach is easier and cleaner than trying to maintain a consistent test environment for multiple test branches on a virtual machine or bare-metal server, where changes to the server could create inconsistency within your test environment.

How Containers Complicate Software Testing

While containers make software testing easier in many respects, they also create new challenges.

1. Containers are not hardware-agnostic

An important nuance to understand about containers is that, although containers allow you to create portable environments that abstract an application away from the underlying host server environment, containers (unlike virtual machines) are not entirely hardware-agnostic. Thus, the behavior of a containerized application could be affected in significant ways by hardware variables. For example, a containerized application could behave in different ways depending on which type of GPU its host server contains, if the application interacts with the GPU in some way. (Although containers traditionally were not used to host applications with graphical frontends, that’s no longer the case.) Containers in this respect don’t make software testing any harder than it would be if you were testing a non-containerized application on bare metal. But they can make it harder than testing software that is designed to be deployed in a virtual machine, where the underlying server hardware typically won’t impact application behavior. To address this challenge, QA teams need to make sure that they test containerized apps on all of the hardware profiles that might be used in production.

2. Docker versioning differences

In theory, Docker containers allow you to build an application once and run it anywhere that has Docker installed. In reality, there are different versions of Docker, and those differences can impact application behavior and performance. By and large, there is a lot of backward- and forward-compatibility between different versions of Docker. But there are subtle differences, and a container designed to run on one version of Docker could potentially experience problems on a different version due to feature deprecation or other problems. This means software testers need to know which version or versions of Docker will be used in production, and test against those versions. Ideally, they will also communicate with developers and admins to stay abreast of planned application and deployment changes so that they’ll know when to test on new versions of Docker.

3. Testing for microservices

If your software delivery team uses containers, there is a good chance that your application is written as a set of microservices. Although you don’t have to use a microservices architecture if you use Docker, containers make microservices easier to deploy--and supporting a microservices architecture is one major reason why organizations turn to containers in the first place. Microservices have lots of benefits, but they also have drawbacks--including the fact that they make software testing more complicated. Not only does a microservices architecture require that QA teams run separate tests for each individual microservice, but they also have to devise ways to test the application as a whole by running all of its microservices and ensuring that they interact as expected. All of this requires a more sophisticated approach to testing. You’ll almost certainly need to rely on automated tests to cover multiple microservices, and you’ll have to test things like APIs in addition to basic application code.

Conclusion

If you work in software testing, containers are both a bane and a benefit. In most respects, they make software delivery faster, more predictable and easier to manage. But they also make software testing more complex in certain key respects. The challenges they present are eminently solvable, but they require QA teams to adjust their processes and tools.