Test is like the Rodney Dangerfield of DevOps – it gets no respect says Joan Wrabets, CEO of Quali. But that is all about to change. In a recent Gartner survey, over 50% of respondents listed test automation as a top enabler of DevOps success. This is really illustrative of the recognition that DevOps cannot succeed if only one part of the DevOps continuum is automated.

If DevOps is a continuum, it starts with planning and development and ends with deployment into operations. Much of the focus in DevOps has been on application build automation, or development, and deployment of apps into operations. But for DevOps to pay off, there needs to be continuous automation from development all the way through to operations – and that means automating test.

DevOps projects tend to start in only one part of the continuum. When the driver for DevOps is in the test organisation, it is called continuous integration. However, it is also important to work to continuously expand automation across the continuum. When test teams take the lead, they need to find ways to push automation back into development and forward into staging and production. In fact, the problem of test automation is particularly difficult and this means that the team that implements test automation is in a great position to lead DevOps initiatives beyond test.

Continuous integration challenges

Why is automating the various testing steps (regression, performance, stress, GUI, QA, Interoperability and security testing) so difficult? There are five key components that must be automated successfully:

1. The infrastructure: It starts with one very important characteristic that is critical to testing which is not automated – the ability to automatically mimic the infrastructure in which the software (or hardware) being tested is expected to run in when it gets to production. Having the ability to give each tester (or set of automated tests) their own personal replica of the target production infrastructure – aka a ‘sandbox’ – is critical to ensuring that the software coming out of test can be automatically and continuously deployed into production. Automating tests is not nearly as difficult as automating the underlying infrastructure in which those tests run.
2. The production environment workload: Often not considered in automating testing is the requirement to mimic the workloads that would be common in production. This includes the network traffic, other application workloads, and network security profiles. These activities need to be automated within the sandbox to create a more realistic environment for testing.
3. The tests themselves: Of course, the tests themselves also need to be automated and there are a variety of tools to assist in this process. As software gets more complex, this continues to be a challenge, with mobile testing replacing GUI testing as the most challenging type of testing to automate.
4. Reporting and result analysis: In order to provide continuous automation from development through testing to delivery, it is important to see and analyse test results automatically.
5. Tool integration: Finally, automation into and out of testing needs to be enabled. This means that sandboxes and test automation suites need to be API-driven so that they can be started by a previous tool in the DevOps toolchain and can also initiate a tool that follows testing in the toolchain.

Sandbox is a key enabler for DevOps

Sandboxes are self contained infrastructure environments that can be configured to look exactly like the final target deployment environment, but can be created and run anywhere. For example, developers can create a sandbox that looks like the production environment – from network and hardware to OS versions and software to cloud APIs. They do their development in that sandbox for a short period of time and when they are done they tear down the sandbox.

Testers can do the same thing, and in addition, they can run a bunch of tests with the sandbox configured to look like their internal IT environment and then automatically re-configure the sandbox on the fly to look like the external cloud environment and run more tests. This allows them to test all of the possible environments that the application could run in without disrupting the actual production infrastructure.

Technically, what is a sandbox? Intuitively we know that a sandbox is a protected space where you have complete control and others are allowed in only if you invite them. You can bring your own toys into the sandbox and make anything you want in the sand. Just stomp it out if you don’t like it and start over. Technically, sandboxes follow these same rules. A number of vendors are now providing Sandbox solutions (some are called ‘Environment as a Service’) that have a simple interface for creating any target infrastructure environment and configuring it with as much control as you want. They allow you to bring applications, tools, tests and automated processes into that sandbox. They provide protections so that others cannot mess with any infrastructure that you are currently using in your sandbox. They also provide reservation and scheduling for many people so that whole teams of developers and/or testers can share physical and virtual infrastructure on-the-fly for hours, days or weeks at a time. Finally, a good Sandbox solution can be triggered from the outside (for example, from a DevOps tool).

In the world of hybrid clouds, applications need to be deployable on both on-premise infrastructure and on public cloud infrastructure. Sandboxes allow testers and developers to mimic both environments and define applications that can run successfully in both types of infrastructure.

Edited for web by Jordan Platt.