Firstly, let’s understand what is meant by continuous testing. The following definition from Wikipedia, is good enough for our purposes: ‘continuous testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release candidate as rapidly as possible’.

There are some key points that come out of this definition:

• Automated tests: firstly, whatever type of testing you may be looking to execute, you generally can’t deliver continuous testing in any meaningful way without using automation
• Feedback: secondly, the point of continuous testing is to provide regular feedback to the business on the impact of change as a result of Continuous Integration and continuous delivery
• Business risk: finally, increased velocity in release deployment increases the risk of introducing functional error, performance and capacity regression to a new deployment. Introducing CT at each stage of the software development life-cycle (SDLC) helps mitigate the risk of this occurring.

It would be true to say that continuous testing, continuous integration and continuous deployment are all part of DevOps, enabling the feedback loop that drives an informed, agile and robust approach to enterprise software development and deployment.

What is non-functional testing?

Non-functional testing refers to testing the way a system operates rather than specific system behaviour. Whilst non-functional testing can also encompass security and acceptance testing, the most common use case is testing software performance: reliability and scalability under load.

Let’s look in more detail at the key points from our definition of continuous testing and how they align with non-functional testing requirements.

Automated testing

Delivering non-functional testing requires automation and there is a wide range of tooling available, both licensed and open source. Historically this has applied mainly to system testing carried out late in the SDLC, leaving little time to correct any problems uncovered by testing.

Non-functional test automation now increasingly extends into Dev, where the move to modular or microservice-based architecture simplifies testing at the component level. This is often accomplished using open source tooling, such as JMeter (jmeter.apache.org) which has a powerful scripting engine and is easy to deploy.

Continuous non-functional testing is relatively simple to integrate into the Dev process, either ad-hoc or as part of the overnight build/test/deploy. This allows trending of component performance across builds and early detection and correction of coding changes that inadvertently cause performance regression.

The increasingly common use of highly scalable cloud-based test environments leveraging endpoint mock, stub or service virtualisation can allow developers to test component performance in sprint against production SLA’s, something unheard of even five years ago.

Feedback

Non-functional testing that is limited to system testing late in the SDLC is poorly placed to provide timely feedback to the Business on application performance. The feedback loop that is a key goal of implementing DevOps needs to function right across the SDLC to be truly effective.

Ideally continuous non-functional testing should be implemented in the following areas:

• development
• component and integrated component testing
• QA
• system testing
• production
• scheduled BAU testing.

Feedback data can take several forms but the most important consideration is visibility of application performance, good and bad, to relevant stakeholders. This means rapid feedback to Dev of performance-related bugs so they can be corrected and, if necessary, a new release prepared – including the dashboarding of test results demonstrating that an application is meeting or failing to meet the assigned performance goals.

Many organisations leverage full-stack monitoring solutions like ELK and Graphite / Grafana to visualise application behaviour. It is vitally important that performance data is included to inform stakeholder groups tasked with key responsibilities, such as approving Dev team work items and application release candidates.

Business risk

Mitigating business risk is all about ensuring that core applications remain available, functionally stable and performant to end-users regardless of load. Implementing continuous non-functional testing in the areas previously described helps to make this happen by maintaining and improving the quality of release.

The business impact of problems uncovered at each stage of testing can be quickly assessed, prioritised and corrected. This promotes early discovery of problems so the number and severity of non-functional software bugs naturally decrease and shifts left, benefiting the speed and quality of delivery.

It is equally important to understand the ongoing performance of core software applications post-deployment. Regularly scheduled non-functional testing of production systems can help detect creeping performance regressions, often the cumulative result of many small releases.

Scheduled BAU testing also provides confidence that production deployments are performance-ready in advance of peak events like Black Friday and Cyber Monday.

Written by Independant Consultant, Ian Molyneaux