The answer, perhaps obvious to some, is the performance of software tests – and yet it still seems to be a niche and poorly comprehended area within the testing industry. Load and performance tests provide a crucial role within our business and are one of the few areas where the benefits of testing can be tangible and very closely related to user experience and its impact against sales.

We are the UK market leader in holiday rentals with over 18,000 individual properties listed through our company and our websites provide 67% of our transactional revenue. Therefore, the online experience we offer our customers is crucial to our business and we want to make sure that it’s stable, responsive and performant at all times.

Our strategy for load testing is geared very much around patterns of business activity, or substantial project change, such as the introduction of a new website or significant architectural restructure. We have two major PBAs each year, around the New Year period and school summer holidays, when most of our customers are looking to take a break away or grab a great deal on their next holiday. Load tests are run against our production environment around eight weeks prior to the expected peak period and are invaluable for identifying areas for improvement, before hitting that crucial trading period.

The importance of analysis

Data analytics and business analysis play a critical part in the planning of a load test. Total sessions and user volumes from previous peak periods are used to derive load volumes, alongside business expectations around sales growth. Typically, we will run load at peak volumes, and at peak plus a 10%-20% tolerance for growth. Once the load tests are performed, a ramped stress test is conducted with a further 10-20% contingency. The tests themselves only form part of the load test activity as the analysis and interpretation of the results, using the information gathered from logs and our load test tools, then drives a substantial body of effort to determine what changes are required to fine-tune our production environments and to then put these changes into effect before peak.

Performance testing, for us, is a subtly different style of test; we are driving traffic at smaller volumes of virtual users, compared to the load test, as we look to identify issues with speed of response, page rendering/loading, resilience, and performance errors. Performance testing is an entirely in-house activity whereas we utilise consultants to assist with our load testing, and we are much more likely to run our performance tests in a non-production environment.

With DevOps, my strategy for performance testing is now to include performance tests as part of our CD/CI models for development so that performance forms part of our continuous automation pattern for build and release. Performance tests should check performance metrics on each deploy to ensure no performance degradation has been encountered. It is important that this should be a completely automated process with very little maintenance required in order to get the most benefit.

As well as performance testing, ongoing performance monitoring is an important aspect of keeping check of your company’s performance requirements. My team use a range of tools to provide 24/7 monitoring of performance across our infrastructure, both within production and our non-production environments.

The benefits of cloud flexibility

The transition to cloud technologies has opened up some interesting new avenues for performance and load testing with a cloud-hosted production environment versus physical hardware giving a range of different options for how we support our websites and tailor our infrastructure to meet our expected needs. Cloud technologies also allow for a greater tool choice and flexibility to support our business that can provide the required volume of concurrent users to meet our test demands.

There are also environmental considerations; as how you choose to host your production and non-production environments, and any variations between them, can have consequences for how you conduct your tests and the validity of your results.

I have had to undertake a significant investment programme at Vacation Rentals UK in order to provide for a software development lifecycle in our non-production infrastructure that is closer to mimicking production and therefore provides a more accurate reflection of how our future developments may impact performance when live.

One of the biggest challenges I have found, however, is the shortage of individuals with the required skill sets in the testing market. As there only seem to be a small pool of talented specialists in this area, they have influence over the hiring market and it has proven difficult to attract talent, particularly in more regional locations. For my purposes, I have chosen a model that is a mix of in-house self-generated capability and consultants or third-party vendors. By developing our own in-house capability, it gives us the ability to make performance testing an embedded part of the build and release activity. Performance testing can be scheduled and tailored to meet the needs of each squad or set of project deliverables. It gives our teams’ independence and accountability over their performance and load testing needs.

The introduction of new functionality into our infrastructure can be a difficult thing to then interpret into non-functional needs and requirements – but having some in-house knowledge and understanding of performance testing can help derive more meaningful performance goals. This has led to an interesting development for my team as they need the technical know-how to structure, run and interpret load and performance tests, but they also need the skills to assist non-technical members of our organisation with teasing out their business needs so these can be translated into suitable non-functional requirements.

As more organisations strive for a ‘digital first’ mentality, I believe the importance of load and performance testing will become ever more critical to a company’s success. As in agile methodology, where quality is now being widely recognised as everyone’s responsibility, so too should everyone be seen as accountable for performance and how this directly relates to the success of their organisation.

Written by QA Manager at Vacation Rentals UK, Felicity Lord