Fragmented test automation landscape slows adoption of continuous testing


Continuous testing is growing, but technology and team structures must improve for organisations to gain a competitive advantage, says a new report from on continuous testing from Capgemini and Sogeti, part of the Capgemini Group, in collaboration with Broadcom Inc.’s Enterprise Software Division.

According to the report, continuous testing is gaining ground in large enterprises, with almost a third of IT executives (32%) stating that their IT departments had ‘fully embraced continuous testing’.

However, with 58% of enterprises surveyed deploying a new build daily (and 26% at least hourly), the report highlights that companies must work to improve their continuous testing effectiveness by streamlining their test environment within an agile development ecosystem.

With 58% of enterprises surveyed deploying a new build daily (and 26% at least hourly), continuous testing is becoming a necessary part of the DevOps and agile development ecosystem

Continuous testing report

The report is based on 500 interviews with senior-level IT executives from large and medium-sized companies (over 1,000 employees) in industries including financial services, high tech, healthcare and life sciences, telecommunications, media and entertainment, and manufacturing.

Companies must unlock the potential of automation in testing:

  • The study strongly highlights that there is a significant scope to optimise continuous testing processes through technology. It found that automation was only being used to execute 24% of test cases, 24% of end-to-end business scenarios, and to generate 25% of the required test data
  • Greater use of automation could significantly improve the velocity of testing activities in the agile teams; for example, over a third (36%) of those surveyed said that more than 50% of testing time is spent searching, managing, maintaining and generating test data. The report recommends that to respond to customer and market needs, all enterprises need to take significant action
  • Use cases, detailed in the report, include a leading Australian bank, which had over 5,000 builds for more than 100 applications. By harnessing a single platform for automated build and deployment, integrated with testing and automation tools, it was able to reduce build cycle time by 40%, improve time to market and significantly increase environment uptime.

Enterprises need smarter orchestration and enablement for testing

The self-empowerment of autonomous teams resulted for many enterprises in an uncontrolled landscape with a broad diversity in QA and test automation approaches. To regain control, cites the report, organisations need to improve the central quality enablement of the agile teams with clearer QA guidelines and smarter QA technology provisioning.

A promising development is to make the test orchestration and execution much smarter via artificial intelligence (AI) technologies which provide ‘smart’ test orchestration. With the addition of machine learning capabilities, systems will be able to automatically determine the tests that are required in the release and production cycles.

The report showed a clear need for improved transparency and orchestration in agile testing. Among the executives surveyed, 35% identified a ‘complete audit trail of testing activities’ and a ‘consolidated test and release pipeline’ as the most important test orchestration capabilities, with 32% highlighting a need for a ‘single-place for cross-team collaboration’ and ‘continuous delivery pipeline visibility’.

The shortage of centralised enablement on technology provisioning becomes most apparent when investigating the challenges with test environments. Teams wasted too much time procuring complete test environments. Four in ten (40%) of the respondents said their teams spend more than half of their time building and maintaining their test environments.

Mark Buenen, global leader, digital assurance and testing for the Capgemini Group, said: “The next two to three years will be a critical time for continuous testing as organisations need to solve the dilemma of transitioning to self-empowered autonomous teams where quality is everyone’s responsibility, while improving the central quality support and central quality oversight to create an agile development environment, that is truly able to deliver value in an increasingly complex IT landscape.

“This report clearly demonstrates the need to expand the levels of test automation, make provisioning of test environments and test data smarter, implementing meaningful quality metrics and smart quality dashboards as the core focus areas.”

Inter-disciplinary teams require new skills and support

In the last three to five years, the roles of both developers and testers have evolved significantly. According to the report, developers are now much closer to the customer, with a prominent role in shaping the user experience, while testers have moved out of siloed teams to work in parallel with developers and business teams, meaning that they are involved much earlier in the development life cycle.

The roles and responsibilities of developers and testers are blurring, but clearly it remains important to have QA and test focused experts in the agile teams.

These inter-disciplinary teams mark a move forward, but also create challenges, details the report. They require every team member to have a holistic understanding of the entire process, with testers needing to upgrade their technical skills. Companies must address the requirement for up-skilling and a new, integrated approach to truly achieve continuous testing’s full potential.

Ashok Reddy, senior vice president and general manager, enterprise software division at Broadcom, said: “It’s difficult to fully capitalise on the benefits of agile and DevOps without continuous test automation, which enables organisations to leverage data and intelligence to accelerate the creation of high quality software at the pace of digital business.

“As the chasm between IT and business leaders continues to close, new DevOps practices and techniques, such as automating continuous testing processes, will begin to emerge as necessary tools to drive digital transformation in a hyper-competitive marketplace.”

The report can be downloaded here.