The journey started a few years ago within our large organisation in charge of building a world-class product largely used by professional users of the financial markets. To accelerate our business growth and delight our customers, the business model has moved to have one unique platform-based product. This model aims at reducing costs and improving the time to market by getting fast and short release cycles with much more features.

To support this change with the quality assurance (QA) group, many challenges were coming up quickly: e.g. for a large-scale system which is evolving fast, how to keep a very high quality product, how to maintain very high performance and stability of the platform, how to remove the testing bottlenecks, how to test as early as possible and quickly catch bugs impacting our customers, etc.

The previous traditional QA model (e.g. independent QA team, Iterative waterfall cycle, main regression was manual, test management, etc.) became outdated and didn’t satisfy the new challenges. We decided to change our QA model by getting rid of the central group and putting in place agile approaches.

Methods of work and progress

Below are 4 main principles we focused on by leading the related changes in terms of organisation, methods of work and processes:

• The quality is first in lines of code: We moved the quality ownership to development.
• Testing and coding at the same time: We moved testers onto same “Delivery” teams as developers and product owners. The testers sit in the middle of the developers.
• Regression tests are automated: Few manual and much more automation testing, we reduced the manual testers and hired more automation engineers and implemented the continuous integration testing.
• Lead the transformation by the metrics: Use metrics to bring transparency and drive efficiency: Define clear quality goals to share across the delivery organisation and provide the related metrics in an accessible and transparent way.

Different challenges appeared, some are related to the method of work and process (e.g. how to ensure the right test coverage without test management, how to leverage the automation testing, how to put in place the continuous integration and deployment, etc.) and some are organisational (how to drive the developer quality mindset, how to assign quality accountability and keep the transparency, how to manage the transformation of traditional QA roles in an agile world, etc.). But one of the most important and exciting items was with introducing the culture of metrics and leading the transformation by the metrics. This paper gives insights into how we managed to build smart quality metrics.

Quality assessment

Indeed, talking about the quality metrics in general, not specific to the Software Engineers, it is still tricky, questionable, contentious and even sometimes a boring topic despite its importance and its impact. This is not because people lack interest or they don’t commit on the objectives behind the metrics, but often the metrics implementation is not so SMART (i.e. no purpose, not actionable, not meaningful, not intuitive, or even counterproductive). How many times we hear such feedback as “building metrics, just for having metrics”, “nice metrics but I don’t need them, or they can’t help me”, etc. Unfortunately, these statements might be true, if metrics are not SMART. On top of that, and beyond the implementation challenges, measuring the Software Quality is one of the most heated and fascinating debates in the software industry. There is no single way to measure an absolute quality for software. Indeed, there are different angles, meaning and drivers to assess the quality.

Often, many assumptions and interpretations are made to explain standard metrics. Currently, with the digitalisation era; cloud and open platforms, IoT, etc. It is easier to bring more transparency as the data is recorded, structured and consumed. Indeed, this pushes the software engineers to design their products with more checkpoints, counters, usage tracking, performance monitoring, etc.

Building smart quality metrics

This is offering new ways to build and consume effective Metrics and significantly empowering the quality analytics approach. Below are 5 tips to help us building smart quality metrics:

• Introducing the culture of metrics: Accept metrics to measure our performance, take benefits of the digitalisation and consume our own data to bring transparency and improve a process, etc.
• Defining our own metrics: The new data sources are a huge opportunity to define and tailor our own metrics beyond the standards.
• Building trustworthy metrics: Make metrics easy to explore, to access, to drill down, to get access to the underlying data, etc. For instance, from a defect chart summary report, navigate to the defect and open the details.
• Visualising self-explanatory metrics: Make metrics based on data that easy to collect, easy to generate, to visualise, informative, self-explanatory, meaningful for both executives and team members
• Building predictive metrics: Use Big Data approaches to build metrics, use Data Mining tools, build risk analysis tools to get predictive metrics.

“If you can not measure it, you can not improve it”. The power of metrics in improving and leading transformation was a key. The smart way we did produce metrics, was the main driver to get people to trust and consume them.

Written by Hussein Jibaou, Head of QA Office at Thomson Reuters Eikon