Testing the expected & intelligent automation vs. exploring the unexpected: Shift Left & Shift Right practices
Are you worried about your organisation’s ability to cope with the complexity of delivering at high velocity, with excellent quality, in multi-speed IT landscape and hybrid environments?
DevOps – an art (combination of philosophies, methodologies & tools) of delivering products and services at a higher velocity as compared to traditional SDLC methodologies – now, I always like to add a statement that Velocity without quality is of little use – and that’s where the question of testing comes up.
There has always been a confusion (to say the least) about the role of testing or testers in the DevOps world.
While ‘continuous testing’ and ‘quality engineering’ are some phrases that have taken a front seat when it comes to DevOps – one always wonders where to draw the line. How to assess how much QA is actually needed, and in which phase, or is there really a phase?
Whilst Shift Left is what everyone is talking about, what does that actually entail? And then of course we have ‘testing the unexpected’, ‘testing in production’, ‘exploratory testing’ etc., – or Shift Right.
From my experience across multiple enterprises, as they go through the transformation journey from traditional SDLC to DevOps, and while we try to align practices to meet difficult demands from the complexities arising in a true multi-speed IT environment – I have built a perspective.
Shift Left is something that is simpler to implement. The IT community has put in lot of effort into it, in terms of setting up guidelines, tools and frameworks for early validation and verification. More unit and component level testing, early validation of services and techniques like service virtualisation for early integration testing are practiced.
However, it is more about ‘testing the expected’ and does very little for real-time feedback and exploring and base-lining the undefined and unknown. However, this still remains an integral part of the continuous testing framework.
I have read, learnt, experienced and implemented many paradigms of intelligent automation. But, if I have to define it simply, then the features that one should expect from an automation framework in the DevOps world are:
- Agility to Adapt to Multiple Tools: Intelligent automation frameworks will not force a specific tool on the developer and testing community – instead they would align with multiple tools used across the enterprise. Many times I am asked if multiple tools imply anti-standardisation? My answer to that is as far as all developers / SDETs / QA community are using the same framework, it does not matter which execution engine they use. For the enterprise there should be a standard automation framework across Unit / Component / Component Integration / System Testing with integrated reporting. That’s how we at Tech Mahindra have developed our Intelligent Automation Framework. Point to note here is that multiple tools does not imply unlimited tools – it is in fact a set of tools that are able and enough to cover the enterprise technology landscape from an automation perspective.
- Ability to overcome application changes: This is a vast subject in itself and there are multiple ways we have achieved this. While auto-healing scripts is a word many use, eventually I believe it is the framework design that leads to scripts that are easy to maintain and adaptable to application changes. Also headless automation (services layer automation) is the key to achieving this efficiency.
- Promotes early / progressive automation: That’s what Shift Left is all about. Automation framework plays a major role here. The key consideration is how much unit, component and component integration automation it can support.
- Helps in building regression through progression: Very detailed subject in itself but bottom line is that the in- sprint automation shall help in building the end to end test scripts as well.
- Supports hybrid infrastructure & varied testing personas: Question to ask here is that does the framework support write once and run in many environments (on premise/cloud, Web / Mobile, multi browser) with minimal configuration changes – if yes, it is indeed intelligent automation framework.
In a nutshell
Progressive / in-sprint automation and Shift-Left is the backbone of continuous testing that is mandatory for ‘validating the expected’ for successful DevOps and intelligent automation framework is what makes it possible. We have implemented different flavours of such intelligent automation frameworks leading to 20-25% savings on total cost of quality ownership for some large enterprises.
Shift Right in the context of services and APIs testing
There is a testing pattern in which before replacing an existing service with a newer one in production, we replicate and direct the real time production (with older version of service) request traffic to a staging environment (with newer version of service). Then results from new service are validated against production results.
This is defining the unknown and unexpected as to how the new service behaves with the production request data. This is a neat way of Shift Right testing – without breaking anything in production as such, and without spending too much money on creating huge amounts of test data to mimic all production scenarios.
Chaos Engineering
The idea behind this is fairly simple – can we avoid 100% of failures in production in complex distributed and hybrid environments – application issues, environment issues, service failures, configuration issues etc.? If not, then why not induce controlled failures and see how resilient the product or application is under the failure circumstances.
Typically organisations use this to experiment how a distributed system behaves in rough production conditions. Chaos Monkey is a system that was originally designed by Netflix to induce pseudo failures or sudden termination of services in the system architecture to “explore the unknown and undefined” system behaviour under failure scenarios. This helps to proactively identify issues and fix them; a very neat technique that we will see a lot more adoption of.
The last concept I will discuss about on Shift Right testing is the A/B testing – this is slightly different in the terms of that it’s not to test the application behaviour as much as it is to analyse the consumer patterns or behaviour. Basically two stable versions of the same application with some modifications (to analyse the patterns), are deployed in production and traffic is divided between the two versions.
An example of this is with one of our customer’s eCommerce sites – we applied this to see if one version of the site with a different way of displaying search results lead to more customer purchases.
There are a few more scenarios and use cases for Shift Right testing – however they are mostly similar concepts with varied techniques.
So where do we put the money for quality engineering in DevOps scenario – Shift Left or Shift Right testing? Where is the ROI?
- while Shift Right can get more real time feedback defect fixes when in production and late in game can be very costly
- at the same time just relying on early validation of expected results can lead to ignoring that unexpected that can cause major disruption at some point.
In my view, both Shift Left and Shift Right testing concepts are important and required – how much of each is what needs to be defined and implemented, as it varies for enterprises based on business scenarios, priorities, technology, landscape etc.
It’s been a great learning experience so far, while working on defining these strategies for some of our clients as they progress on their digital transformation road-maps and DevOps adoption. There is lot more to explore – and i’m looking forward to this journey of exploring the ‘unknown and undefined’!
Anjali Chhabra Nandwani is VP & Head, Digital Assurance Services Tech Mahindra Americas Inc.
