Is it time to take a look at automated visual testing?

The impact and adoption of automated visual testing has rocketed. With so many large companies and progressive engineering teams using it – perhaps you should take a look too?

Visual testing – in this article I’ll explain what it is, why it adds value to your delivery pipeline and some tooling options you can explore yourself.

We’ve all been there – where a seemingly innocuous CSS change passed all the way through your delivery pipeline, passing tests before rendering your site unusable due to layout or similar issues on a particular browser.

This hurts, but you’ve bought into continuous delivery and it’s impossible to catch those sort of issues automatically, right?

Well, maybe not…

I remember sitting an interview for what was my first job in software and they asked about automated testing and what areas were hard to automate. I didn’t have any experience at all with it but the obvious example was the ‘look and feel’ of a website. It’s possible for our automated tests to still be ‘functionally’ correct and working even though elements may be overlapping or in totally wrong locations.

Visual testing attempts to fix that by allowing you to make assertions about how an application or website is rendered.

Some applications now lay claim to AI powered visual testing, but at its core, and for most commonly used offerings, it’s a much simpler strategy – one of screenshot comparison. Using a browser, or commonly these days a headless browser, an image is taken or stitched of the viewport and then, using computer vision techniques, can be compared against a baseline. Some tools offer differing levels of matching – be that pixel perfect or less granular.

So, armed with a collection of screenshots that represent the desired look and feel of your application (and those can also differ by device/browser etc.), you then use visual testing to ensure that your application hasn’t diverged from that in an automated way.

This can be built right into your existing delivery pipeline and sit beside your existing functional tests. In fact with a lot of these tools it can be interwoven into your existing functional tools and the visual test takes the place of a test assertion. The image above shows the workflow and type of output you can expect when using visual testing.

This example, from github.com/mjhea0/cypress-visual-regression, shows the incredible power of having it live with your existing tests.

So, you know you can use this technique in your current workflow; you know a little bit about how it works, but why use it?

Your existing tests may thank you!

A common pattern you may find in a lot of tests is something like:

cy.get(‘.action-div’).dblclick().should(‘not. be.visible’); cy.get(‘.action-input-hidden’).should(‘be. visible’);

And what you’re really doing there is trying to codify visual layout confirmation through a sequence of individual assertions. It’s painful when the screen is complex and can be pretty brittle when selectors change etc.

Often times when you’re doing this it could be replaced with a visual test, which will not only ensure the right elements are visible but also their placement and styling.

Automate cross-browser testing

Some tools integrate nicely with providers like Saucelabs and Browserstack and allow multiple baselines allowing you to quickly visually test your application across the entire board.

This is a huge time-sink for your engineers who are doing this testing and automating this opens them up to do higher level work.

Catch visual regressions

Small visual details are often thought of as the best-fit for a manual tester to find – but let’s be honest – we all suffer from fatigue or the effects of muscle memory when working on an application we’re overly used to, so small visual differences can pass your tests by.

Automated visual testing will flag them by either failing the test or asking for your approval. It adds another comfort level to continuous delivery.

Makes application visual identity clear

These baselines aren’t only useful to run tests, but also they give a clear indication of the expected look and feel of the application.

Visual testing is still a developing and pretty dynamic field and so a lot of things are still a bit open, vague or divergent in terms of enterprise workflows.

Some of those drawbacks you should be aware of include:

  • Dynamic content: having dates, image carousels or other dynamic regions can make this hard. Some tools allow you to define/blackout or ignore regions and others suggest using static test data
  • Speed: if the assertion is happening in the cloud it can add a fair amount of time to your tests, so use it wisely. Screen-shotting after every focus and interaction probably isn’t the best usage if execution time is something you’re focused on (and you probably should be)
  • Continuous delivery: when using a SaaS solution you can find that updated visual expectations need manual intervention to approve or deny. This puts manual gates and toggles in to your platform. Prioritise the services which offer API or programmatic interfaces to toggle assertions, or that allow the baselines to travel with the code
  • Animation support: our visual baselines identify static points of what pages should look like, but with animation becoming a central point to a lot of websites and interactions, there is the chance that discrepancies will be missed. Some of the applications handle this better than others but personally I think this is something that also impacts some design apps too
  • Usability: visual testing is mainly about checking layout and it’s based on assertions or expectations that you’ve set. This does nothing for usability testing and understanding if your website actually is enabling and helping the users through their journey.

So, if your interest is piqued and you’re interested in trying out visual testing the good news is this: it’s a well-trodden path now and there’s a number of exciting open source options for you to look at. Including but not limited to Wraith and BackStop.

I find that some of the test framework- facing OSS projects help speed up adoption, so with that in mind I can heavily endorse cypress-visual-regression for use with Cypress and nightwatch-vrt for Nightwatch.

If you’re in an enterprise though, the thought of spinning up infrastructure or dealing with open source may not be appealing in order to just kick the tyres on something.

With that in mind there are a few SaaS offerings you can check out:

  • Percy.io: with an impressive list of users including Google and Spotify, the newly announced free tier and abundant SDKS should make a POC of visual testing a reality for you. However, it only supports Firefox and Chrome and may offer some trickiness with dynamic content
  • Applitools: Having previously used this in a former job, I can say it’s probably the most mature SaaS offering in my opinion. Some interesting working on branching for assertions and integrations with cross-browser test offerings like Saucelabs. No Free Tier.

Another good place to look is screener.io, which features many more A-list companies, there’s also Internet Explorer support. No free tier.

The future of visual testing

So, what does the future potentially hold? Well Cypress have indicated some interest into rolling this functionality into their test framework. They already have rich screenshot and video support with test runs, so there is a chance this would be the biggest step to bring visual testing to the masses. No additional tools needed and what I’m expecting to be a very clean API.

So, in summary visual testing is finding some sweet spots in modern web development and I think it’s definitely worth trying out to see if it can improve either your current tests or current workflow.

Patrick Walker, senior software engineer, teamwork.com

More
articles

Menu