Getting started with visual regression testing and WebdriverIO

Power up your visual regression testing using Webdriver with these tips, tricks and tutorials from frontend developer, Arek!

Imagine your company created a new fancy website and now they want to extend it with more features.

You add a new shiny animated button to the home screen, check if it works locally and then deploy to production. But suddenly, your users start to report that buttons on other pages seem to be broken. What happened? You thought you had this “bulletproofed” following the unit tests...right?

Well, you probably forgot visual regression testing.

Developing modern apps and websites require a good code validation process. Integration and unit tests are a must, but often they do not cover all the changes that are introduced by developers. From a frontend perspective, creating a new component or making simple style adjustments may cause some visual glitches in the other parts of the application. Additionally, some users might work with older browsers like IE11, so catching all the bugs immediately is not obvious. 

 

What is Visual Regression Testing? 

Visual regression testing is the process of validating visual parts of an application or website to ensure that code modifications do not introduce any unintentional changes. It aims to prevent any user interface bugs when, for example, implementing new site functionalities or adding new elements. Of course, issues can be easily caught by manual testers (if you hire one) but to speed up the development process, it is always good to automate it!

Here at Cogworks we do visual regression testing for our static layouts hosted on Netlify and depending on the project’s specifics and browser requirements, we use various tools for automating our user interface validation process:

- WebdriverIO (https://webdriver.io) with wdio visual regression service when projects require IE11 support
- Percy (https://percy.io/)
- Cypress (https://www.cypress.io/) with Cypress Visual Regression module ( https://docs.cypress.io/guides/tooling/visual-testing)

WebdriverIO for visual regression testing.

Definitely one of the most useful tools to use is Webdriver.io. This framework allows cross-browser and mobile test automation with Webdriver Protocol as it has a lot of plugins and built-in services to run project-specific tests (as well as support for all modern frameworks like Angular, Vue or React!).

 

WebdriverIO tutorial (with Cogworks examples).

Let’s take a look at an example of WebdriverIO in action, so you can take a peek at just some of its capabilities. Our toolset consist of 3 elements:

- WebdriverIO
- WDIO image comparison service
- WDIO Devtools Service


In order to run the WebdriverIO setup you need to have Node, npm and npx installed (we assume this is already done). Then, in the command line go to the preferred directory of your project and run!

npm install @wdio/cli

This should install the WebdriverIO command-line interface. Now we are good to run the configuration generator. Let’s type:

npx wdio config

You will see a list of preferred options. Here you can choose a testing framework, specifically a compiler (we use Babel in our example project), base URL and any additional services you want to be installed like ChromeDriver (to run tests in Chrome) or DevTools.

After selecting all the options, the Webdriver.io framework will be installed. You can now see and modify all the options in the wdio.conf.js file that was generated in the working directory. 

Webdriver.io allows you to have multiple configuration files that can be inherited from the main config file, something that’s especially useful when you need to have different settings for eg. different environments (more details here). 

Additional configuration.

So far our configuration is the initial one and looks like this:

exports.config = {
    runner: 'local',
    specs: [
        './test/specs/**/*.js'
    ],
    maxInstances: 10,
    capabilities: [{
        maxInstances: 5,
        browserName: 'chrome',
        acceptInsecureCerts: true
    }],
    logLevel: 'info',
    bail: 0,
    baseUrl: 'http://wearecogworks.netlify.app',
    waitforTimeout: 10000,
    connectionRetryTimeout: 120000,
    connectionRetryCount: 3,
    services: ['chromedriver','devtools','image-service'],
    framework: 'mocha',
    reporters: ['spec'],
    mochaOpts: {   
        require: ['@babel/register'],
        ui: 'bdd',
        timeout: 60000
    },
}

This should be enough to run tests but you can add a few additional services that require individual configuration. 

First, set up the image comparison service. You can install it with:

npm install --save wdio-image-comparison-service

After a successful install, you’ll need to adjust the  wdio.conf.js file. At the top, import the Path module that will help set up files and directories in the config.

const { join } = require('path')

Now, modify the ‘services’ settings and provide the basic setup for the image comparison service. Here, point out the paths for the baseline images and screenshot folders. (Don’t worry if you don’t have them yet in the project, running tests for the first time should create those for you). You can also customize the screenshot image name and set additional things, like disabling CSS animations during tests or clearing the runtime folder before a test starts!

services: ['chromedriver', 'devtools',
    ['image-comparison',
      {
        baselineFolder: join(process.cwd(), './tests/visualRegressionBaseline/'),
        formatImageName: '{tag}-{logName}',
        screenshotPath: join(process.cwd(), './tests/visualRegressionDiff/'),
        autoSaveBaseline: true,
        blockOutStatusBar: true,
        blockOutToolBar: true,
        clearRuntimeFolder: true,
        disableCSSAnimation: true
      }]
  ],


The number of things that are customisable when using the image comparison service is pretty impressive (and should be enough to meet even the most demanding test scenarios, you can see the full list of options here). As you can see above, our service configuration includes ChromeDriver that will allow you to test pages in Chrome browser and DevTools to test pages in custom browser settings. 

Our testing site is hosted on Netlify and is password-protected, so to avoid reentering passwords before every test spec, we move our login function to the WebdriverIO config file. Using the beforeSuite method we simply create a scenario for login action. 

beforeSuite: function (suite) {
    let url = browser.options.baseUrl
    browser.url(url)
    $('input[type="password"]').setValue(process.env.NETLIFY_PASS)
    $('.button').click()
  }


We use an environmental variable password because our codebase is publicly available on GitHub. For the ‘process.env’ variable to work you need to install dotenv module then add a simple configuration at the top of the wdio.conf.js file to access password value directly from our environment file.

const dotenv = require('dotenv')
dotenv.config({ path: process.cwd() + '/.env.prod/' })


And that’s it. Let’s take a look at how the complete WebdriverIO configuration file looks: 

const { join } = require('path')
const dotenv = require('dotenv')
dotenv.config({ path: process.cwd() + '/.env.prod/' })

exports.config = {
  runner: 'local',
  specs: [
    './tests/specs/**/*.js',
    './tests/specs/*.js'
  ],
  maxInstances: 10,
  automationProtocol: 'devtools',
  capabilities: [
    {
      maxInstances: 3,
      browserName: 'chrome',
      acceptInsecureCerts: true
    }
  ],
  logLevel: 'info',
  bail: 0,
  baseUrl: 'https://wearecogworks.netlify.app',
  waitforTimeout: 10000,
  connectionRetryTimeout: 120000,
  connectionRetryCount: 3,
  services: ['chromedriver', 'devtools',
    ['image-comparison',
      {
        baselineFolder: join(process.cwd(), './tests/visualRegressionBaseline/'),
        formatImageName: '{tag}-{logName}',
        screenshotPath: join(process.cwd(), './tests/visualRegressionDiff/'),
        autoSaveBaseline: true,
        blockOutStatusBar: true,
        blockOutToolBar: true,
        clearRuntimeFolder: true,
        disableCSSAnimation: true
      }]
  ],
  framework: 'mocha',
  reporters: ['spec'],
  mochaOpts: {
    require: ['@babel/register'],
    ui: 'bdd',
    timeout: 60000
  },


  beforeSuite: function (suite) {
    let url = browser.options.baseUrl
    browser.url(url)
    $('input[type="password"]').setValue(process.env.NETLIFY_PASS)
    $('.button').click()
  }
}


Writing tests.

We got our configuration ready so we can start writing the first visual regression test.

There are multiple approaches to how you can organise testing suites, so feel free to do it your way. We just wanted to quickly show you how the test can be run. In our test or spec directory, we have created, for example, a homepage.js file to check the main page of our site. You can write a testing scenario that looks like this:

describe('Test Homepage Desktop', () => {
before(() => {
    browser.emulateDevice({
      viewport: {
        width: 1440,
        height: 900
      },
      userAgent: 'Mozilla/5.0'
    })


    browser.url('/homepage.html')

    // Dismiss cookie banner
    const elem = $('.cc-dismiss')
    elem.waitForExist({ timeout: 3000 })
    if (elem.isDisplayed()) {
      elem.click()
    }
  })


  it('should save desktop page screenshot', () => {
  browser.saveFullPageScreen('homepage-desktop', {
    hideAfterFirstScroll: [$('.m-main-header')],
    fullPageScrollTimeout: 1500,
    disableCSSAnimation: true
    })
  })


  it('should compare successful with a baseline', () => {
    expect(browser.checkFullPageScreen('homepage-desktop', {
      hideAfterFirstScroll: [$('.m-main-header')],
      fullPageScrollTimeout: 1500,
      disableCSSAnimation: true
    })).toEqual(0)
  })
})


Prior to any screenshot actions taken by the browser, there is a ‘before’ function that specifies some initial steps like setting viewport size, cookie banner removal etc. In our scenario, we used the DevTools service to emulate devices and set the viewport width and height.

DevTools is super useful when you want to test responsive designs as the plugin has predefined device specs so you can simply write, for example, “browser.emulateDevice('iPhone X')”. The full list of supported mobile and desktop items is available here.

After setting the viewport sizes and user agent for the browser we provide the desired URL to load. As you can see in the image above we do not want the cookie banner to be visible (it can be tested separately) so there are some quick: find, click and dismiss actions specified. 

Everything is ready so it’s time to specify a screenshot action. We use the saveFullPageScreen method with custom options to grab the whole page view and because the header is fixed and the page is quite long, apply the hideAfterFirstScroll setting to hide it, adjust the scroll timeout and remove CSS animations that may impact the screenshot generating process.

browser.saveFullPageScreen('homepage-desktop', {
    hideAfterFirstScroll: [$('.m-main-header')],
    fullPageScrollTimeout: 1500,
    disableCSSAnimation: true
  })

Then we specify the compare function that will check the saved screenshot with our baseline image. If we run the tests for the first time and include autoSaveBaseline: true in our settings, the baseline images will be generated automatically.  Our compare part looks like this:

it('should compare successful with a baseline', () => {
    expect(browser.checkFullPageScreen('homepage-desktop', {
      hideAfterFirstScroll: [$('.m-main-header')],
      fullPageScrollTimeout: 1500,
      disableCSSAnimation: true
    })).toEqual(0)
})


As you can see we use the checkFullPageScreen method to compare images. By default, this method uses a mismatch percentage at 1.23 level, but this can be adjusted by adding an additional config part, like ‘misMatchPercentage: 2.34’. 

 

Ready, set, go

 With everything defined, you are good to go with starting a test. You can do it in a few different ways. The simple command that will trigger all the specs is:

npx wdio run ./wdio.conf.js

You can also run specific tests:

npx wdio run ./wdio.conf.js --spec homepage.js

After the test is done you should see new folders in the code structure. There’s a new visualRegressionBaseline directory that will hold the base images for future comparisons. Below you can see visualRegressionDiff with an ‘actual’ subfolder that contains images from the actual test run. If there is any difference to the baseline image it will be a part of diff subfolder. 

 


The diff image shows the differences between the current run and baseline so you can easily identify potential issues. In our case, there was a font issue on one of the sections so we quickly adjusted our code.

That's testing done. 

As you can see, WebdriverIO and image comparison service has a lot of options and possibilities so it is worth exploring the documentation to see how they can help design and run specific tests! One of the biggest advantages of WebdriverIO is that you can use things like IEDriver to test visuals in Internet Explorer; all you do is set up IEDriver and update the capabilities section in config!

I hope you’ve found something useful here, stay tuned for more tips like this from me on other testing tools! If you’re in the mood for more testing stuff, why not head over to our blog or dive into the Importance of Modern Web Performance testing.

Arek

  • Image for How to Automate Performance Testing in 5 Hours Build

    How to Automate Performance Testing in 5 Hours

  • Image for How to Test Accessibility With Axe in Cypress Build

    How to Test Accessibility With Axe in Cypress

  • Image for Integrating Document Management Systems With Azure and Umbraco Build

    Integrating Document Management Systems With Azure and Umbraco

  • Image for How to Use Google Lighthouse for Better Web Performance Build

    How to Use Google Lighthouse for Better Web Performance

Ready to collaborate ?

Get in touch to see how we can transform your digital presence.

Send us a message