Visual Regression Testing for Elements

Overview

In Autify, visual regression testing can be performed on elements. You can add the assertion command Element should look the same in a scenario, which compares the current screenshot of the target element and the reference image created from the past test run. Any kinds of elements on the screen, images or tables or divs, can be selected for the assertion as long as they can fit in the viewport.

When a test with the Element should look the same steps is run, the test will fail if the latest screenshot and the reference image did not match a pixel-by-pixel comparison. In such a case, review the result to see if the difference was caused by an expected change. If that was an expected change, click "Accept changes." This is all that is required to update the reference image that will be used for the next run!

Perform Visual Regression Testing for Elements

Record a scenario

Start recording from the New Scenario page or Scenario Details page. On Autify Recorder, click on the checkbox (☑️) and select the element on which you want to create an assertion. Then, select Element Should Look The Same from the assertion commands list.

When you save the scenario, you will see a new Element Should Look The Same step in your scenario. Note that the step has no thumbnail, but it will have a reference image after the first run.

Run the test twice

Once ready, run the scenario. During the first run, the Element Should Look The Same step is skipped with the following message as there is no reference image to compare with yet: This step is skipped on the first run. reference image is created, and future tests will compare the element with this reference image.

If you look at the step detail of the scenario, you can review the reference image created by the first run.

Next, run the scenario again to perform the assertion. This time, the current image will be compared with the reference image created during the previous successful test run at the Element Should Look The Same step!

Open the screenshot of Element Should Look The Same to see the side-by-side comparison. By clicking the "Diff" tab, you will be able to see the images of the element compared with each other.

Visual Regression Diff Threshold

The Visual Regression Diff Threshold feature allows users to control how much difference between the reference image and the current image is acceptable before a test is marked as failed. This can be found and configured in the step details of the scenario.

By setting a Diff threshold ranging from 0 to 100, users can specify the maximum percentage of difference allowed for the test to pass.

Example

If a threshold of 1% is set, the test will pass as long as the difference (Diff score) between the images is 1% or less. If the Diff score exceeds the threshold, the test will fail, indicating a more significant visual change.

Success

Fail

This feature is especially useful for reducing false positives caused by minor, irrelevant changes in visual appearance. It provides greater flexibility in determining what is acceptable in a visual regression test.

Accept changes

The Element Should Look The Same step passes when the current screenshot either exactly matches the reference image or falls within the set Diff threshold. Otherwise, the step fails. The following screenshot shows the failure of a Element Should Look The Same step.

However, there will be times when you will want to accept the change in appearance and use this result as the reference image in the future. That's why we have an "Accept changes" button.

When you click this button, the reference image for this Element Should Look The Same step will be updated, although it won't affect the status of the test result itself.

The next time you run the test, the new reference image will be used for the comparison. It is easy to change the expected appearance of the element.

Notes on use

Make assertions against text

You can use this feature to make assertions against text, but Autify may detect minute differences that occur due to font rendering, causing the assertion step to fail.
Therefore, if you wish to make assertions against text, we recommend that you use text assertion such as the "Element Text Should Be" command.

Make assertions against colors

While you can use this feature to make assertions against colors, Autify might miss changes in colors depending on the type of color change, resulting in a successful step even when there is a change. We recommend using JS steps for color assertions.

• Confirm an element has style

We hope you enjoy using our visual regression testing feature, and look forward to your feedback!