Causes and solutions for failures in the Element should look the same step

Overview

In Autify's "Element should look the same" step (hereafter, the VRT step), image comparison is used to determine if an element's appearance on the screen matches its past state.
If this VRT step fails, there are several possible causes.

Main causes and solutions

1. Impact of Chrome version upgrades

Cause
When the Chrome browser version has changed compared to the previous execution environment and the test fails with the error “The target element had visual changes” the impact of the Chrome version update may be a factor.
The Chrome browser used in Autify's execution environment is regularly updated to the latest version.
This can cause minor visual changes that may affect the VRT's matching judgment, such as:

• Changes in the shape of scrollbars (e.g., from square to rounded).
• Slight shifts in element margins or alignment.
• Minor adjustments to system fonts or changes in the rendering algorithm.

💡 Example: A test passes in the morning but fails in the evening on the same day.
→ This could be because a Chrome version update caused a discrepancy in the VRT comparison result.

Solution

• Execute "Accept changes"
  If the difference in the VRT step is intentional or due to a specification change, please execute Accept changes for the relevant step.
  This action will update the baseline image, and subsequent tests will be judged based on the new appearance.
  

Chrome update frequency and points to note

• Chrome is typically updated once or twice a month.
• These updates can sometimes include slight changes to the UI rendering specifications, which may cause VRT steps to fail.
• Please note that not every update will cause an impact; it often depends on the specific CSS and structure of the elements.

2. The target element is outside the viewport

Cause
If the step fails with the error "The target element was not displayed," the element is likely not within the viewport.
This is highly probable if this error occurs even during the test's initial run when no baseline image has been captured, as it indicates Autify cannot correctly recognize the element because it's off-screen.

Solutions

• ① Specify and assert elements in smaller units.
  Instead of selecting a large area at once, break the element down and create multiple assertion steps. This helps ensure each smaller element fits within the viewport.
• ② Temporarily reduce the screen size to fit the target element into the viewport.
  Add the following JS step before the VRT step to temporarily zoom out, ensuring the entire target element fits within the viewport.

document.body.style.zoom = "60%";

Other recommended solutions

Separate VRT steps into a different scenario

For screens that undergo frequent visual changes, we recommend separating your VRT from your standard E2E tests. This allows you to manage functional checks and visual checks independently.

• E2E Test Scenario: Verifies functional flows and processes.
• VRT-Only Scenario: Focuses specifically on visual verification.

Adjust the threshold [Consultation required]

By adjusting the Visual regression diff threshold, you can sometimes ignore minor visual differences.
However, please be very cautious when adjusting the threshold, as setting it too high may cause you to overlook actual bugs.

Notes

What we currently do not support

• Controlling the image comparison target via locator specification (e.g., specifying only a portion of an image).
  → VRT compares the entire image and cannot be controlled by HTML attributes.
• Fixing the Chrome version for each execution environment.
  → Autify does not offer a feature to lock the browser version.

Internal Error Occurrences

As a known issue, tests may occasionally fail with an Internal Error due to temporary communication errors within Autify's internal system.
In most cases, this issue can be resolved by rerunning the test. If the failure persists or occurs frequently, please contact us.

If you need help

If you are unsure about the cause of a VRT step failure or how to resolve it, please contact our support team with the URL of your test result.
We will review your test result and provide you with the most appropriate solution.