Test Execution and Result

What is the retention period for a test result video log?

Video logs of test results are available for a uniform period of 30 days. After 30 days from test completion the video log will be unavailable. The standard test results will still be available as usual after the 30 day period.

Written by Rieko Araki , Nicholas Carter

Updated on : 25 Nov 2022

What test execution environments support Video Logs?

Video logs are available for all execution environments except for Edge/Windows Server. Video logs for the Edge/Windows Server environment are currently in closed beta. Please contact us if you'd like to join the closed beta.

+ 2

Written by Rieko Araki , Yuka Saito , Meiko Hori and 2 others

Updated on : 25 Jan 2023

Are there any times or days when Autify is not available?

We do not perform scheduled maintenance that requires us to shut down the system at certain times or days. We may perform maintenance once every few months, but it has taken between 30 minutes to 2 hours in the past. We try to notify you in advanc...

Written by Autify Customer success team , Rieko Araki

Updated on : 30 Mar 2022

Can I use a fixed IP address on Autify?

Yes, we have an option for fixed IP addresses for customers who would like to keep the IP addresses from which Autify runs tests the same. This is useful when you use an IP allowlist to accept only requests from specific IP addresses. If your cont...

+ 1

Written by Autify Customer success team , Rieko Araki , Ryota Nishimura and 1 others

Updated on : 03 Nov 2022

How does Autify's AI follow elements within a page?

In addition to general features such as class and id, Autify's AI evaluates each element using the wording, position in the DOM tree, and parent-child relationship of the element and follows the element it determined to be the best match. Therefor...

+ 1

Written by Autify Customer success team , Rieko Araki , Autify Member and 1 others

Updated on : 15 Aug 2022

Do you support BASIC authentication?

Yes, you can set up Basic authentication on a per-domain basis. 1. How to set up Basic authentication Click [Settings] and choose the workspace you want to set up Basic authentication for . "Basic auth credentials" will appear at the bottom...

+ 2

Written by Autify Customer success team , Rieko Araki , Sho Fujita and 2 others

Updated on : 17 Oct 2022

Can I only run tests on Linux/Chrome? I want to run tests on other OS and browsers.

The test execution environment depends on your contract. Please check the pricing page for various contracts. For more information about supported operating systems and browsers, please see the following documents: Supported OS and browser ...

Written by Autify Customer success team , Rieko Araki

Updated on : 25 Apr 2022

What is the [Autify cloud environment]?

[Autify cloud environment] is a test execution environment where the browser is launched on the operating system in the cloud environment prepared by Autify. You can select this from the [Test Plan] execution environment. The [Autify ...

+ 1

Written by Autify Customer success team , Rieko Araki , Yuka Saito and 1 others

Updated on : 21 Nov 2022

What is a [Test Plan] (formerly [Schedule])

A [Test plan] is a mechanism for efficiently executing tests by grouping multiple [Scenarios] . This feature used to called [Schedule] . What can be achieved by a [Test Plan] With the [Test Plan] , the following are possible: ...

Written by Autify Customer success team , Rieko Araki , Yoko Rizzo

Updated on : 20 May 2022

Can I Specify the Execution Order of [Scenarios]?

Yes! When you specify the [Scenario] you wish to run on the [Test Plan] page, you can specify the execution order by selecting [Sequential] as the execution method. See the other document for more details.

+ 1

Written by Autify Customer success team , Rieko Araki , Yuka Saito and 1 others

Updated on : 28 Dec 2022

Why is it slower than when I run it in Selenium?

Autify takes a screenshot of the screen at each step, so it tends to take longer than Selenium. Also, Autify has a function that the AI searches for a substitute element when an element with exact match is not found. The process is as follows: ...

Written by Autify Customer success team , Rieko Araki

Updated on : 30 Mar 2022

The screenshots of the [Results] seem to succeed but fail. Why is this?

There is a slight time lag between running the test and taking the screenshot. Therefore, even though the test was in a failure state at the time of execution, it will be in a success state after that, and the screenshot may be taken in a success ...

Written by Autify Customer success team , Rieko Araki

Updated on : 30 Mar 2022

It’s tedious to click on each step when reviewing each screenshot in the Test Results section. Is there a better way?

Although you can move between steps in Test Results by clicking the icons, you can also use the arrow keys on your keyboard. Left arrow key : Move to the screenshot of the step immediately after the current one. Right arrow key : Move t...

Written by Autify Customer success team , Rieko Araki

Updated on : 30 Mar 2022

What does [Internal error] mean? Why was the test Pending?

Tests are marked as [Internal error] in the following cases: When an internal error occurs within Autify while a test is being executed. When the test exceeds the maximum execution time for some reason. In most cases, an internal err...

Written by Autify Customer success team , Rieko Araki , Yuka Saito

Updated on : 04 Apr 2022

Why do I get a [Review Needed] message at the same step every time even though I click [Save as Passed]?

We use the term [Review Needed] when there is no exact match between the id and text. (Only an exact match will result in a Pass.) Elements tend to result in [Review Needed] if the id or text changes frequently. We plan to enhance the AI...

Written by Autify Customer success team , Rieko Araki

Updated on : 30 Mar 2022

What does [Review Needed] mean?

[Review Needed] is a state where Autify AI cannot find the element that exactly matches the element to be operated or verified and has found an alternate element. On Autify, the AI will attempt to find the target element when it can't find an ...

+ 1

Written by Autify Customer success team , Rieko Araki , Autify Member and 1 others

Updated on : 16 Jan 2023

How long are test results saved?

The period during which the [test results] can be viewed depends on your contract. 1 month 3 months 6 months Etc... It is also possible to extend the viewing period, such as 1 or 2 years. If you need to extend, please contact suppo...

Written by Autify Customer success team , Rieko Araki

Updated on : 30 Mar 2022

Screenshots and icons are not displayed correctly

If the Autify site is: Not displaying screenshots of [Scenarios] or [Results] Not displaying the icon indicating the execution status The layout of the entire site is broken it may be due to the release that is done daily. Most o...

Written by Autify Customer success team , Rieko Araki

Updated on : 30 Mar 2022

It seems that when the test is executed for the first time, it takes longer than the second time. Why is this?

The behavior of Autify is that the test process is not changed between the first execution and the second and subsequent executions. If the characteristics of the elements (position, color, text, id, class, etc) change and is not an exact match, i...

Written by Autify Customer success team , Rieko Araki

Updated on : 30 Mar 2022

After running a test, can I check the content of emails received using the Email Test feature?

How to check the contents of emails received using the Email Testing feature depends on the type of email address you are using. What you can check when using a random and permanent email address You can check the content of the email by openi...

+ 2

Written by Autify Customer success team , Rieko Araki , Autify Member and 2 others

Updated on : 07 Sep 2022

Why is the test result screenshot different from when I recorded the test or the last successful test?

There is a short time lag between the execution of each test step and capturing the screenshot. Because of this, the screenshot may be different, such as when an element disappears immediately after an action. For example, a displayed element, suc...

Written by Rieko Araki

Updated on : 04 Aug 2022

Is parallel execution possible?

Yes! You can execute in parallel. See the other document for more details.

+ 1

Written by Autify Customer success team , Rieko Araki , Yuka Saito and 1 others

Updated on : 28 Dec 2022

Can Autify take a screenshot of the entire screen?

Because Autify captures screenshots with the target element in the center, it does not support taking screenshots of the entire page. That being said, you can view a full-screen screenshot of failed steps, which you can use to determine why the st...

+ 1

Written by Autify Customer success team , Rieko Araki , Ryota Nishimura and 1 others

Updated on : 19 Dec 2022

Is it possible to automatically run tests on a regular basis?

Yes! Set the periodic execution from the [Test Plan] details screen. During the creation/correction of [Scenarios] , it is recommended to use Autify Cloud environment that has a high execution speed. Also in the external device farm e...

Written by Autify Customer success team , Rieko Araki , Yuka Saito

Updated on : 26 Jul 2022

Is there a function to notify the result?

Autify supports Slack integration, and if you set up the integration, you will be able to receive notifications on Slack about tests as follows: Click here for details on how to set up. Other options currently available are API and Webh...

Written by Autify Customer success team , Rieko Araki

Updated on : 31 Mar 2022

Is it possible to ignore the failure of a particular step?

Yes, it is possible! You can select how to handle the failure at each step from the following three patterns. If the test fails, Abort Mark this step as failed and continue Ignore and continue In 2, the test continues, but the fact th...

Written by Autify Customer success team , Rieko Araki

Updated on : 09 Apr 2022

Can I cancel tests that are running?

On Autify, you cannot cancel or stop a test while it's running. The maximum run time for a test is 2.5 hours, so if the run time exceeds this limit, the test will automatically stop. If the test hasn't been executed yet, you can cancel it by clic...

Written by Ryota Nishimura , Autify Member

Updated on : 04 Aug 2022

Does the history (URL, password, etc. of the test execution) remain on the terminal in the test environment?

The environment is rebuilt every time the test is executed, so information such as URL and password is not retained in the environment.Please be assured that the environment is not recycled after each test run in any environment.This is one of the...

Written by Yuka Saito

Updated on : 01 Jun 2022

What does "Diff" mean in test results?

[Diff] in test results shows how much the page’s appearance has changed. It is calculated based on the change in the pixel and color on the screen. Any differences between the previous and current test screenshots are highlighted.

Written by Yuka Saito , Meiko Hori , Autify Member

Updated on : 04 Aug 2022

Diff isn't displayed in the test result. Why is this?

For each step in the test result, the [Diff] tab will appear if a screenshot of the last successful test exists, allowing you to see the difference highlighted between the screenshots of the last and current tests. Therefore, you can only see ...

Written by Yuka Saito , Meiko Hori , Autify Member

Updated on : 04 Aug 2022

Can I delete test results?

Currently, you can't delete test results on Autify's interface. Although you cannot delete test results, once the test result's viewing period is over, you will no longer be able to view it (the length of the viewing period depends on your contract...

Written by Ryota Nishimura , Autify Member

Updated on : 09 Dec 2022

Can I use Autify Connect to test localhost websites?

Yes, you can use Autify Connect to test websites running on localhost. However, it will not work correctly if the target URL of the scenario links to a local host or local network, such as http://localhost and 127.0.0.1 . Therefore, please use ...

+ 3

Written by Autify Customer success team , Hiroki Iizuka , Sho Fujita and 3 others

Updated on : 15 Jan 2023

What should I do if an element is not found or if an incorrect element is selected?

Sometimes, Autify may click on an element different from the one you recorded or be unable to find the correct element, causing the test to fail. This page describes what to do when Autify cannot find the target element or selects the wrong element...

+ 2

Written by Autify Customer success team , Ryota Nishimura , Autify Member and 2 others

Updated on : 15 Jan 2023

Why do some test runs take so long?

For various reasons, it can take a long time for Autify to run tests. In this article, you will find two common examples. A step takes longer than 30 seconds to complete Autify's AI waits a certain period of time (30 seconds by default) until a...

Written by Ryota Nishimura , Autify Member

Updated on : 07 Feb 2023

Why do I get two Slack notifications?

When a test is complete, Autify can send you a Slack notification containing the test results. Slack notifications can be set up for the whole Workspace and for each Test Plan. However, if you set them up so that notifications are sent to the same...

Written by Ryota Nishimura , Autify Member

Updated on : 31 Jan 2023

How Many Test Results will there be from my Test Plan?

How Many Test Results will there be from my Test Plan? The number of tests executed per test plan is the number of execution environments times the number of scenarios. If you have used data from a CSV in one of your scenario's, that scenario wi...

Written by sam ostrom , Yuka Saito

Updated on : 14 Nov 2022

Why does the Chrome address bar show a "Not Secure" warning in the video log?

When you view the video log, you may see the following warning in the Chrome address bar: This warning appears because of the way Basic Authentication and custom hosts are set up. Even if this warning appears in the address bar, requests/resp...

Written by Autify Customer success team , Autify Member , Mami Kono

Updated on : 13 Mar 2023

Can I run tests with Autify Connect to run tests using a Proxy server?

Even if you are using a proxy server, you can use Autify Connect to run tests. Please read through this page and use the appropriate option for your network configuration. 1. When using a proxy server for the traffic between the Autify Connect Cl...

Written by Autify Member , Mami Kono

Updated on : 14 Feb 2023

What are the numbers shown next to "Logs " in the step of test result?

※ English in this website is translated using machine translation. Please note that the content may not be 100% accurate. The number shown next to "Logs " was the number of browser logs output when that step is executed. *The number of ...

Written by Mami Kono

Updated on : 13 Mar 2023