How to speed up test execution?
※ Please note that parts of this page are machine translated.
This page describes the mechanisms related to test execution time (the time required to start and finish a test).
Test Execution Time and Mechanisms for Finding Elements
In Autify, as described in this article, if an element that exactly matches the expected element (hereafter referred to as an exact match) from the recording or the last success case cannot be found then the AI searches for an alternative element. However there is an idle time where Autify waits for an exact match to appear and only after it has ran out should AI search starts.
Let us explain in details the mechanism of waiting for a certain period of time below.
As mentioned above, if the target element differs from the time of recording or the last successful test run, Autify will wait a certain amount of time. If a scenario is run for the first time and an exact match is not found at a certain step, it will wait 20 seconds (the default wait time*) for the element to appear. If the test is successful multiple times and it is determined that it is safe not to wait for the element to appear, the wait time for that step is gradually reduced.
Since JS Step for an element is also subject to AI, the execution time will be shortened under the same conditions.
Thank to this mechanism, initially, we carefully wait for an exact match or search for similar elements, but if the elements are found consistently, the test execution time is expected to gradually decrease as the waiting time decreases.
Basically, it is assumed that the execution time is gradually reduced in the element search mechanism. However, if elements are not properly searched or if a particular step continues to take time, the execution time may be reduced by the methods introduced next.
*The default waiting time can be specified for each workspace. (For details, see the section "2. Shorten the AI Waiting Time" below.)
Methods to shorten execution time and their disadvantages
Thanks to the mechanism described above, it is usually best to just let Autify gradually optimize execution time. However, there are methods to intentionally shorten the execution time if needed. Each of these methods have different effects that you need to consider before implementing so let us introduce them.
Add a locator to steps that are taking longer than 20 seconds
Adding a locator can save time by allowing you to select specific elements without relying on Autify's AI.
Note that it will disable the AI's search function on the step to which the locator is added, so if changes are made to the selector of the target element due to site modifications or anything else, the element will not be found and the step will fail.
For more information on the locator function, please refer to the following document.
Shorten the AI waiting time (default 20 seconds)
Note that for screens that originally take a long time to load, the test may fail because the steps are executed while the screen has not completed loading.
Therefore, if the target application does not include asynchronous transitions, this is not a problem, but it is not recommended for applications with asynchronous transitions. Note that even if the AI wait time is changed from the default 20 seconds, the wait time for the target step may be automatically shortened if the test execution is judged to be stable.
Example: If the AI wait time is changed from the default 20 seconds to 10 seconds, the AI wait time may be reduced to 5 seconds for the target step if the test succeeds continuously (a shorter time is adopted).
If you would like to use this method, please contact our support team with your desired wait time (in seconds) and we will take care of this for you.
Processes that cannot be shorten
Environment start up
In Autify Cloud environment, an environment for test execution is created at the start of each run. It takes a certain amount of time to secure the environment, and as a result, there may be a wait time before the test execution actually starts.
This is integrated as part of the test execution and therefore displayed as "Running".
Step 1 execution
Step 1 is the initial access of the url to be tested and there are cases were it can take a long time to load.
This also includes the time it takes for Autify's test execution engine to start, so it may take longer than if you were to display the page manually.
URL transition steps
Similar to the "Execution Time for Step 1," it may take time for the new page to load.
Test plan post-processing
When executing a test plan, there are post-processes after the execution of each test scenario. Therefore, the total test execution time for each scenario may not necessarily match the test plan execution time.
A screenshot is taken at each step and it is included in the total time. It cannot be shortened.
If the test execution still takes a long time even after implementing the methods described in this article, or if you have any questions, please feel free to contact our support team.