Testing on Real Mobile Devices

One of Autify’s most significant advantages is that, aside from full-sized PC browsers, you can also run tests on a wide variety of mobile devices.

The benefits of running automated tests on real mobile devices with Autify are:

• You don’t have to source your own mobile devices
• You can check behaviors specific to real devices, which may not be seen on emulators
• You can run automated tests repeatedly

However, there are characteristics and limitations specific to real mobile environments. Therefore, the key to successful mobile testing is understanding these limitations and considering using emulators when building tests.

In this article, you will find tips and tricks on how to test on real mobile devices, as well as common issues and how to solve them.

1. Mobile devices you can test on with Autify

Autify supports a variety of smartphones and tablets, which run on iOS or Android operating systems. For more information, please reference the following documentation:

• Supported OS and Browser Versions (Mobile)

2. The difference between testing on mobile devices and emulators

When you run tests on real devices and emulators (Device Mode in Chrome), there are slight differences in behavior due to their rendering engines. Here are a few key differences:

• Testing on an emulator is quick and easy. However, there may be slight variations in the user environment.
• Testing on real devices results in longer execution speeds, but rendering will be the consistent with the user environment.

We recommend that you use emulators when creating [Scenarios] or checking its stability, and use real devices once you’ve confirmed that there are no issues when running on emulators.

3. How to automate real mobile device testing with Autify

Here is the recommended order in which to run mobile device testing with Autify:

1. Create a [Scenario] with mobile-sized window dimensions
2. Run the test on a mobile emulator
3. Run the test on a real mobile device

Let’s take a look at each step in more detail.

3.1. Record a [Scenario] with a mobile-sized window dimensions.

You can find detailed instructions on how to facilitate this operation in the article below.

• Can I test my smartphone web? How can I record a scenario?

Please record the [Scenario] in Device Mode on Chrome and select a screen size you expect the user to have.

3.2. Running the test on a mobile emulator

The key to running tests on real devices is to first ensure that the test consistently succeeds on mobile emulators. Testing on emulators is a quick process, so you can perfect the [Scenario] by running the test repeatedly.

To run tests in mobile-size windows, you need to create a [Test Plan].

Create a new [Test Plan] and specify the execution environment by clicking the [Add] button in the execution environment section.

In the execution environment selection screen, you can select “iPhone X” and “Pixel 2” from the [Choose from Autify environments] section at the top of the screen. On the left side of the Device drop-down menu below [Choose from all environments], you can also select emulators for devices such as iPad and Galaxy. These mobile emulators run in Device Mode on Chrome/Linux in the Autify cloud.

iOS emulator

Android emulator

Once you have selected the execution environment for your [Test Plan] and the [Scenario], run the [Test Plan] and check the results. For tests executed on a mobile emulator, the screenshots displayed on the result page will be mobile sized dimensions.

3.3. Running the test on a real mobile device

Once the tests run on mobile emulators stably, you are ready to run the tests on real devices. Prepare a [Test Plan] for real device testing. Please select the appropriate device from the list of smartphones and tablets on the right side of the Device drop-down menu.

Real iOS device

Real Android device

Please note that in some older versions of real device environments, a notice will appear saying that the device (version) you have selected is not supported. You can still run tests in those environments, but test results are not supported.

4. How to choose between real mobile devices and emulators

Testing on real devices and testing on emulators both have their unique characteristics. We recommend that you take advantage of them, even after tests are running smoothly on real mobile devices.

Here are two points to keep in mind.

4.1. Plan test execution according to frequency and time required

Generally, testing on real mobile devices results in longer execution speeds. Real mobile device testing typically over 10x longer execution speeds compared to testing on mobile emulators in Autify's cloud environment. Therefore, we recommend that you test on emulators and real devices for different purposes. For example, use emulators for tests that need to be run frequently and run tests on real mobile devices over the weekend.

As shown in the figure above, real device tests are executed one at a time by default. As an option, you can increase the number of simultaneous test runs. For details, please refer to the  "How to run several tests on real devices simultaneously" section at the very bottom of the page.

4.2. Utilizing emulators when using functions unsupported on real devices

Some functions cannot be tested on real mobile devices and therefore must be tested on emulators. The following are typical examples:

• Uploading files
• Switching between windows on a real iOS device

When running tests that involve the above, please use an emulator to test the functions.

5. Issues with testing on real mobile devices

Although we are constantly working to improve Autify so that you can test comfortably even in mobile device environments, mobile device testing is an area with many technical challenges. Below, you will find a list of known issues with testing on real mobile devices.

5.1. Execution speed

• Test execution speed on real mobile devices depends on the [external device farm environment] , and is much slower than running tests on the [Autify cloud environment].

5.2. Device-specific behaviors

• In some cases, elements not present in the PC’s recording environment may affect test stability.

• E.g., Confirmation dialog displayed by the OS

5.3. Implementation of the Click step

• When you record a Click step on an element, a click will be executed instead of a tap in most cases. If the webpage expects and listens to a TouchEvent, the click step may not trigger the event and subsequent steps in the scenario may fail. Update your webpage so that it listens to MouseEvent including click, or apply JS Steps for Elements to dispatch a TouchEvent programmatically. (This behavior is the same in emulators)
• In some cases, the test execution engine cannot acquire the correct coordinate where a click/tap should be executed and the click/tap is executed in the wrong place. Please get in touch with support if you suspect the coordinate of an executed click/tap step was misaligned.

5.4. Susceptibility to timing issues

• In real mobile devices, timing issues tend to occur more often than in other environments. These issues stem from a discrepancy between the time it takes to display a page and the time it takes to execute the test. Please reference the section “Sometimes the page doesn’t transition after clicking a button/link” for an example.

5.5. Temporary issues

• Unknown errors or interruptions may occur depending on the device status in the external device farm. In most cases, you can solve the issue by rerunning the test. However, please do not hesitate to contact us if failures occur repeatedly.

6. Troubleshooting when testing on real devices

6.1. Sometimes the page doesn’t transition after clicking a button/link

Consider the following example:

• Step 3: Click the “Request Demo” button, and the page transitions.
• Step 4: Verify the text shown on the screen after the page transition.

Below, you can see an example result of running the above steps on a real iPhone. The page didn’t transition in step 4.

This problem can occur due to a discrepancy between the timing of page display and test execution. It tends to occur more frequently in real device environments.

If the page transition fails only on real devices and not in other environments, it may be solved by adding the [URL Should be] assertion after the operation that causes the page transition.

If you experience a similar issue, please give this method a try.

6.2. How to use different user accounts for each test execution environment (not limited to mobile devices)

We often hear from users who want to use different accounts for each test execution when there’s a possibility of running tests in parallel on various environments, including real devices. In this case, we recommend preparing a [JS Step] as shown in the following example and using the returned value in the subsequent steps.

/* An example where a different user account is returned depending on the execution environment */  
var userAgent = window.navigator.userAgent;  

if (userAgent.indexOf('Android') >= 0) {  
return ‘Android user’;  
}  
else if (userAgent.indexOf('iPhone') >= 0) {  
return ‘iPhone user’;  
}  
else if (userAgent.indexOf('iPad') >= 0) {  
return 'iPad user';  
}  
else {  
return ‘Default user’;  
}

6.3. How to run iPhone tests sequentially within an iPhone [Test Plan] and Android tests sequentially within an Android [Test Plan].

We will be releasing a new feature which will allow you to specify the order in which tests are executed within a [Test Plan]!

6.4. How to run several tests on real devices simultaneously

By default, you can run one test at a time in a real device environment. Optionally, this can be increased to two or more! You can reduce overall testing time by running multiple tests on real devices in parallel.

To send your feedback, please contact us via autify.com/contact.