Flexible test execution via API
  • 07 Sep 2022
  • 5 Minutes to read
  • Contributors
  • Dark
    Light

Flexible test execution via API

  • Dark
    Light

Overview

Previously, it was necessary to create a plan to run tests via Autify API, but as of May 2022, you can run any scenarios via our execute_scenarios API without creating a test plan. What is more, it’s also possible to specify execution environments, execution type (parallel/sequential), and URL replacements on the fly!

In this document, we'd like to show two ways to utilize this API via Autify Command Line Interface (CLI): 1. autify web test run for easy execution and 2. autify web api for more customizable execution.

Prerequisite: Setup Autify CLI

Follow the Getting started of Autify CLI. It only takes a few minutes.

1. Execute one test scenario in one execution environment (capability)

This should be the simplest but the most popular use case i.e. you want to run a test scenario in a specific execution environment (capability). Autify CLI provides a very powerful command for this use case called autify web test run. The usage is like below:

autify web test run https://app.autify.com/projects/0000/scenarios/0000 \
  --os Linux \
  --browser Chrome \
  --url-replacements http://example.com=http://example.net \
  --name "Sample"

This simple command starts the specified test scenario on the Linux/Chrome capability with a URL replacement from http://example.com to http://example.net and name the execution as "Sample" (used when displaying the test result on Autify UI).

The available capabilities vary in your workspace but you might be able to use some examples below if your workspace is allowed:

Desktop capabilities

  • --os:
    • Linux / Windows Server / Windows / OS X
  • --browser:
    • Chrome / Firefox / Safari / Edge

Mobile capabilities

  • --device:
    • iPhone X / Galaxy S20 / ...
  • --device-type:
    • emulator / mobile
  • --os-version:
    • 11.0 / ... (Android)
    • 15 / ... (iOS)

You can see the all available capabilities for your workspace by the command below. autify web test run tries to filter the same list by the given options. So, you don't have to specify all options if the capability is unique by a subset of the options like just --os and browser:

$ autify web api list-capabilities --project-id 0000
[
  {
    "os": "Linux",
    "os_version": "",
    "browser": "Chrome",
    "browser_version": "104.0",
    "device": null,
    "device_type": null,
    "unsupported": false
  },
  {
    "os": "Windows",
    "os_version": "10",
    "browser": "Chrome",
    "browser_version": "104.0",
    "device": null,
    "device_type": null,
    "unsupported": false
  },
  ...
autify web test run doesn't support all possible use cases.

Here is the list of some unsupported use cases by autify web test run as of today, but we can support them if they are desired:

  • Running multiple test scenarios in one command
    • Workaround: Execute multiple autify web test run per test scenario instead.
  • Running multipe capabilities in one command
    • Workaround: Execute multiple autify web test run per capability instead.

2. Execute via API directly

If you need more complex solutions not covered by autify web test run, you can still craft your command by using autify web api commands. They are commands with one-to-one mappings of each Autify API.

You can do this by following these steps:

  1. Get the scenario IDs of the scenarios that you want to run, either by the URLs you see on the Autify UI or by autify web api list-scenarios.
  2. Get the capabilities (execution environments available to you) using autify web api list-capabilities.
  3. Execute autify web api execute-scenarios

Let’s take a closer look.

Get the scenario IDs

First, you need to get the scenario IDs of scenarios to be run.

Get the scenario IDs from URL

If the scenario has the following URL, {workspace_id} is your workspace ID (former project ID) and {scenario_id} is the scenario ID that will be required for running scenarios.

Grab scenario IDs by autify web api list-scenarios

It’s also possible to grab scenario IDs in bulk by executiong autify web api list-scenarios like this:

autify web api list-scenarios --project-id {workspace_id}

If you want to use your own HTTP client, send a reqeust like below:

GET /projects/{workspace_id}/scenarios HTTP/1.1
Accept: application/json, text/plain, */*
Authorization: Bearer YOUR_PERSONAL_ACCESS_TOKEN
Host: app.autify.com

Get the capabilities

Next, let’s get the list of capabilities that are available for you. Capabilities are the execution environments that can be specified when you run autify web api execute-scenarios. You can get the list of capabilites by autify web api list-capabilities:

autify web api list-capabilities --project-id {workspace_id}

The response may look like the following array of objects. Each object represents a capability - execution environments available for you.

[
  {
    "os": "Linux",
    "os_version": "",
    "browser": "Chrome",
    "browser_version": "100.0",
    "device": null,
    "device_type": null,
    "unsupported": false
  },
  {
    "os": "Windows Server",
    "os_version": "2019",
    "browser": "Edge",
    "browser_version": "100.0",
    "device": null,
    "device_type": null,
    "unsupported": false
  },
  {
    "os": "iOS",
    "os_version": "",
    "device": "iPhone X",
    "device_type": "emulator",
    "unsupported": false
  },
	{
    "os": "iOS",
    "os_version": "14",
    "device": "iPhone 11",
    "device_type": "mobile",
    "unsupported": false
  }
]

If you want to use your own HTTP client, send a reqeust like below:

GET /projects/{workspace_id}/capabilities HTTP/1.1
Accept: application/json, text/plain, */*
Authorization: Bearer YOUR_PERSONAL_ACCESS_TOKEN
Host: app.autify.com

Execute autify web api execute-scenarios

Now that you have your scenario IDs and capabilities, it’s possible to execute autify web api execute-scenarios to run your tests. The information must be set as JSON format data along with other optional parameters.

Parameters for execute_scenarios API

scenarios and capabilities are required parameters, while others are optional.

Field name Data type Name Required? Default value
scenarios List of objects Yes
number id
name (*1) string No API execution + date the test is executed
execution_type "parallel” / “sequential” No “parallel” (”sequential” In case parallel execution was disabled in the workspace)
capabilities List of objects Yes
string os
string os_version
string device
string browser
url_replacements List of objects No blank (No URL replacemnts)
string pattern_url Both pattern_url and replacement_url are required for each url_replacements items
string replacement_url

*1 This name will only be used for displaying the test result. Note that this has nothing to do with the names of the scenarios you would like to run.

Example

Options included:

  • Run multiple scenarios sequentially on Chrome/Linux and Edge/Window Server
  • Name for the test result: “Order flow”
  • URL replacement: “https://example.com” to “https://autify.com”
autify web api execute-scenarios --project-id '{workspace_id}' --execute-scenarios-request '{
  "name": "Order flow", 
  "execution_type": "sequential",
  "url_replacements": [
    {
      "pattern_url": "https://example.com",
      "replacement_url": "https://autify.com"
    }
  ],
  "capabilities": [
    {
      "os": "Linux",
      "browser": "Chrome"
    },
    {
      "os": "Windows Server",
      "os_version": "2019",
      "browser": "Edge"
    }
  ],
  "scenarios": [
    {"id": 1234 },
    {"id": 2345 },
    {"id": 3456 },
    {"id": 4567 }
  ]
}'

Here is the result:

Screen Shot 2022-05-11 at 15.53.40.png

Screen Shot 2022-05-11 at 15.54.31.png

If you want to use your own HTTP client, send a reqeust like below:

POST /projects/{workspace_id}/execute_scenarios HTTP/1.1
Accept: application/json, text/plain, */*
Authorization: Bearer YOUR_PERSONAL_ACCESS_TOKEN
Content-Type: application/json
Content-Length: 323
Host: app.autify.com

{"name":"Order flow","execution_type":"sequential","url_replacements":[{"pattern_url":"https://example.com","replacement_url":"https://autify.com"}],"capabilities":[{"os":"Linux","browser":"Chrome"},{"os":"Windows Server","os_version":"2019","browser":"Edge"}],"scenarios":[{"id":1234},{"id":2345},{"id":3456},{"id":4567}]}

Conclusion

This API brings true flexibility to your testing. You no longer need to create test plans to run the tests you need at any given situation, and you don’t have to think about maintaining ad-hoc test plans. We cannot wait to see how dynamically you will run your tests on Autify!


First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.