Flexible test execution via API

Overview

The execute_scenarios API can be used to execute any scenario without creating a test plan.
Furthermore, it is also possible to specify execution environments, execution type (parallel/sequential), and URL replacements on the fly!

In this document, we'd like to show two ways to utilize this API via the Autify Command Line Interface (CLI):

1. autify web test run for easy execution
2. autify web api for more customizable execution

Prerequisite: Setup Autify CLI

Follow this guide to get started with Autify CLI. It only takes a few minutes.

1. Execute one test scenario in one execution environment (capability)

This is the simplest and most common use case - you want to run a test scenario in a specific execution environment (capability). Autify CLI provides a very powerful command for this use case called autify web test run. An example usage is as follows:

autify web test run https://app.autify.com/projects/0000/scenarios/0000 \
  --os Linux \
  --browser Chrome \
  --url-replacements http://example.com=http://example.net \
  --name "Sample"

This simple command starts the specified test scenario on the Linux/Chrome capability with a URL replacement from http://example.com to http://example.net and names the execution as "Sample", which is used when displaying the test result on the Autify UI.

The available capabilities will vary based on your plan and workspace settings.

Desktop capabilities

• --os:
  • Linux / Windows Server / Windows / OS X
• --browser:
  • Chrome / Firefox / Safari / Edge

Mobile capabilities

• --device:
  • iPhone X / Galaxy S20 / ...
• --device-type:
  • emulator / mobile
• --os-version:
  • 11.0 / ... (Android)
  • 15 / ... (iOS)

When running autify web test run, you only have to specify the minimum options needed to uniquely identify a set of capabilities. For example, if you only have one environment that uses Chrome on Linux, specifying --os Linux and --browser Chrome is enough to identify it.

You can see all available capabilities for your workspace by running the command autify web api list-capabilities --project-id {WORKSPACE_ID}.

$ autify web api list-capabilities --project-id 0000
[
  {
    "os": "Linux",
    "os_type": "linux",
    "os_version": null,
    "browser": "Chrome",
    "browser_type": "chrome",
    "browser_version": "126.0",
    "device": null,
    "device_type": "computer",
    "unsupported": false
  },
  {
    "os": "Windows",
    "os_type": "windows",
    "os_version": "10",
    "browser": "Edge",
    "browser_type": "edge",
    "browser_version": "124.0",
    "device": null,
    "device_type": "computer",
    "unsupported": false
  },
  ...

2. Execute via API directly

If you need more complex solutions not covered by autify web test run, you can use autify web api commands. They are commands with one-to-one mappings of each Autify API.

You can do this by following these steps:

1. Get the scenario IDs of the scenarios that you want to run
   1. From the URL https://app.autify.com/projects/{WORKSPACE_ID}/scenarios/{SCENARIO_ID}
   2. By using the command autify web api list-scenarios
2. Get the capabilities (execution environments available to you) using autify web api list-capabilities
3. Execute autify web api execute-scenarios

Let’s take a closer look.

Get the scenario IDs

First, you need to get the scenario IDs of scenarios to be run.

Get scenario IDs via URL

If the scenario has the following URL, {WORKSPACE_ID} is your workspace ID (formerly known as project ID) and {SCENARIO_ID} is the scenario ID that will be required for running scenarios.

https://app.autify.com/projects/{WORKSPACE_ID}/scenarios/{SCENARIO_ID}

Get scenario IDs using the CLI

It’s also possible to get scenario IDs in bulk by executing autify web api list-scenarios like this:

autify web api list-scenarios --project-id {WORKSPACE_ID}

If you want to use your own HTTP client, send a request like the example below:

GET /projects/{WORKSPACE_ID}/scenarios HTTP/1.1
Accept: application/json, text/plain, */*
Authorization: Bearer YOUR_PERSONAL_ACCESS_TOKEN
Host: app.autify.com

Get the capabilities

Next, get the list of capabilities that are available for your workspace. Capabilities are the execution environments that can be specified when you run autify web api execute-scenarios. You can get the list of capabilites by autify web api list-capabilities.

autify web api list-capabilities --project-id {WORKSPACE_ID}

The response may look like the following array of objects. Each object represents a capability - execution environments that are available to you.

$ autify web api list-capabilities --project-id 0000
[
  {
    "os": "Linux",
    "os_type": "linux",
    "os_version": null,
    "browser": "Chrome",
    "browser_type": "chrome",
    "browser_version": "126.0",
    "device": null,
    "device_type": "computer",
    "unsupported": false
  },
  {
    "os": "Windows",
    "os_type": "windows",
    "os_version": "10",
    "browser": "Edge",
    "browser_type": "edge",
    "browser_version": "124.0",
    "device": null,
    "device_type": "computer",
    "unsupported": false
  },
  ...

If you want to use your own HTTP client, send a request like the below example:

GET /projects/{WORKSPACE_ID}/capabilities HTTP/1.1
Accept: application/json, text/plain, */*
Authorization: Bearer YOUR_PERSONAL_ACCESS_TOKEN
Host: app.autify.com

Execute autify web api execute-scenarios

Now that you have your scenario IDs and capabilities, it’s possible to execute autify web api execute-scenarios to run your tests. The information must be set as JSON format data along with other optional parameters.

Parameters for execute_scenarios API

scenarios and capabilities are required parameters, while others are optional.

*1 This name is only used for displaying the test result. Note that this has nothing to do with the names of the scenarios you would like to run.

Example

Options included:

• Run multiple scenarios sequentially on Chrome/Linux and Edge/Windows Server
• Name for the test result: “Order flow”
• URL replacement: “https://example.com” to “https://autify.com”

autify web api execute-scenarios --project-id '{WORKSPACE_ID}' --execute-scenarios-request '{
  "name": "Order flow", 
  "execution_type": "sequential",
  "url_replacements": [
    {
      "pattern_url": "https://example.com",
      "replacement_url": "https://autify.com"
    }
  ],
  "capabilities": [
    {
      "os": "Linux",
      "browser": "Chrome"
    },
    {
      "os": "Windows Server",
      "os_version": "2019",
      "browser": "Edge"
    }
  ],
  "scenarios": [
    {"id": 1234 },
    {"id": 2345 },
    {"id": 3456 },
    {"id": 4567 }
  ]
}'

Here is the result:

If you want to use your own HTTP client, send a request like the example below:

POST /projects/{WORKSPACE_ID}/execute_scenarios HTTP/1.1
Accept: application/json, text/plain, */*
Authorization: Bearer YOUR_PERSONAL_ACCESS_TOKEN
Content-Type: application/json
Content-Length: 323
Host: app.autify.com

{"name":"Order flow","execution_type":"sequential","url_replacements":[{"pattern_url":"https://example.com","replacement_url":"https://autify.com"}],"capabilities":[{"os":"Linux","browser":"Chrome"},{"os":"Windows Server","os_version":"2019","browser":"Edge"}],"scenarios":[{"id":1234},{"id":2345},{"id":3456},{"id":4567}]}

Conclusion

This API brings true flexibility to your testing. You no longer need to create test plans to run the tests you need at any given situation, and you don’t have to think about maintaining ad-hoc test plans. We cannot wait to see how you will run your tests dynamically with Autify!