CI/CD with Autify - Run automated tests in dynamically generated environments
  • 07 Jun 2022
  • 4 Minutes to read
  • Contributors
  • Dark

CI/CD with Autify - Run automated tests in dynamically generated environments

  • Dark

One of the best ways to use Autify is to run tests after adding a feature or before a stable release to ensure there are no problems with existing features.

You can automate this process by running tests with Autify in the CI (Continuous Integration) and CD (Continuous Delivery) pipeline. However, for verification environments automatically generated by the CI/CD flow, it is not uncommon for a different URL to be issued each time.

Fig 1. Example of when the Artifact is deployed to an environment that always has the same URL.

Fig 2. Example of when the Artifact is deployed to an environment with a different URL each time.

To automate testing an environment that dynamically changes, like the one described above, we will use Autify API.
In this article, you will learn the best practices for how to deal with these situations.


In this article, we will assume that the following two points already exist:

  • A CI/CD workflow created using tools such as CircleCI, Jenkins, and GitHub Actions.
    • We also assume that URLs of generated verification environments can be identified in the workflow.
  • A [Test Plan] that specifies which [Scenario] you want to execute automatically at a specific timing.

As of October 2021, Autify has the following four APIs. In this article, we will use the APIs in bold.

  • schedules: Executes a [Test Plan]
  • scenarios: Gets a [Scenario]
  • results: Gets test results
  • url_replacements: Actions to replace a URL

For more information on APIs, please see Autify API page.

Issue a personal access token

You will need a personal access token to use APIs, so make sure you generate one in advance.

Please refer to "Issue a personal access token" in "Integrate with API".

Add a job that calls an Autify Web API to an existing workflow

Add a job for the following process:

  • 3-1. Add a [URL Replacement] for the target [Test Plan].
  • 3-2. Execute the target [Test Plan]
  • 3-3. Delete the [URL Replacement] you added in 3-1.

Let's take a closer look.

Add a [URL Replacement] for the target [Test Plan]

Add a [URL Replacement] to the target [Test Plan] using a [url_replacements API].

For example, suppose the [Test Plan] runs a [Scenario] in which user interaction with is recorded. Also assume that, when the workflow is executed, a verification environment with the URL is generated. In this case, you can add a [URL Replacement] for the [Test Plan] by issuing the following API request:

curl -X POST -H "Authorization: Bearer PERSONAL_ACCESS_TOKEN"{test_plan_id}/url_replacements -H "Content-Type: application/json" -d '{"url_replacement": { "pattern_url":"", "replacement_url": ""}}'

Replace PERSONAL_ACCESS_TOKEN with the token you generated in 1. and {test_plan_id} with the ID of the target [Test Plan] (for example, when you open the [Test Plan] on a web console and the URL is, the Test Plan ID is 3).

Issuing an API request is the same as adding the following configuration in the web console:

For more information about this feature, please see this document.

By taking these steps, even with a [Scenario] in which user interaction with is recorded, the test will be executed after the URL is automatically replaced to

If the above API request is successful, you will get the following response:


Note that 1111 is a [URL Replacement] identifier. We will use this ID later.

Execute the target [Test Plan]

Use [schedules API] to run the test. For example, you can run the target [Test Plan] by issuing the following API request:

curl -X POST -H "Authorization: Bearer PERSONAL_ACCESS_TOKEN"{test_plan_id}

Replace PERSONAL_ACCESS_TOKEN and {test_plan_id} in the same way as 2-1.

If the request is sent to the [schedules API] properly, it will return the following response:


id is a test result identifier (in this example, the ID is 999999). To proceed to the next step of the workflow, we need to get the result of the initiated [Test Plan]. Use the ID and send a request to the [results API]. For example,

curl -X GET -H "Authorization:Bearer PERSONAL_ACCESS_TOKEN"{project_id}/results/{result_id}

Replace PERSONAL_ACCESS_TOKEN... You know what to do!

Replace {project_id} with the Autify Project ID (for example, if the URL when you log in to Autify is, the Project ID is 1. For {result_id}, specify the response content you got from sending a request to [schedules API] (in the above example, specify 999999).

It will take some time for Autify to finish running tests. How long it will take depends on the test, so in practice, you will need to regularly run the [results API] and check whether the test is complete.

It will take some trial and error to create the system yourself. You can find samples for the following services, so please take a look. (we plan to gradually add more in the future!)

Delete the [URL Replacement] you added in 3-1.

After the test is finished, make sure to delete the [URL Replacement], which was dynamically added by the job flow (if you don't, the test will be rerun on https://1234567.example.comthe next time you run this [Test Plan]).

You can delete the [URL Replacement] added in 2-1 by sending the following API request:

curl -X DELETE -H "Authorization:Bearer PERSONAL_ACCESS_TOKEN" "{test_plan_id}/url_replacements/{url_replacement_id}"

I'm sure you are already familiar with how to replace the character stings in the above command, but{url_replacement_id} is the [URL Replacement] identifier you got in 2-1. In this example, it is replaced with 1111.

If this API request is sent successfully, the [URL Replacement] has successfully been deleted. Now everything is back to how it was before we started.

There you have it! Now you know how to automate testing on verification environments that are dynamically generated. I hope this helped make the most of Autify!

Reference articles

First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.