- Print
- DarkLight
In this document, we'll explain how to use Autify Connect with Autify Command Line Interface (CLI) and the other existing CI/CD integrations. Autify CLI provides unified and streamlined experiences for Autify Connect just like bunch of other commands and our CI/CD integrations already support it as well.
Read Autify Connect Overview first so that you understand the idea of what Autify Connect is.
Autify Connect's use cases
Before jumping into the details, let's set our tone about the use cases of Autify Connect. Autify Connect provides one significant functionality to you i.e. running Autify's test even if the endpoint is unavailable via the internet. This opens three different use cases for you:
- Test against endpoints in your private network with long-running static Access Point
- Test against endpoints in your private network via ephemeral Access Point on your local machine
- Test against your
localhost
endpoints via ephemeral Access Point on your local machine
Use case 1
The use case 1 is for the cases you want to run Autify's test against your private website such as staging environment behind your firewall, or corporate internal website, etc. In this case. you need to run at least one Autify Connect Client inside your private network with a static Access Point like "corp network". Then, you can use that Access Point for any test you want to run inside your private network as long as someone maintains the availability of the Access Point i.e. keeps the Autify Connect Client up and running for the static Access Point by daemonizing and moniroting it.
Use case 2
The use case 2 is for the same target endpoins as the use case 1 but Access Point is ephemeral and running on your local machine. In this case, you don't have to manage such long-running process. Instead, you can create an ephemeral Access Point when needed and run Autify Connect Client with the ephemeral one. Now you can run Autify test against private network via your local machine as long as you keep running the Autify Connect Client and your local machine has access to the target endpoints.
Use case 3
The use case 3 is for a completely different purpose from the use case 1 and 2. The target endpoints are localhost
/127.0.0.1
e.g. local development server but you still want to run Autify test against such local endpoints. The ephemeral Access Point described in the use case 2 can be also useful for this case because your local machine has access to your localhost
. Therefore, even though the target endpoints are totally different, the solution is identical for both the use case 2 and 3.
Autify CLI provides the easiest ways to support all the use cases above. We'll go though them one by one but you can jump into the section of the specific use case you want to know.
Getting started
Autify CLI provides a set of commands to operate Autify Connect on your local machine easily. Let's walk through them and you realize there are no frictions to get started with Autify Connect.
Prerequisites
Record a test scenario
You need to create a test scenario at Autify NoCode Web. Record a scenario against your private endpoint or your localhost
endpoint e.g. development server. Note the Autify's URL of the recorded test scenario e.g. https://app.autify.com/projects/0000/scenarios/0000
.
All the following secitons use a test scenario as a target but you can also use a test plan by passing its URL like https://app.autify.com/projects/0000/test_plans/0000
.
For localhost
/127.0.0.1
recording, you should use localhost.autify.com
while recording instead of localhost
/127.0.0.1
for now.
If you record the scenario with localhost
/127.0.0.1
, you need to ask URL replacement to localhost.autify.com
when you run the test. We're working on eliminating this restrictions. Until then, either recording with localhost.autify.com
or URL replacement to localhost.autify.com
are needed.
Install Autify CLI
You need to install Autify CLI and set a personal access token of Autify NoCode Web. See Autify CLI's document for more details.
Install Autify Connect Client
Just run the command below. This will download an appropriate autifyconnect
binary to the cache directory for Autify CLI:
autify connect client install
Example output:
$ autify connect client install
Installing Autify Connect Client (version: v0.6.1)... done
Successfully installed Autify Connect Client (path: /home/node/.cache/autify/autifyconnect, version: Autify Connect version v0.6.1, build ca0972e)
1. Test against endpoints in your private network with long-running static Access Point
The steps below are only for getting started or demo purpose. For the production use, you probably want to run autifyconnect
binary directly from your supervisor like systemd
, supervisord
, etc. See this document about how to run and operate autifyconnect
in production.
In this use case, you'll create a static Access Point once and re-use it. First, create an Access Point by Autify CLI:
autify connect access-point create --name ACCESS_POINT_NAME --web-workspace-id WORKSPACE_ID
This command stores the generated key in the config directory of Autify CLI automatically.
If you prefer to create an Access Point via UI, you can also do that. Once created, store the name and key of the Access Point by the command below. You'll be asked to enter the generated key via STDIN:
autify connect access-point set --name ACCESS_POINT_NAME
Let's start the Autify Connect Client with the created Access Point. Keep this terminal up and running somewhere:
autify connect client start
Now, the static Access Point is ready to use. Run the Autify test scenario you recorded with the Access Point:
autify web test run TEST_SCENARIO_URL --autify-connect ACCESS_POINT_NAME --wait
This command will start the test scenario with connecting to the Access Point and wait until the test finishes.
2. Test against endpoints in your private network via ephemeral Access Point on your local machine
In this case, you don't need a static Access Point but just an ephemeral one. Autify CLI supports automatic creating/dereting an ephemeral Access Point when running Autify Connect Client.
Run the command below. It will create an ephemeral Access Point, run Autify Connect Client with the generated key, run the test scenario. Once the test scenario finishes, it will also stop the Autify Connect Client and delete the ephemeral Access Point automatically:
autify web test run TEST_SCENARIO_URL --autify-connect-client --wait
Example output:
$ autify web test run https://app.autify.com/projects/000/scenarios/0000 --autify-connect-client --wait
[Autify Connect Manager] 2022-09-15T19:54:39.213Z info Ephemeral Access Point was created: autify-cli-UUID
Starting Autify Connect Client...
[Autify Connect Manager] 2022-09-15T19:54:39.253Z info Starting Autify Connect Client (accessPoint: autify-cli-UUID, debugServerPort: 51035, path: /Users/runner/Library/Caches/autify/autifyconnect, version: Autify Connect version v0.6.2, build 0913a76)
Waiting until Autify Connect Client is ready...
[Autify Connect Client] 2022-09-15T19:54:39.262Z info start serving a debug server on http://localhost:51035
[Autify Connect Client] 2022-09-15T19:54:39.262Z info Starting to establish a secure connection with the Autify connect server. Your session ID is "...".
[Autify Connect Client] 2022-09-15T19:54:40.509Z info Successfully connected!
Autify Connect Client is ready!
✅ Successfully started: https://app.autify.com/projects/000/results/1111 (Capability is Linux Chrome 104.0)
🕐 Waiting for the test result: https://app.autify.com/projects/000/results/1111
✔ Waiting... (timeout: 300 s)
✅ Test passed!: https://app.autify.com/projects/000/results/1111
Waiting until Autify Connect Client exits...
[Autify Connect Client] 2022-09-15T19:54:51.932Z info Interrupt received.
[Autify Connect Client] 2022-09-15T19:54:51.932Z info Shutdown completed.
[Autify Connect Manager] 2022-09-15T19:54:52.358Z info Ephemeral Access Point was deleted: autify-cli-UUID
[Autify Connect Manager] 2022-09-15T19:54:52.362Z info Autify Connect Client exited (code: 0, signal: null)
Autify Connect Client exited.
This command should cover most of the use case but if you want to run Autify Connect Client separately, first run it with your workspace ID to which your recorded test scenario belongs (the workspace ID is inside the URL after /projects/
):
autify connect client start --web-workspace-id 000
Then, run the test scenario against the created ephemeral Access Point (the name should be shown in the previous command's output like Starting Autify Connect Client (accessPoint: autify-cli-UUID,...
):
autify web test run TEST_SCENARIO_URL --autify-connect "autify-cli-UUID" --wait
3. Test against your localhost
endpoints via ephemeral Access Point on your local machine
In this case, you just want an ephemeral Access Point on your local machine. Autify CLI supports automatic creating/dereting an ephemeral Access Point when running Autify Connect Client.
First of all, run your localhost
server in a separate terminal. It would be like rails server
, npm run start
, etc. depending on your task runner. Also assuming your test scenario is recorded against the local endpoint like http://localhost.autify.com:3000
(Remember: We'd like to support localhost
out-of-the-box but currently you need to use localhost.autify.com
for recording or URL replacements.)
Then, run the same command as the use case 2:
autify web test run TEST_SCENARIO_URL --autify-connect-client --wait
If you recorded the test scenario against localhost
/127.0.0.1
, you need to request URL replacements like below (for now):
autify web test run TEST_SCENARIO_URL --autify-connect-client --wait --url-replacements http://localhost=http://localhost.autify.com
A demo of localhost
development use case
Here is a step-by-step example to demonstrate a typical development cycle with the use case 3:
- Launch a dead simple
localhost
application.- The command bellow will launch a static http server at
http://localhost:8000
on macOS. Keep it running in the foreground while going through the rest.
- The command bellow will launch a static http server at
echo Hello world! > index.html
python3 -mhttp.server
- Record a test scenario against the
http://localhost.autify.com:8000
server.- Add an assertion on the text element that the text shold be "Hello world!".
- In real use cases, this test scenario should be shared among your team and optionally used for testing staging/production endpoints as well.
- Run the test scenario from Autify using Autify Connect Client.
autify web test run https://app.autify.com/projects/000/scenarios/000 --autify-connect-client --wait
- Confim the test passed.
- Modify your application.
echo Hello Autify! > index.html
- Run the same test scenario from Autify using Autify Connect Client by the same command.
autify web test run https://app.autify.com/projects/000/scenarios/000 --autify-connect-client --wait
- Confirm the test failed this time.
- This imitated when your local change causes a regression failure.
As you can see above, you can verify your localhost
application by Autify with a single command. You can embed that command to your task runner e.g. Rake/npm/etc. to run it natively if you want.
CI/CD integrations for Autify Connect
All of our official CI/CD integrations use Autify CLI under the hood. Therefore, you can do exactly the same things above with these CI/CD integrations, too.
Run a test scenario against localhost
server inside CI/CD runners during code review process
The use case 3 can be extended to do the same thing inside CI/CD runners e.g. GitHub Actions hosted runners. This allows you to run Autify test during code review processes without extra infastructure. It's a critical heavy lifting for the blocker of Autify integration with code review process. See more details about code review integration here.
To enable this integration, you need to ensure your localhost
server can launch and work properly inside CI/CD runners as well. Typical pitfalls are database (seed), other microservices (mock, monorepo or Docker), credentials (must set in the secret store) and network (CI/CD managed runners might not reside in your private network).
Once it works just like your local machine, you can run a test scenario against the localhost
server inside CI/CD runners.
For example, GitHub Actions and Rails application. You can add this workflow to your Rails applicaiton's GitHub repository. Then, each pull request will launch rails server
inside CI/CD runners and run Autify test via Autify Connect Client:
on:
pull_request:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: ruby/setup-ruby@v1
with:
ruby-version: '3.0'
# Running your Rails server in the background. It might require additional setup like database or passing credentials via environment variables.
- run: bundle exec rails server --daemon
- uses: autifyhq/actions-web-test-run@v2
with:
access-token: ${{ secrets.AUTIFY_WEB_ACCESS_TOKEN }}
# Assuming the test scenario was recorded against http://localhost.autify.com
autify-test-url: https://app.autify.com/projects/000/scenarios/000
autify-connect-client: true
wait: true
timeout: 300
This will consume your GitHub Actions hosted runner's minutes while waiting. Be careful when extending the timeout value.
If you want us to support running these processes inside our infrastructure on behalf of you, please contact us because we'd like to hear from you.
One concern here is that CI/CD providers might disconnect/reject long-lasting TCP connections because it looks like abuse e.g. backdoor. Self hosted runner can avoid such limitation as well as the cost issue (although they need to pay for the runner's platform like Amazon EC2.)
Run a test scenario with Autify Connect Client already running somewhere else
This is for use case 1 and it's pretty simple because Autify Connect Client is already running. Assuming you already deployed your software to your private network application. Then, just run a test scenario like you can do for a public network applicatoin.
Here is an example of GitHub Actions after your deployment step. This step will start Autify test using Autify Connect Access Point named YOUR_ACCESS_POINT_NAME
.
- uses: autifyhq/actions-web-test-run@v2
with:
access-token: ${{ secrets.AUTIFY_WEB_ACCESS_TOKEN }}
autify-test-url: https://app.autify.com/projects/000/scenarios/000
autify-connect: YOUR_ACCESS_POINT_NAME
Conclusion
In this document, we explain the use cases of Autify Connect, how to use Autify Connect Client with Autify CLI and how to easily run Autify test along with Autify Connect Client. This integration should open various capabilites for you to test your websites uniquely like test against private network or test against localhost
development server easily even inside CI/CD runners.