Integrate with code review process

You can run Autify's end-to-end testing during code review process so that the proposed changes can be verified just like you do on staging/production environments. This document describes what is the benefit of this integration and how to do that.

Why running Autify during code review process?

Code review is the first step of the software release process. A person proposes some changes to the code base and the other person reviews the change whether they are valid, working as expected, and won't break unnecessary functionalities, etc. There are lots of automated techniques to make code reviews easier, reliable and wider perspective, such as unit test. Unit test can cover bunch of functionalities of the software, but you also want to test some kind of end-to-end experience such as login operation by real browser. Historically, implementing such end-to-end test scenario by code is not an easy task but Autify can provide a solution here. By using Autify's intuitive recorder and scenario editor, you can create an end-to-end test scenario in 5 minutes and no need to worry about the maintenance of the scenarios because Autify can resolve and learn majority of UI changes like CSS class. Therefore, Autify can validate whether the proposed code changes can pass the same end-to-end scenario recorded in the production environment and this gives more confidence to the reviewer.

How to run Autify during code review process?

Autify needs an endpoint i.e. URL accessible from its test execution engine in cloud for running test scenarios. Therefore, you need to setup an ephemeral endpoint per Pull request on GitHub, for example, and tell Autify to replace the recorded URL e.g. production URL to the generated ephemeral URL.

How to setup such ephemeral endpoint depends on your infrastructure. For example, some deployment providers like Vercel have a preview feature and a preview endpoint will be automatically created for every pull request out-of-the-box. Or, you can setup your own deployment workflow by using Infrastructure as a Code solution and spin up an ephemeral environment for the pull request.

In summary, you will need a workflow engine that controls task dependencies, and a mechanism to generate ephemeral endpoints to the apps built based on your changes. See the table below to get more insights:

If you already have such workflows, though, you can easily hook it up and run Autify test scenarios against the ephemeral endpoint. In short, the steps are like below:

1. Fetch the ephemeral endpoint in your CI/CD solution when a code review is created/updated
2. Run Autify tests to the ephemeral endpoint
3. Feedback to the code review

In the following example, we will demonstrate this with GitHub Actions and Vercel.

Example setup: GitHub and Vercel

In this example, we're going to use GitHub as a source code repository and Vercel as an ephemeral deployment environment but you can still get some idea if you use different solutions.

Vercel's preview URL is an ephemeral endpoint

Once you connect your GitHub repository to Vercel, Vercel automatically executes preview deployment on each push to any pull request and generates a unique preview URL out-of-the-box. This is a good example of the ephemeral endpoint describe above and Vercel bot notifies the preview URL to you via GitHub issue (pull request) comment. See Vercel's document for more details.

GitHub Actions can be used as the workflow engine

GitHub Actions is a workflow triggered by various events on GitHub such as code push and it runs pre-defined jobs on the runners. There is a GitHub event when a comment on a GitHub issue is created/edited/etc. so that we can trigger a GitHub Actions workflow whenever the Vercel bot writes a comment on your pull request e.g. preview deployment is done and run any arbitrary jobs like starting Autify tests and feedback the result URL as a comment on the same pull request.

How to setup GitHub Actions to integrate with pull request?

Since Vercel's preview URL is generated without any other action, we don't have to worry about how to setup the ephemeral endpoint alerted above. We just need to implement the steps 1-3 above. Here is an example setup of GitHub Actions on the repository you deploy to Vercel.

Put the file below at .github/workflows/issue_comment.yml under your repository:

on:
  issue_comment:
    types: [created, edited]

jobs:
  vercel_commented:
    runs-on: ubuntu-latest
    if: ${{ github.event.issue.pull_request && github.event.comment.user.login == 'vercel[bot]' }}
    steps:
      - name: Get preview URL
        id: get-url
        run: |
          body=$(cat <<EOS
          ${{ github.event.comment.body }}
          EOS
          )
          url=$(echo -e "$body" | grep -Eo 'https://[^\/]+\.vercel\.app') || true
          echo ::set-output name=result::"$url"

      - name: Run test on Autify for Web
        id: run
        if: ${{ startsWith(steps.get-url.outputs.result, 'http') }}
        uses: autifyhq/actions-web-test-run@v2
        with:
          access-token: ${{ secrets.AUTIFY_WEB_ACCESS_TOKEN }}
          autify-test-url: https://app.autify.com/projects/00/scenarios/000
          url-replacements: YOUR_RECORDED_VERCEL_URL=${{ steps.get-url.outputs.result }}
        continue-on-error: true

      - name: Comment Autify test result URL on the pull request
        if: ${{ steps.run.conclusion == 'success' }}
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          number: ${{ github.event.issue.number }}
          message: |
            Autify test was started. Check ${{ steps.run.outputs.result-url }}
            ```
            ${{ steps.run.outputs.log }}
            ```

      - name: Comment failed to start Autify test on the pull request
        if: ${{ steps.run.conclusion == 'failure' }}
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          number: ${{ github.event.issue.number }}
          message: |
            Autify test was failed to start.
            ```
            ${{ steps.run.outputs.log }}
            ```

Once you create a pull request, Vercel bot will comment on the pull request once the preview deployment is done like below:

Then, the GitHub Actions workflow you created above will be triggered, then it will start the Autify test scenario you specified and comment the Autify test result URL and log like below:

Now, the reviewer (and of course the requester) can check the test result by clicking the link easily!

Conclusion

This document describes the benefit and the concept of Autify integration with code review process and shows an example to run Autify test against Vercel preview URL using GitHub Actions. We hope it would help you to implement your Autify integration with code review process.