Integrate with code review process
  • 31 Aug 2022
  • 6 Minutes to read
  • Contributors
  • Dark
    Light

Integrate with code review process

  • Dark
    Light

You can run Autify's end-to-end testing during code review process so that the proposed changes can be verified just like you do on staging/production environments. This document describes what is the benefit of this integration and how to do that.

Recommend to read CI/CD with Autify first

CI/CD with Autify describes general concept of CI/CD and several possible CI/CD integrations including code review process.

If you haven't read it yet, we highly recommend to read it first!

Why running Autify during code review process?

Code review is the first step of the software release process. A person proposes some changes to the code base and the other person reviews the change whether they are valid, working as expected, and won't break unnecessary functionalities, etc. There are lots of automated techniques to make code reviews easier, reliable and wider perspective, such as unit test. Unit test can cover bunch of functionalities of the software, but you also want to test some kind of end-to-end experience such as login operation by real browser. Historically, implementing such end-to-end test scenario by code is not an easy task but Autify can provide a solution here. By using Autify's intuitive recorder and scenario editor, you can create an end-to-end test scenario in 5 minutes and no need to worry about the maintenance of the scenarios because Autify can resolve and learn majority of UI changes like CSS class. Therefore, Autify can validate whether the proposed code changes can pass the same end-to-end scenario recorded in the production environment and this gives more confidence to the reviewer.

CI_CD-with-Autify-2.png

How to run Autify during code review process?

Autify needs an endpoint i.e. URL accessible from its test execution engine in cloud for running test scenarios. Therefore, you need to setup an ephemeral endpoint per Pull request on GitHub, for example, and tell Autify to replace the recorded URL e.g. production URL to the generated ephemeral URL.

How to setup such ephemeral endpoint depends on your infrastructure. For example, some deployment providers like Vercel have a preview feature and a preview endpoint will be automatically created for every pull request out-of-the-box. Or, you can setup your own deployment workflow by using Infrastructure as a Code solution and spin up an ephemeral environment for the pull request.

In summary, you will need a workflow engine that controls task dependencies, and a mechanism to generate ephemeral endpoints to the apps built based on your changes. See the table below to get more insights:

What you need Why you need? Examples
Workflow engine To kick a job to setup ephemeral endpoint GitHub Actions, GitHub App, CircleCI, etc...
(Deployment) Platform To setup and run an ephemeral endpoint Vercel, Netlify, Amazon EC2, etc...
In most cases, this setup of the ephemeral endpoint is the hardest part for you.

Because you need to think carefully about all other resources like database, microservices, 3rd partiy integrations such as Auth0, Stripe, etc. Some can be created as an isolated resource but some others can't be.

If you already have such workflows, though, you can easily hook it up and run Autify test scenarios against the ephemeral endpoint. In short, the steps are like below:

  1. Fetch the ephemeral endpoint in your CI/CD solution when a code review is created/updated
  2. Run Autify tests to the ephemeral endpoint
  3. Feedback to the code review

In the following example, we will demonstrate this with GitHub Actions and Vercel.

Example setup: GitHub and Vercel

In this example, we're going to use GitHub as a source code repository and Vercel as an ephemeral deployment environment but you can still get some idea if you use different solutions.

Vercel's preview URL is an ephemeral endpoint

Once you connect your GitHub repository to Vercel, Vercel automatically executes preview deployment on each push to any pull request and generates a unique preview URL out-of-the-box. This is a good example of the ephemeral endpoint describe above and Vercel bot notifies the preview URL to you via GitHub issue (pull request) comment. See Vercel's document for more details.

GitHub Actions can be used as the workflow engine

GitHub Actions is a workflow triggered by various events on GitHub such as code push and it runs pre-defined jobs on the runners. There is a GitHub event when a comment on a GitHub issue is created/edited/etc. so that we can trigger a GitHub Actions workflow whenever the Vercel bot writes a comment on your pull request e.g. preview deployment is done and run any arbitrary jobs like starting Autify tests and feedback the result URL as a comment on the same pull request.

How to setup GitHub Actions to integrate with pull request?

Since Vercel's preview URL is generated without any other action, we don't have to worry about how to setup the ephemeral endpoint alerted above. We just need to implement the steps 1-3 above. Here is an example setup of GitHub Actions on the repository you deploy to Vercel.

Put the file below at .github/workflows/issue_comment.yml under your repository:

This is just an example to demonstrate the idea of the integration with code review process.

on:
  issue_comment:
    types: [created, edited]

jobs:
  vercel_commented:
    runs-on: ubuntu-latest
    if: ${{ github.event.issue.pull_request && github.event.comment.user.login == 'vercel[bot]' }}
    steps:
      - name: Get preview URL
        id: get-url
        run: |
          body=$(cat <<EOS
          ${{ github.event.comment.body }}
          EOS
          )
          url=$(echo -e "$body" | grep -Eo 'https://[^\/]+\.vercel\.app') || true
          echo ::set-output name=result::"$url"

      - name: Run test on Autify for Web
        id: run
        if: ${{ startsWith(steps.get-url.outputs.result, 'http') }}
        uses: autifyhq/actions-web-test-run@v1
        with:
          access-token: ${{ secrets.AUTIFY_WEB_ACCESS_TOKEN }}
          autify-test-url: https://app.autify.com/projects/00/scenarios/000
          url-replacements: YOUR_RECORDED_VERCEL_URL=${{ steps.get-url.outputs.result }}
        continue-on-error: true

      - name: Comment Autify test result URL on the pull request
        if: ${{ steps.run.conclusion == 'success' }}
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          number: ${{ github.event.issue.number }}
          message: |
            Autify test was started. Check ${{ steps.run.outputs.result-url }}
            ```
            ${{ steps.run.outputs.log }}
            ```

      - name: Comment failed to start Autify test on the pull request
        if: ${{ steps.run.conclusion == 'failure' }}
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          number: ${{ github.event.issue.number }}
          message: |
            Autify test was failed to start.
            ```
            ${{ steps.run.outputs.log }}
            ```

YOUR_RECORDED_VERCEL_URL in "Run test on Autify for Web" step must be replaced by your Vercel application URL e.g. https://***.vercel.app that was used when you record the target test scenario on Autify. This url-replacements option tells Autify to replace that URL to the ephemeral endpoint when executing the test scenario.

If you'd like to know more about our CI/CD integrations like GitHub Actions integrations above, see CI/CD with Autify.

Once you create a pull request, Vercel bot will comment on the pull request once the preview deployment is done like below:

Screen Shot 2022-08-20 at 11.40.21 PM.png

Then, the GitHub Actions workflow you created above will be triggered, then it will start the Autify test scenario you specified and comment the Autify test result URL and log like below:

Screen Shot 2022-08-20 at 11.51.05 PM.png

Now, the reviewer (and of course the requester) can check the test result by clicking the link easily!

This example intentionally just starts the Autify test but not waits for the test result inside the GitHub Actions workflow.

Depending on the length of the scenarios, your Autify test execution might take longer time like one hour. If you wait for the test result and feedback whether it succeeded to the comment or GitHub Checks, you need to keep running the "Run test on Autify for Web" step longer like one hour and it might consume your GitHub Actions hosted runner's minutes for one hour if the repository is private. In other word, it could cost much higher. Be aware of that cost if you want to do so.

Conclusion

This document describes the benefit and the concept of Autify integration with code review process and shows an example to run Autify test against Vercel preview URL using GitHub Actions. We hope it would help you to implement your Autify integration with code review process.


First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.