Measuring the success of a deployment with GitHub actions

Chad Carlson
Chad Carlson
Manager, Developer Relations
23 Dec 2021

Last week, we took a look at how shareable activities - specifically the link made available to a particular activity’s logs in the management console - could be surfaced as a part of a GitHub integration using GitHub actions. It was a nice bit of fun playing around with GitHub actions for the first time, and it let me make something new and useful for my team’s work.

But I caught a little bit of the curiosity bug. What else could I link up to my workflow, now that I can trigger events in response to a Platform.sh environment deployment? Well, since I linked an activity’s logs to a failed deployment, it only makes sense to trigger some things in response to a successful one.

Reacting to deployment states

Take a moment to check out the previous post centered around GitHub actions. It’ll provide a little context, plus go into much more detail about how the status of a Platform.sh deployment can actually be used to trigger workflows on GitHub. Plus, we’re going to build additional jobs in the same post-deploy.yaml file we started in that post.

Let’s take a look at the file, ignoring the displaylogs job we made previously, and instead focusing on the build job that sets everything else up:

---
name: Post-deploy
on:
  push:
    branches-ignore:
      - 'master'
env:
    PLATFORMSH_CLI_TOKEN: ${{ secrets.CLI_TOKEN }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

jobs:
    build:
        runs-on: ubuntu-latest
        name: 'Get environment URL'
        outputs:
            commit_status: ${{ steps.status.outputs.env_status }}
            env_url: ${{ steps.url.outputs.env_url }}  
            integration_status: ${{ steps.wait.outputs.integration }}  
        steps: 
            - uses: actions/checkout@v2
            - name: 'Await deployment'
              id: wait
              run: |
                COMMIT_STATUS="pending"
                sleep 10
                STATUSES=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r 'length')
                if [ $STATUSES == 0 ]; then
                    echo "Not on a Platform.sh integrated environment. Skipping."
                    echo "::set-output name=integration::none"
                else
                    until [ "$COMMIT_STATUS" == "success" ] || [ "$COMMIT_STATUS" == "failure" ]; do
                      sleep 10
                      ENV_URL=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r '.[0].target_url')
                      COMMIT_STATUS=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r '.[0].state')
                    done
                    echo "Environment deployed. Finished."
                    echo "::set-output name=integration::platformsh"
                fi
            - name: 'Pass status'
              id: status
              if: steps.wait.outputs.integration == 'platformsh'
              run: |
                COMMIT_STATUS=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r '.[0].state')
                echo "::set-output name=env_status::$COMMIT_STATUS"
            - name: 'Pass URL'
              id: url
              if: steps.wait.outputs.integration == 'platformsh'
              run: |
                ENV_URL=$(curl -s https://api.github.com/repos/$GITHUB_REPOSITORY/statuses/$GITHUB_SHA  | jq -r '.[0].target_url')
                echo "::set-output name=env_url::$ENV_URL"

This job (build):

  • Is triggered by all push events that don’t occur on the default branch.
  • Places requests on the Status API, until Platform.sh informs GitHub that the environment has either succeeded or failed to deploy the new commits.
  • Passes both that final status and the environment’s URL to any subsequent jobs that define needs.build in their configuration.

Checking simple statuses

One of the great things about GitHub actions is that I don’t have to write them. If I, for example, wanted to make a quick status check on the environment, I can leverage the generous work of another published action in my workflow pretty easily.

   test:
        name: 'Status tests'
        runs-on: ubuntu-latest
        needs: build
        if: needs.build.outputs.commit_status == 'success'
        steps:
            - name: 'Application should deploy successfully'
              uses: lakuapik/gh-actions-http-status@v1
              with:
                sites: '["${{ needs.build.outputs.env_url }}"]'
                expected: '[200]'

In this case, I’ve written a new job called test which will:

  • Only run if the output of the build job (and the status of my Platform.sh environment) is successful.
  • Will use the lakuapi/gh-actions-http-status public action to place a request on the environment URL also retrieved from the build job.
  • Apply an expected status to that request. If the statuses match, the step will pass, adding another check to my pull request. Should it receive a 404 or 503, the step fails and we can go investigate where the deployment fell short.

This snippet only covers the homepage of the development environment, but I could also include more endpoints in the sites array, with matching expected statuses that will show up as separate steps in my final workflow for the push.

The Lighthouse

Status checks are great, but what I really want to know is whether a pull request is modifying the experience a user has when they visit my application in some negative way. Once again, I can add a job that uses a publicly available GitHub action - in this case, to perform a Lighthouse audit on my environment URL.

To start, we can add a few workflow-level environment variables to post-deploy.yaml

env:
    PLATFORMSH_CLI_TOKEN: ${{ secrets.CLI_TOKEN }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    ACC_MIN: 0.90
    BPR_MIN: 0.90
    PRF_MIN: 0.90
    SEO_MIN: 0.95

These variables are going to set a few thresholds for the four main audits we’d want to keep high on our homepage: Accessibility, Best Practices, Performance, and SEO.

Here’s what the Lighthouse report looks like for our Platform.sh homepage:

Success GitHub Status

Not too shabby!

Let’s add a new job called audit which uses the jakejarvis/lighthouse-action GitHub action, first focusing on the Accessibility audit.

  audit:
        name: 'Lighthouse Audit'
        runs-on: ubuntu-latest
        needs: build
        if: needs.build.outputs.commit_status == 'success'
        steps:
            - name: Audit live URL
              uses: jakejarvis/lighthouse-action@master
              with:
                url: ${{ needs.build.outputs.env_url }}
            - name: Upload results as an artifact
              uses: actions/upload-artifact@master
              with:
                name: report
                path: './report'
            - name: Accessibility test
              run: |
                  REPORT=$(ls report | grep \.json$)
                  SCORE=$(cat report/$REPORT | jq -r '.categories.accessibility.score')
                  if [ 1 -eq "$(echo "${SCORE} < ${{ env.ACC_MIN }}" | bc)" ]; then
                      echo "Threshold Failed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.ACC_MIN }}"
                      exit 1
                  else
                      echo "Passed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.ACC_MIN }}"
                  fi

This audit job will:

  • Only run on pushes to a branch associated with a pull request integrated with a Platform.sh environment that has successfully deployed. (See build above)
  • Use the jakejarvis/lighthouse-action action with the environment URL passed along from the build job.
  • Upload the full report as a workflow artifact, so that we can inspect individual recommendations in detail later.
  • The action will also save a local copy of the report in two formats - html and JSON - in a report subdirectory. We can then use the jq tool to retrieve the accessibility score.
  • Run a simple test based on the Accessibility audit using the ACC_MIN threshold we set previously. In this case, if the accessibility score for our environment is below 90%, the step and workflow will fail for the push.

Using this same logic we can add tests for Best Practices:

            - name: Best practices test
              run: |
                  REPORT=$(ls report | grep \.json$)
                  SCORE=$(cat report/$REPORT | jq -r '.categories."best-practices".score')
                  if [ 1 -eq "$(echo "${SCORE} < ${{ env.BPR_MIN }}" | bc)" ]; then
                      echo "Threshold Failed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.BPR_MIN }}"
                      exit 1
                  else
                      echo "Passed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.BPR_MIN }}"
                  fi

Performance

            - name: Performance test
              run: |
                  REPORT=$(ls report | grep \.json$)
                  SCORE=$(cat report/$REPORT | jq -r '.categories.performance.score')
                  if [ 1 -eq "$(echo "${SCORE} < ${{ env.PRF_MIN }}" | bc)" ]; then
                      echo "Threshold Failed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.PRF_MIN }}"
                      exit 1
                  else
                      echo "Passed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.PRF_MIN }}"
                  fi

and SEO.

    - name: SEO test
              run: |
                  REPORT=$(ls report | grep \.json$)
                  SCORE=$(cat report/$REPORT | jq -r '.categories.seo.score')
                  if [ 1 -eq "$(echo "${SCORE} < ${{ env.SEO_MIN }}" | bc)" ]; then
                      echo "Threshold Failed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.SEO_MIN }}"
                      echo "Download the full report under *Artifacts* at: https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
                      exit 1
                  else
                      echo "Passed:"
                      echo "  - Score:     $SCORE"
                      echo "  - Threshold: ${{ env.SEO_MIN }}"
                      echo "Download the full report under *Artifacts* at: https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
                  fi

There’s two more things down that can be connected to Platform.sh deployments using GitHub actions. You can find the full Gist for the workflow described above on GitHub.

Goodbye for now - until I can think of something else interesting to play with here. If you have a custom workflow, or if you’d like to share tips and tricks of your setup, let us know by sharing your actions with the rest of our Community.