Umami distribution testing automation with Diffy and

Yuriy Gerasimov
Yuriy Gerasimov
Founder, Diffy
28 Jun 2019
Drupal 9

Deploy our Drupal 9 template for free

Deploy on

One of the big things that drives Drupal 8 adoption is an easy ability to “try it out.” That is a fantastic Umami distribution project.

Umami is part of the Drupal 8 Out of the Box (OOTB) initiative, which aims to provide a demo of Drupal’s capabilities—complete with sample content and a well-designed custom theme.

Recently, we were working on automating visual regression testing so maintainers have an easy way to check the patches.

The only thing that is needed right now is to copy patch’s URL, paste it into one of the test branches in GitHub project — and in 5 – 10 minutes, you’ll get a report with all visual changes that this patch brings. And even more — visual testing covers both anonymous users view and an admin as well.

GitHub setup

The main repository is available at

As you can see, it is fairly standard Composer-based Drupal setup.

The main logic resides in composer.json file. It uses latest Drupal 8.7 dev branch, and whenever you’d like to test the patch there, you’d simply add it in “patches” section:

"patches": {
  "drupal/core": {
    "Fix styling of Umami for layout builder": ""
} setup has provided an account for testing purposes. Thanks to out-of-the-box integration with GitHub, we have configured “tracking” on three testing branches. Once any commit lands (i.e., testing a patch), the related environment gets rebuilt. All logic resides in file.

Here are the hooks that get triggered on each deployment:

  # The deploy hook runs after your application has been deployed and started.
  deploy: |
    cd web
    drush si demo_umami -y --account-pass=nKraDzZq8ZUq747Bn9PaXAB83 --account-mail=""
    drush cron
    drush status > sites/default/files/drush-status.txt
    php ../scripts/diffy-visual-testing.php

So you can see that we:

  • Reinstall the site from scratch by using the Umami distribution
  • Run cron
  • Set up some debugging information
  • Trigger visual regression testing

Visual regression testing with Diffy

The script that triggers visual regression testing jobs does a few things:

  • It gets all the credentials from variables
  • It checks if the current branch is master or one of the test branches
  • If it is a master branch, simply create screenshots
  • If it is a test branch, create screenshots and compare them with latest screenshots from master branch environment (production in terms of Diffy)

Once jobs are completed, we send a notification email. Also, the diff gets renamed based on the branch name. So it is easy to track what environment was tested and when.

Diff report for test branch Diff report for test branch

As we are testing Umami with an authenticated user, Diffy is configured to have two separate projects: one project for an anonymous user and one for an authenticated user.

Admin authenticated user Admin authenticated user

Security and passing variables from to Diffy

There was one very valid argument raised by Mark Conroy, what about security? We do publish admin password in public GitHub repository.

The way we solved it is by setting up HTTP basic authentication in and passing credentials to our script that triggers jobs. allows passing variables to your build scripts allows passing variables to your build scripts

So, even with a public GitHub repository, only individuals with a password would be able to access our builds and login.

Other use cases

Because this setup uses Composer, it is fairly easy to use for other Drupal projects/distributions. You can test themes, modules, or any other products that have some changes in UI that you’d like to track.

Yuriy Gerasimov

Skilled in back end, front end (vuejs fan), and DevOps work, Yuriy has presented at DrupalCon and at various camps across Europe and US. He’s also a founder of Diffy—a visual regression testing tool that helps dozens of companies ship higher-quality sites with less stress.