One codebase, many projects: Drupal multisite fleet management

For years, Chromatic managed virtual servers on behalf of our clients. As tools like Platform.sh matured, our team realized we were spending much of our clients’ budgets on simply maintaining those servers—instead of making their sites better (which is kind of our reason for existing). 

For one of these clients, the costs of maintaining their fleet of sites/servers were really adding up. So, we decided to move their entire Drupal multisite application over to Platform.sh. On its face, this is a pretty straightforward task. But the wrinkle in this case? It’s a single codebase that powers nearly 20 sites today, with more being added each quarter; each site requires its own, siloed production environment, with slightly different resources and configurations. 

When we think about Drupal multisites, we often envision a single codebase running multiple sites in a single production environment. But in this case the environments for each of the 20 sites have always been completely siloed. So while it’s multisite from a codebase perspective, from an infrastructure perspective, these are a fleet of independent sites/projects.

When moving this codebase to Platform.sh, we had to determine how to restructure our deployments to maintain the benefits of a single codebase—without sacrificing siloed production environments. After a fair bit of discussion, we determined that the best solution was to retain our single canonical repository, but to create an individual Platform.sh project for each site. While Platform.sh source integrations work well for keeping a repository in GitHub, GitLab, or BitBucket in sync with the a single Platform.sh project, this approach wouldn’t work for our implementation for one main reason: while the application code is identical between our sites, our Platform.sh configuration files would require slight differences for different disk size requirements or custom web server rules. As it stands in August 2022, the locations of these files are quite specific, and there isn't a way to provide a different set for each site and then have Platform.sh dynamically know which one to use.

Here are our requirements:

• One source codebase/repository in GitHub
• Separate hosting environments for each site, or individual Platform.sh projects
• Platform.sh configuration files that differ slightly for each site

Let's build artifacts!

We decided to create a deployment artifact for each site. Instead of directly connecting the source repo to the Platform.sh project repos, we would generate an artifact for each site on merge, then commit/push that to the corresponding Platform.sh project repository. In practice, the artifact is very similar to the source—if we’re looking at the number of lines of code changed. But to Platform.sh, they are some very important lines.

Using Robo to generate the artifact

At Chromatic, we use the Robo task runner extensively (see: Usher, one of our custom tools) across all of our projects, and this project was no exception. Here, we created a command that accepts a site name argument, then generates a build artifact for the specified site. The steps involved:

1. Replace a unique placeholder string in .platform.app.yaml (ex. sitename-w4BzzMcc), with the site name argument that was specified with the call to generate the artifact.
2. Insert custom web headers and other per-site configurations into .platform.app.yaml. These rules are stored in a YAML file and keyed with the same site name used above.

sitename:
  web_server_headers:
    default-src: self
    connect-src: 'self https://www.google-analytics.com https://www.googletagmanager.com'

3. Check if an overridden sitename.routes.yaml file exists. If so, copy it into place (/.platform/routes.yaml) instead of our default version.

Sample .platform.app.yaml
# Additional configuration not directly relevant to this topic has been removed
# for clarity.
---
name: sitename-w4BzzMcc
type: 'php:8.1'
runtime:
  extensions:
    - redis
variables:
  php:
    memory_limit: 256M
    date.timezone: "UTC"
relationships:
  database: 'db:mysql'
  redis: 'cacheredis:redis'
disk: 3072
mounts:
  '/web/sites/sitename-w4BzzMcc/files': 'shared:files/files'
  '/tmp': 'shared:files/tmp'
  '/private': 'shared:files/private'
build:
  flavor: composer
hooks:
  build: |
    composer robo theme:build sitename-w4BzzMcc
  deploy: |
    composer robo deploy:drupal $PLATFORM_APP_DIR sitename-w4BzzMcc
web:
  locations:
    '/sites/sitename-w4BzzMcc/files':
      allow: true
      expires: 5m
      passthru: '/index.php'
      root: 'web/sites/sitename-w4BzzMcc/files'
      scripts: false
      rules:
        '^/sites/sitename-w4BzzMcc/files/(css|js)':
          expires: 2w

Pushing the artifact

We do a great deal of automation with GitHub Action workflows, so using them here was a natural fit. When code is merged to our default branch in GitHub, a matrix workflow job is triggered for each version of the site that does the following to deploy the changes to each Platform.sh project automatically:

1. Install dependencies.
2. Generate the artifact as detailed above.
3. Clone the project artifact repo for the given site from Platform.sh.
4. Move our changes into the project repo working directory via rsync.
5. Commit and push the changes back to the project repo at Platform.sh (thus deploying production in our case).

A simplified example of this workflow:

---
name: 'Platform.sh Deploy Artifact'
on:
  push:
    branches:
      - main
env:
  ARTIFACT_REPO_PATH: /tmp/artifact-repo
jobs:
  psh-deploy-artifact:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        site:
          - project1
          - project2
          - project3
        include:
          - site: project1
            clone_url: REDACTED.git
          - site: project2
            clone_url: REDACTED.git
          - site: project3
            clone_url: REDACTED.git
    steps:
      # We have omitted the usual operational steps necessary to checkout the
      # repo, set up PHP, install dependencies etc.
 
      - name: 'Apply ${{ matrix.site }} PSH config overrides'
        run: composer robo artifact:generate ${{ matrix.site }}
 
      - name: Checkout branch in artifact repo to match source repo
        working-directory: ${{ env.ARTIFACT_REPO_PATH }}
        run: git checkout ${{ env.GITHUB_REF_SLUG }} || git checkout -b ${{ env.GITHUB_REF_SLUG }}
 
      - name: 'Rsync our GH repo with modified/overridden config for ${{ matrix.site }}'
        run: rsync -zirhl --stats --delete --exclude='/.git' --filter="dir-merge,- .gitignore" "$GITHUB_WORKSPACE/" ${{ env.ARTIFACT_REPO_PATH }}
 
      - name: Commit and push PSH repo back to PSH as newly built artifact
        working-directory: '${{ env.ARTIFACT_REPO_PATH }}'
        run: |
          if [[ $(git status --porcelain) ]]; then
            echo "Changes found. Making git commit."
            git config --local user.email "github-actions[bot]@users.noreply.github.com"
            git config --local user.name "github-actions[bot]"
            git add -A
            git commit -m "Artifact built from $GITHUB_SHA by GitHub Action workflow."
          else
            echo "No changes found. Skipping commit."
          fi
          git push -u origin "$GITHUB_REF_SLUG"

Wrap-up

Our approach has enabled us to retain all of the benefits of a single codebase powering dozens of sites and all of the advantages Platform.sh grants us. When we’re pushing out new features on the application or simply updating Drupal core, we have an efficient, repeatable, and automated deployment. It saves us and our clients time and money and, ultimately, lets us deliver more value.

Interested in more technical insights from the Chromatic team? Explore their blog.

Mark Dorison is Chromatic’s chief technical officer and partner. Away from Chromatic, Mark is an avid traveler, cyclist, amateur TV critic, and splits a game design studio 50/50 with his daughter Kai.

Feel free to reach out to him via Twitter.