Admiral: How to keep your fleet afloat

So far in our mini-series we've talked about the new Source Operations feature, and how it makes keeping Platform.sh projects up to date. In part two, we talked about managing fleets of sites through the Platform.sh API. As hinted at the end of the last post, it's also entirely feasible with current tools to build a custom dashboard to make managing hundreds or thousands of sites even easier. Today, we'd like to dig into that approach a bit more.

We're happy to share our latest project, Admiral. Admiral is a reference implementation and proof of concept for commanding fleets of sites on Platform.sh. To be clear, it’s not a complete application; it’s not a supported product. Admiral is a toolkit and demonstration to help you build your own fleet management tools, released under a permissive Free Software license (MIT).

Admiral's goals are to demonstrate how to build a custom fleet management tool for your organization and to give you a point to start from. In its current form, Admiral lacks some important details that you probably want to include, like user authentication. But it does demonstrate the full lifecycle of creating and updating Platform.sh projects en masse. As configured, Admiral will run almost out of the box on Platform.sh, but it could also run anywhere else you want. All it needs is PHP 7.3 and MySQL/MariaDB.

The project's README file covers installation and requirements, so we won't go into that here. Instead, I want to go over the architecture and call out the key points to bear in mind while building atop it, or in replicating it in your own preferred language/framework/toolchain.

Admiral is written in Symfony 4 with the EasyAdminBundle. That combination was chosen mainly due to our own familiarity with it and the ready availability of RAD tools in the ecosystem. But the same functionality could be implemented in any other toolchain. We also offer a PHP client library that abstracts over the Platform.sh API. A JavaScript version (for either client-side or Node.js) is also available. It's not yet available in other languages, but certainly could be written.

A brief tour of Admiral, our fleet management tooling example project

Admiral is built on the model of one Platform.sh user that manages some large number of individual sites, and those sites are all built off of a small number of common templates. Admiral calls those templates "Archetypes," and you can create those through the UI:

The most important part of the Archetype is the Git repository from which we spawn new projects. The Update branch is the branch in each project to use for updating, while the Update operation is the source operation to call; it can be anything, but it’s only useful if it's the name of the defined source operation in the Archetype (and thus, in every project) that runs a code update task. What that task looks like is largely up to you, but there are some common patterns.

Admiral can support any number of Archetypes:

The UI is stock EasyAdminBundle and is fairly easy to configure.

The other object that Admiral tracks is Projects, which are simply stubs that correspond to a project on Platform.sh:

Admiral allows a project to be created in any Platform.sh region, although your own implementation could easily hard-code a specific region, if desired. Projects also specify their Archetype. After creation, only the title is editable.

When a Project record is created, Admiral fires off API calls to Platform.sh to create a project, then associates the resulting project ID with the Project record in Admiral. After that, the Project is really just a stub. Upon creation, Admiral also populates the new project with code from the Archetype and sets some project-level variables from the Archetype. More on that in a moment.

Once it's been created, the Project page now shows information pulled from the Project API on Platform.sh. We're showing just a few fields to demonstrate that you can do so, but anything available through the API can be listed here quite easily:

More importantly, all projects can be listed and acted upon:

The Project list allows the user to trigger various actions on Projects, either individually:

Or in bulk:

Although we're not using it, EasyAdminBundle also has support for more advanced filtering and sorting to help manage multiple sites.

Admiral includes what we consider the basics for fleet management, but the whole point of having a flexible API is that you can build whatever actions you want. For instance, Admiral doesn't have functionality to change the plan size for a project from Development to a production-level plan (since that can be done from the individual project page on Platform.sh already), but there's no reason it couldn't do that.

Website fleet management actions that Admiral supports (and you’ll want, too)

At the moment, the following actions are implemented. If you're building your own fleet management tool, these are (approximately) the primitives you’d want to support.

Backup

This is the most basic action, which triggers a "Backup" command on Platform.sh for the master branch. Generally speaking, using scheduled backups is a better solution, but if you want to trigger a backup of many projects manually, this is a way to do so.

Delete

When a Project object is deleted in Admiral, we hook into the delete action to fire the same delete command at the project on Platform.sh. This is a destructive operation. Please use with care.

Update

Update is the fun one. The Update command first checks if a project has a branch named for the "Update branch" on its Archetype. If so, it runs a Sync command to ensure it has the latest code and data from the master environment. If not, it creates one (and then the Sync is implied).

In either case, once the named update branch is confirmed created and has the latest code and data from the production environment, Admiral makes the API call to run the source operation specified by the Archetype.

That source operation can do any number of things. The most common options would be to either:

• Run a local package update command (composer update, npm update, bundle update, etc.) and commit the results.
• Add a Git remote based on the Archetype's Git repository and merge its master branch into the local code, which also results in a commit.

Both options have their use cases, as well as their fussy edge cases. Admiral itself doesn't care; it just triggers a source operation and calls it a day.

Once any resulting changes are done deploying, you can visit that branch and test it out, just like any other dev branch on Platform.sh.

Merge update

Another simple action, Merge first triggers a "backup" command on production, then triggers a "merge" API call at the update environment, causing it to get merged to its parent (presumably master/production). That, in turn, triggers a rebuild/redeploy of master on Platform.sh, and the updates are now live.

All of these commands may be triggered for one or a hundred Projects at a time, making it easy to say "well, there was a security release, I guess it's time to update everything." Just "Select All," "Update," wait for them to finish, review, "Select All,", Merge, get $beverage.

What you’ll want to remember when building a website fleet management tool like Admiral

There are a few important notes for anyone trying to build or extend a system like this.

Creation

First is the question of how to initially populate the project repository on creation. There are broadly two options, and which option you want to use depends on your use case. Admiral includes both, but as it ships it hard codes to use the second. (Switching is a matter of swapping a commented line.)

• Use the initialize API call. This approach works with any public Git repository and is a simple one-line command. It's the same API call our Management Console uses when you select a project template. With this approach, the project is populated with just a single commit of whatever code is at the HEAD of the master branch in the Archetype repository. This approach is simple and straightforward, but results in a separate Git history for every project. If you’re updating projects locally that's fine, but if you then want to merge code from the Archetype repository later that becomes problematic and you'll need to use the -X theirs flag for the git merge command to avoid merge conflicts.

• Manually git push a copy of the Archetype's repository to the project. This approach results in a common Git history for all projects, which makes merges easier. On the flip side, it's also considerably more involved and requires setting up a public/private SSH keypair for Git to use in order to write to the Platform.sh repository. If you're planning to merge from the Archetype repository, then this is the recommended approach. If not, it's most likely more complexity than you need.

Synchronization

The second key question is around synchronization. Most API calls on Platform.sh are asynchronous. That is, they return immediately to the caller and then continue in the background, and you can issue more API calls that will queue up behind it and run when they're able. There are two important caveats, however:

1. On project creation, Admiral needs to wait for the project to be created so that it can get the new project ID and record that in its own database. That process may take anywhere from 20 seconds to 2 minutes. Currently, Admiral simply blocks until that task is complete and then finishes validating and writing the Project record. If creating projects one at a time that should be entirely adequate. If you need to mass-create a series of projects, however, a more robust and queued approach is recommended.

2. Some API commands are only available if a project is in a given state at the time the API call is issued; it can’t tell that the project will be in that state once other queued tasks are run. For example, attempting to branch from an environment that isn't fully initialized yet will fail, even if the environment is in the process of initializing at the time. The solution here is, again, to block until the operation becomes available—if you’re certain that operation will become available. (Remember to have a timeout!)

That need to block at times, combined with the simple fact that even asynchronous API calls require some network time, introduces a scaling problem. When running an update against two or three projects, it may be acceptable to wait 10 or 20 seconds for a page to respond. When running an update against 200 projects, not so much.

For that reason, we recommend moving everything but project creation off of the main request thread to a background process. The way to do so will vary depending on your platform.

Admiral is written in PHP, so it uses Symfony's MessageBus for all actions other than project creation (which must be synchronous, as above), as it can be shifted to a background queue worker very easily. In fact, as configured out of the box, it uses a database-backed queue and a worker instance to do exactly that, which frees up the main process to respond to the user quickly. If executing an action across 200 projects, Admiral triggers 200 command messages, which all go into the queue and will be processed in the order they are received.

Languages with more built-in asynchronous functions can use other approaches. A Go implementation, for instance, could invoke a distinct goroutine for each action and project.

Processing actions in the background also introduces a question around error handling. If the UI doesn't block on each action, how is the user notified if it did or did not work? There's no one right answer here. For Admiral, we took the approach of just logging the failure in Symfony's standard logging toolchain and silently failing. Because the activity log is shown on each Project page most failures would be reported there, just as they would be on the project itself on Platform.sh. A more robust approach would use a webhook to subscribe to the activity stream for each managed project and surface any errors to the user.

Managing your own website fleet at scale

This application is, as noted, only an example. There’s no single right way to manage fleets on Platform.sh; the right way depends on what workflow you want to have.

• If you're managing a large number of similar sites, then taking Admiral and adding some user management on it would probably suit you fine (and you’re welcome to do so)
• If you want to build a complete and transparent SaaS on top of Platform.sh you could use Admiral as a basis or build your own workflow into your existing ecommerce system.
• You can also structure your Source Operations in any number of ways. Technically, you don't even have to use them, although they are, in practice, the best way to handle code updates across a large number of projects.

Sounds exciting, right? But you may not be sure what the right approach is for you. No worries. Contact our team and we'll be happy to help you design the workflow that best suits your organization and keeps your fleet afloat.