Build a real-time SaaS with an open-source Drupal core

Open Social provides online community solutions that enable organizations to create engaging platforms, connecting members through rich social experiences. Built on top of Drupal, we use an open-core business model in which our core features are completely open to the public, but additional support and features become available only to our Software-as-a-Service (SaaS) customers.

The model gives us the ability to gather input about new features from customers and the broader open-source community. Those features are controllable, as we’ve discussed previously, through the Platform.sh API, and can be dispersed quickly across all of our communities. With the continued shift to online community spaces, we heard our customers express the need for more dynamic features on their platforms. One of our first major planned changes in this vein has been to introduce a Real-time Chat extension that will enable community members to talk with one another simply by toggling an on-page chat window. Members will be able to easily jump in and out of a conversation—without navigating away from their current activities.

Implementing this new, real-time feature in Drupal posed a significant challenge. Traditionally, Drupal is built for synchronous calls rather than the highly dynamic, asynchronous messaging a chat application requires.

Our discussion led us to explore new tools that would best complement Open Social’s continued use of Drupal, putting our fleet of communities on Platform.sh to the test. Having only recently started rolling out this new feature, we thought we’d share how our conversations about Real-time Chat development unfolded.

The challenge: real time and Drupal

Drupal is often referred to as a content management framework (CMF) or a digital experience platform (DXP), with the focus of the Drupal project having evolved over the years to meet the needs of its growing community. Today, Drupal can even serve as an API for those who want to decouple and offers an expanded out-of-box experience, with a more robust module integration system. Labeling this well-developed module ecosystem just a content management system really sells it short.

So much of the Open Social product relies on building engaging extensions on top of this system, so we had to find an innovative way to develop and launch Real-time Chat. In the project’s early stages, we had to choose how we were going to build out an API to support the feature: JSON:API or GraphQL. Both have plenty of module support, but we’d decided that the GraphQL Subscription model was where we wanted to develop our chat feature. That said, its limitations became apparent, too.

Although Drupal handles GraphQL requests for simple queries and mutations beautifully, it doesn’t deal well with subscriptions—the main driver of real-time GraphQL function. Subscriptions require keeping many connections open at once and interleaving work for each of them as data becomes available. But this highly asynchronous process becomes bottlenecked completely by Drupal’s synchronous communication with its database.

Making chat run on Platform.sh

With this asynchronous limitation in mind, we knew we’d have to build something outside of Drupal entirely. Something that could access its data, while still handling the asynchronous nature of GraphQL Subscriptions. We found that a similar implementation of the JavaScript event loop for PHP (event) worked nicely and could be enabled very easily on our current Platform.sh PHP containers.

Factoid

Although server-side event loops in JavaScript gained traction sooner (in the form of NodeJS), the initial release of the PHP equivalent actually predates NodeJS by some years.

We needed something, then, to build on top of the PHP event extension to handle our asynchronous requirements. Of the two main choices—Amp and ReactPHP—we decided on the more popular ReactPHP. It uses the more traditional then methods with closures and callbacks; has a community size we felt confident in tying ourselves to; and in the end, has a simpler syntax (not everyone is familiar with generators yet). ReactPHP also provides an HTTP client and server, asynchronous database clients, and clear ways to communicate with Redis and RabbitMQ. So, we could build on top of ReactPHP and use the Ratchet and Bunny libraries to create a Websocket server and to talk to RabbitMQ, respectively.

We now had a direction to connect our Drupal Open Social installation with a second real-time chat application. A message sent on Open Social is pushed out to RabbitMQ via our ReactPHP solution. Then, it’s consumed by the chat service, which can leverage the project's internal network on Platform.sh, to securely retrieve data from Open Social over GraphQL. The chat app will use Open Social’s current access data to determine the message’s destination (i.e., who is subscribed to that message and allowed to see it), finally delivering it to the client.

We still get to keep Drupal as our core, but can now extend it outside its common module method.

A lot of what informed this approach—adding an entirely new application to a community installation—is that it worked well with what we were already doing on Platform.sh. Our Multiverse tool can help us push any updates across any and all communities where we want to enable the feature; actually adding to that installation is supported completely by multiapp configuration.

From Real-time Chat to fully decoupled

Real-time Chat represents only the first step toward more widespread, real-time communication in our communities—with less overhead. The service that we created to solve this challenge can open up collaborative document editing and peer-to-peer WebRTC apps, which won’t even require GraphQL to implement.

Although we’ve only started to release Real-time Chat to our customers, it already represents an important step towards fully decoupling our community installations and providing more real-time interactions to our users in the future.

Not only does our work on Real-time Chat broaden the general Open Social API with GraphQL, it also enables us to contribute back to the Drupal community through that module in the process. In that spirit, we've open-sourced the work we've done to handle the GraphQL WebSocket connections in the form of the graphql-php-ws package. This implements the protocol of the popular graphql-ws JavaScript package—the standard for GraphQL over WebSockets, supported by all major GraphQL clients.

As we work on the decoupling process at Open Social, we’re faced with many technical challenges, such as the one described in this blog post. In researching and solving these challenges, we generate new knowledge and insights that let us push toward a fully decoupled setup, in which the responsibilities of storage and presentation are neatly separated. This approach lets us create an Open Social that’s faster, leaner, more flexible, and easier to continuously update—providing us and third-party integrators with a platform that offers exciting new possibilities for online communities.

Pandemic-inspired bodyweight athlete, poker-player, programming veteran, Alexander is a Drupal guru currently studying business administration. You can reach him via twitter or Linkedin or check out his website.