Blackfire’s unique Way of Delivering Python binaries

We may not always know what is happening behind the scenes when we add a dependency to our projects. We expect this process to “just work” to carry on building applications.

Little we may know about the level of commitment from maintainers and the amount of work needed to have everyone seamlessly use a library. Let’s hop in backstage to discover the unique way Blackfire releases its Python binaries.

Blackfire provides a leading Continuous Observability tool helping its customers understand their application behavior and take actions to improve their performance. Blackfire is now part of Platform.sh and powers the unique Observability Suite available to Enterprise and Elite customers.

When a Python Package Manager requests a dependency, it checks if a binary is available for the user’s system and architecture. If not, it downloads the source code and compiles it locally.

This last part is problematic for tools such as Blackfire, which code is understandably private. As there cannot be a fallback with source code compilation, there is no other option than to provide a binary for every possible environment.

There are countless combinations of systems and architectures. Even the most famous and widely used packages offer a relatively limited number of binaries. Therefore, Blackfire providing almost universal coverage of binaries is unique in the industry. If not 100% of the possibilities are covered, which would have been an impossible effort, all Blackfire’s Python customers have access to a binary tailored to their architecture.

The deployment pipeline is triggered by GitHub and powered by Buildkite and Appveyor. There are three major platforms/architectures that Blackfire has binary support on:

• Linux (x86_64)
• macOS (x86_64 + aarch64 for M1)
• Windows (x86_64 and x86 32bits)

The manylinux project is used to distribute Linux binaries. manylinux relies on the fact that most distributions are taking great care of backward compatibilities. It provides a container image that runs on an old Linux distribution (e.g.: CentOS 5.0) and contains multiple Python versions. The C extension is then compiled with an old version of glibc, hoping that newer versions would not have any trouble running this binary as, indeed, glibc has strong backward compatibility.

Yet, the road toward universal coverage hasn’t been an easy one, even with manylinux. One major issue encountered was with EOL Python versions like 2.7. The latest image from manylinux was constantly changing and dropping support for these older versions. One workaround is to use explicit tags that include the Python version we want to keep providing a new version of the probe, even for the legacy version of Python.

Alpine was another hassle to overcome as pip had no native support for this platform. Blackfire conveyed an approach similar to manylinux to distribute the binaries manually. The wheels are compiled on an old Alpine image then ultimately uploaded to a public S3 bucket.

Fortunately, this would no longer be a hassle. PEP 656 now defines the necessary steps to distribute Alpine binaries. We have switched over to this recently in version 1.11.0

For macOS, we have our own Buildkite agents on both x86 and M1 versions. We installed target Python versions using pyenv and provided the necessary wheels to the release pipeline.

And finally, for Windows, we used Appveyor. Appveyor comes with pre-installed compiler toolchains and SDKs that helped the compilation/generation of the binaries. All the generated binaries are checked for potential errors via twine before being uploaded to PyPI.

Finally, after running all the tests in CI, generating and uploading all the binaries usually takes around 2-3 minutes. A blazing fast process for such comprehensive coverage.

This may come as a surprise, but this achievement is due to the tremendous work of a single engineer. Sümer Cip is a Cloud Engineer in the Blackfire team, who is the brain of Blackfire Python support, has singlehandedly designed the Python probes while carefully assembling the pieces of an automated build and delivery process.

Sümer Cip started his journey in code early when, at 15, he discovered Delphi. Then Open-Source changed his life. He felt its world-shifting power when strangers contributed to his first projects, teaching him new things along the way. At this point, he knew the path he wanted to take.

While working for world-class tech companies, he created Yappi, a project with a straightforward name: Yet Another Python Profiler. That’s when his path crossed Fabien Potencier’s one. Fabien is the founder of Symfony, Blackfire, and the CPO of Platform.sh. At that time, Fabien was looking for someone to create the Python Probe of Blackfire. A challenging work tailored for Sümer and at the crossroads of closed and open source.

The journey of Sümer Cip echoes the ones of quite some platformers. It is exhilarating to take up unique technological challenges and bring value to our customers by simplifying their lives. Selflessly contributing to a greater good at the same time by playing a part in the languages/frameworks/tools a whole industry is using goes far beyond. It’s bigger than oneself. It’s about making the world a slightly better place, one commit at a time.