CVE-2021-44228 Apache Log4j vulnerability

[Last updated 24 December 2021 16:32 UTC]

This is the final update, and we at Platform.sh would not let you go into your end of year holidays without ensuring that your projects are safe and secure.

Since the last update and Log4j 2.17.0 release, we have been relentlessly upgrading all our Elasticsearch, Opensearch and Solr services to include it. At this point all supported Grid services and the majority of Dedicated ones are upgraded. We are still urging you to upgrade outdated Elasticsearch 5.x and prior versions to to more recent ones.

We believe your projects are fully protected from issues found in this library, but of course, we will continue to monitor in case new threats are discovered. We will not be updating this blog post anymore, but we are always available through our Customer Support team and our public Slack channel.

[Last updated 20 December 2021 10:37 AM PST]

Late Friday, Apache released another update to Log4j version 2.17.0. The update includes a patch to a denial-of-services (DoS) vulnerability resulting from uncontrolled self-referential recursion. We’re currently working to deploy affected services with 2.17.0.

[Last updated 17 December 2021 12:09 PM PST]

As we recognize the severity of the ongoing Log4j issue, we’re continuing our efforts to address this vulnerability. Platform.sh security specialists and software engineers are working around the clock to analyze new findings and mitigate the threat. We’ll continue the cadence of posting updates here.

[Last updated 16 December 2021 7:23 AM PST]

The upgrade that includes the fixed Log4j version 2.16 is ongoing. It’s important to note that Elasticsearch versions 5.x and prior won’t be included as they’re past EOL. While we’ll still allow versions 5.x and prior, they only receive protection in the form of disabling message lookups and complete removal of the exploitable class. We urge all users with these outdated versions to upgrade to more recent versions to improve protection.

[Last updated 15 December 2021 9:04 AM PST]

We’ve added protection for CVE-2021-45046 (Log4j DoS). This threat is less critical as we’ve already patched the RCE and removed the exploitable class from the installed classpath. Now, we’re working to deploy the fully patched library from upstream with the fixes integral, rather than relying on workarounds on compromised versions. And we’re upgrading all the affected services with Log4j 2.16.

[Last updated 14 December 2021 11:42 AM PST]

The Log4j vulnerability’s impact is two-fold: first, remote code execution, and second, a generic, phone-home vulnerability. Our team has verified we were never vulnerable to remote code execution. Additionally, we’ve taken the measure to patch the phone-home vulnerability to protect our customers.

We’ve mitigated the issue on our Dedicated and Grid environments and will continue to monitor for further issues.

[Last updated 13 December 2021 7:35 PM PST]

On 9 December 2021, NIST disclosed the remote code execution (RCE) vulnerability related to Apache Log4j (CVE-2021-44228)—the logging tool used in many Java-based applications. Don’t worry, we got to work right away. Platform.sh security specialists and software engineers immediately began to analyze the threat to understand where Apache Log4j may be used.

A thorough investigation of our products and services verified that our services are protected from the worst form of remote code execution. The threat of information disclosure is reduced because we aren’t sending application variables to these services. To further reduce that threat, these services are also receiving updates to disable the undesired behavior of Log4j. We’ve received confirmation from our backend IaaS providers and from Fastly that they've either been unaffected or have mitigated any exposures. We’ll continue to update this post with the status of our mitigation efforts.

Now, what should you do? Protect your applications by following our mitigation recommendations.

Steps you should take to further increase protection

Permanent mitigation

Platform.sh recommends customers apply the latest security updates to remediate this vulnerability. Please review the Apache CVE and the Apache security advisory for further details.

All systems, including those that aren’t customer-facing, are potentially vulnerable to this exploit, so backend systems and microservices should also be upgraded. If your project is running custom Java code in an app container, your developers should review your code for Log4j 2 usage and update it to the latest release (2.15), which should be readily available on Maven Central. A service restart will be required.

The Platform.sh team has reviewed, tested, and is rolling out mitigation patches for services like Elasticsearch and Solr. We have no evidence at the moment that an RCE was possible in our combination of services. If you run your own Java code on Platform.sh, you should investigate and update Log4j 2.

Temporary mitigation

If you're running a Java app on Platform.sh, consider the following steps to mitigate the risk of this vulnerability until a more complete security update can be applied. A service restart will be required for these changes to take effect. While these workarounds will help mitigate risk, they should not be considered a complete solution to resolve this vulnerability.

• In case the Log4j 2 vulnerable component cannot be updated in Log4i 2 versions 2.10 to 2.14.1, ensure -Dlog4j2.formatMsgNoLookups=true. is configured in the startup scripts of the Java Virtual Machine.

• Alternatively, customers using Log4j 2.10 to 2.14.1 may set the LOG4J_FORMAT_MSG_NO_LOOKUPS=”true” environment variable to force this change.

• For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath:

  zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

What is Log4j?

The vulnerability—tracked as CVE-2021-44228 and referred to as Log4Shell RCE 0-day exploit—affects Java-based applications that use Log4j 2 versions 2.0 through 2.14.1. A Java-based logging library, Log4j 2 is widely used in development, included in various open-source libraries, and directly embedded in major software applications.

The scope of impact has expanded to thousands of products and devices, including Apache products like Elasticsearch, Solr, and OpenSearch, among many others. The cross-platform nature of Java means the vulnerability is exploitable on many platforms, including both Windows and Linux. As many Java-based applications can leverage Log4j 2, organizations should contact application vendors or ensure their Java applications are running the latest version. Developers using Log4j 2 should ensure they’re incorporating the latest version of Log4j into their applications as soon as possible in order to protect users and organizations.

How the exploit works

The vulnerability is a remote code execution vulnerability that can allow an unauthenticated attacker to gain complete access to a target system; it’s also an information disclosure vulnerability that can leak content of environment variables. The former is mitigated by recent Java Virtual Machines already in use by our services, and the latter (the lesser of the two issues) is only mitigated by the steps above or Log4j 2.15+. These can be triggered when a specially crafted string is parsed and processed by the vulnerable Log4j 2 component, which could happen through any user-provided input.

Attackers don’t need prior system access to log the string and can remotely cause the logging event by using commands like curl against a target system to log the malicious string in the application log. Successful exploitation allows for arbitrary code execution in the targeted application. When processing the log, the vulnerable system reads the string and executes the code from the malicious actor, which can grant the attacker full access and control of the application.

Logging code and functionalities in applications and services are typically designed to process a variety of external input data coming from upper layers and from many possible vectors. With that, the biggest risk factor of this vulnerability is predicting whether an application has a viable attack vector path that will allow the malformed exploit string to reach the vulnerable Log4j 2 code and trigger the attack.

A common pattern of exploitation risk, for example, is a web application, with code designed to process usernames, referrer, or user-agent strings in logs. These strings are provided as external input (e.g., a web app built with Apache Struts). An attacker can send a malformed username or set user-agent with the crafted exploit string hoping that this external input will be processed, at some point, by the vulnerable Log4j 2 code and trigger code execution.

We're here to help

Platform.sh recommends that you regularly redeploy your environments to pick up the latest security patches as an ongoing effort to keep you safe. As we continue to monitor this threat, we’ll update our customers with technical information to detect, investigate, and mitigate potential attacks. Need more help? We’re always available through our Customer Support team and our public Slack channel.