Switching to Platform.sh can help IT/DevOps organizations drive 219% ROI
Organizations, the ultimate way to manage your users and projects
Security vulnerability uncovered and patched in the golang.org/x/crypto /ssh package
Misimplementation of PublicKeyCallback leads to authorization bypass in Go's x/crypto/ssh
Platform.sh teams are always striving to ensure a safe space for all developers within our product. And this consistent diligence led to the Platform.sh Engineering team discovering a security vulnerability in the golang.org/x/crypto/ssh package on 5 September 2024.
Upon investigating an unexpected Panic: runtime error: invalid memory address or nil pointer dereference message in our edge proxy, the engineers discovered a misimplementation of the PublicKeyCallback function. A quick remediation was carried out, and no evidence was found to believe this vulnerability had been exploited before the patch was implemented.
Systems that implement this callback function incorrectly end up with a vulnerability that allows an authorization bypass in Go's x/crypto/ssh. Our analysis suggests that this issue is prevalent across multiple projects utilizing this (golang.org/x/crypto/ssh) package, leading to potentially severe security implications. At the time of writing, there are ~19k known importers of this ssh package.
While this is ultimately a bug in the usage of the package by end-users, the design of the PublicKeyCallback function and its documentation contribute to the vulnerability, potentially affecting a broad spectrum of users in the wild. Given the broad impact potential, the Platform.sh Engineering team disclosed the vulnerability to the Go Security team.
How the event unfolded
- 5 September 2024: Platform.sh Engineering team discovered security auth bypass in x/crypto/ssh.
- 12 September 2024: Reported Findings to the Go Security team and implemented mitigations.
- 2 October 2024: Go security team reviewed our report and confirmed our findings.
- 6 December 2024: The Go security team made a pre-announcement.
- 12 December 2024: Go security team made public disclosure and released v0.31.0 of golang.org/x/crypto
Why this matters
The ssh-2 protocol, specified in RFC 4253, was designed in its current form back in 2006 as an evolution of the original ssh-1 protocol from 1995. Back in ssh-1 times, the client authentication was done as a challenge/response protocol outlined under SSH__AUTH_RSA_ on pages 11-12 here. This allowed clients to ask the server where a particular key had the potential to be accepted:
"The client sends SSH_CMSG_AUTH_RSA with public key modulus (n) as an argument. The server may respond immediately with SSH_SMSG_FAILURE if it does not permit authentication with this key. Otherwise, it generates a challenge."
The ssh-2 protocol changed this to a static signature based on data that is already known by the two parties but kept a query mechanism in place as a way to improve performance and usability. Secion-7 of RFC 4252:
"Private keys are often stored in an encrypted form at the client host, and the user must supply a passphrase before the signature can be generated. Even if they are not, the signing operation involves some expensive computation. To avoid unnecessary processing and user interaction, the following message is provided for querying whether authentication using the "publickey" method would be acceptable."
Let’s start at the beginning
The golang.org/x/crypto/ssh package is an implementation of the ssh-2 protocol. The Golang implementation hides the complexity of the "query that a key might be accepted/actually authenticate" dance under a single API.
PublicKeyCallback is called only once for a particular key before the client has proven possession of the private key via a cache implemented in the server. Go #L621. The comments in the PublicKeyCallBack func below have critical implementation details that are easily overlooked.
// PublicKeyCallback, if non-nil, is called when a client
// offers a public key for authentication. It must return a nil error
// if the given public key can be used to authenticate the
// given user. For example, see CertChecker.Authenticate. A
// call to this function does not guarantee that the key
// offered is in fact used to authenticate. To record any data
// depending on the public key, store it inside a
// Permissions.Extensions entry.
PublicKeyCallback func(conn ConnMetadata, key PublicKey) (*Permissions, error)The ease of misimplementation for this function was mentioned in issue #20094 from April 2017, and clarifying code comments were subsequently added.
Despite the description hinting that PublicKeyCallBack func needs to be implemented in a stateless way, what can happen if the func is stateful is a client can successfully query a public key that it doesn't use. Where doing so only requires knowledge of the public key, which, by nature of being public, is very knowable. For example, the public keys of all GitHub users are public if you append .keys to the user URL.
Here are Damien Tournoud, Platform.sh CTO's, public keys for your enjoyment: GitHub Public Keys
The flaw in this process occurs because the PublicKeyCallback is not called again after the client presents the key, allowing the attacker to gain access without possessing the private key for the queried key.
How does this align with Platform.sh and Upsun
Our ssh gateways allow our customers to access containers running on our infrastructure using standard ssh tools and are based on the open source golang.org/x/crypto/ssh package.
Platform.sh favors open source technologies and enjoys contributing to and supporting open source projects it loves. We take great pride in catching and addressing an issue like this. It genuinely feels awesome when you can make the tools you use better for everyone else.
Given the reach of this issue (19k importers), the upstream Go team was best suited to handle the disclosure. Our Platform.sh Engineering team encourages others to also disclose responsibly and have confidence in their upstream maintainers. As developers, we understand it can often be a thankless job, and unwarranted complications are not welcome. Eliminating unnecessary complexities for developers is the core on which our products are built.
What we recommend
If you have a Go project depending on crypto/ssh, update the package to the newest version with:
go get -u golang.org/x/crypto/ssh
Then, if your code has a PublicKeyCallback method (implementing the ssh.ServerConfig interface), you will need to ensure that:
- The callback must be stateless, or in other words, it must not have any side effects. As the comments on the interface suggest, return any state directly in the function as part of the ssh.permissions object.
- Do not assume that the public key is authenticated at this stage. Use the PostSuccessAuthCallback if you need to run code after the client has proved possession of the private key. This will be passed through the same ssh.permissions object.
Even if the PublicKeyCallback is not part of your own code, remember to check that you are not using any other packages that misimplement the same callback. If so, they must also be updated, fixed, or removed.
The package maintainers were very understanding and professional as we communicated that users of the x/crypto/ssh package expect that the key used by the client for authentication (if successful) matches the one passed to the most recent call to PublicKeyCallback.
The maintainers' cache fix should mitigate this vulnerability for the majority of implementations. Unfortunately, it cannot fix everything.
