A couple of weeks ago, Docker disclosed that approximately 190,000 accounts may have been exposed and unauthorized actors could potentially have access to their images. The docker hub breach brings out some of the most critical vulnerabilities of the software delivery pipeline, and it’s worth exploring the root cause of such incidents and how concerned enterprises and users should be.
The compromise would allow a third party to re-use passwords and access or modify images in the Docker Hub. The result of these actions could be the installation of backdoors or rootkits on what users would think are legitimate images. Deployment of these images could propagate the attack to hundreds or thousands of deployments.
Since this attack is at the core of the software delivery pipeline, it would actually go undetected by most security tools.
A false sense of security around this issue is that an image vulnerability scanning tool would actually identify and catch such an attack. Unfortunately, most of these tools provide a false sense of security, often needed for satisfying auditing bureaucratic requirements and doing very little for actual security.
In most cases, vulnerability scanning tools parse the Dockerfile and the image layers, identify the list of packages installed in an image, and compare this list with an existing database of vulnerabilities. However, what they do not do is validate the integrity of each and every binary file stored in the image, since they do not really have the fundamental information to perform that task. Most vulnerability scanning tools will not be able to identify malicious code in your own binaries for example.
Let us illustrate this with an example. Download your favorite image from a public or private registry. With a simple “`docker exec“` or “`docker cp“`update one of the binary files of the image and populate it with different content. You can choose any binary file, preferably one installed through a package. Save the image, and then push it back to your registry. Look for vulnerabilities through your favorite vulnerability scanning tool. In most cases, they will not be able to identify the new potentially rogue binary that was inserted in your image.
Now consider what an attacker with the same access to your registry can do. By replacing binary files in your images, you can end up with a software deployment ready for blockchain mining without your knowledge. The attacker just made his profit. Indeed, most of the vulnerability scanning solutions, did not disclose any mechanism that could help you with this particular attack, other than urging you to manually inspect the last update times for your images and the activity in your Docker Hub account.
In some instances, image signing with technologies like the Docker Notary could provide some limited help, as long as your signing keys themselves are not leaked in the same way that your registry keys are leaked.
There is always a limit to what vulnerability scanning can do for you and we must all be aware of the limitations. Yes, it is needed for basic image hygiene, but there are some fundamental limits.
The root cause of the docker hub breach is secrets. Your CI/CD pipeline and your users depend on passwords and static tokens to access the docker registry or even your GitHub repository, and create, modify or download images. Even though in this case the problem was an attack on the Docker Hub itself, another common threat vector is a rogue actor attacking your CI/CD pipeline or GitHub repository and stealing access tokens, or more often than not, it is due to some user misconfiguration leaking the access tokens. An attacker with access to your secrets or tokens could repeat the exact same attack without being detected for a while since this could be targeted at a specific organization.
So, if we really want to address the root cause of the problem, we need to think about “secrets”.
What was lost in the hype of this discussion, is that if you were actually using a Google Cloud Registry or an AWS Container Registry, you would be able to minimize the probability of secrets leaking and becoming the tool of such an attack. We have no affiliation with either of these providers, and no reason to believe that they could protect your secrets better than Docker, but we do recognize the benefits of “secretless” access, something that the Docker Hub does not offer you.
Provided you are running in the cloud, your CI/CD pipeline or automation tools would only need to rely on a service account or service role to automatically authenticate with the corresponding registry and modify or update images. Similarly, your users can access these registries based on dynamic authentication and authorization with Single-sign-on (SSO) tools and 2FA. You would have no need to store secrets or tokens anywhere in your infrastructure. Just assign a Service Role or Service Account to the corresponding VMs or Lambda functions, provide the right identity and access management configuration, and off you go. You never need to worry about leaking secrets, rotating secrets or losing secrets again. Even secrets management tools (like Vault or AWS EKS), at the end of the day, require you to somehow manage, rotate, and update secrets.
The Docker Hub breach is yet another proof point that secrets are still and always will be the weakest link in any infrastructure. Leaked secrets are the number one tool for most attacks, whether for users or automated software deployments. Security and monitoring tools fail over and over again to address this issue, and promise wonderland by yet another machine learning algorithm. Secrets stores just push the problem around since you still need to authenticate to the secrets stores with some secret and manage the secrets stores themselves.
The answer to this problem is not more monitoring tools or more secrets stores.
We need to solve the problem at the root. We need to completely get rid of the need of secrets, and use a root-of-trust infrastructure that will dynamically provide credentials to applications and users and allow fine grained authorization controls without any hardwired secrets anywhere.
We need to move to “Secretless” infrastructure.