The term of blast radius has often been used to describe the effect of a security breach to an application or organization. The term tries to capture the worst-case scenario of a breach and how organizations should protect themselves against such scenarios.
In the days of private data centers and perimeter security the blast radius was often the perimeter of the data center. Once an attacker found a way inside the perimeter they would laterally move and amplify the attack across applications and networks. As technology became more mature and the concept of segmentation appeared, solutions emerged that attempted to use network boundaries to minimize the blast radius. These techniques were viable and often achieved significant benefits.
The topic that we will be covering in this blog post however, is the new blast radius of an attack in a cloud native environment. We can start with a simple example of a cloud account (AWS, GCP, Azure). It has been well documented one cannot think of blast radius in these environments as defined by network boundaries since the weakest link is often in the credentials and access management capabilities.
IaaS and Multi-Account Techniques
It has been widely recommended that in order to minimize the blast radius in an environment such as AWS one has to use multiple-accounts. The API keys that applications or users use give them lots of permissions within the environment. If an API key is lost and this key has significant privileges one can easily use it to create, modify, or destroy resources even with an AWS VPC. They could for example create an attacker VM inside the VPC, they could change firewall rules and so on. So, using multiple accounts minimizes the blast radius of an API key being lost.
GCP has a little different philosophy than AWS with the concept of projects. A given user can be provided access to a subset of the projects of the account and one has to establish proper identity and access management rules. A user that needs access has to authenticate remotely through the CLI and/or a web-browser and then temporary credentials are stored in a file (default: ~/.config/gcloud). If this file is lost or copied by another user and moved to a different machine then the victim can easily access any project that this user has access to. Because of the single-sign-on capability and users requiring access to multiple accounts, the trick with multiple accounts that is often recommended in AWS does not really work. Users must pay extra attention on protecting these API access keys. This is especially dangerous with lost or stolen laptops.
When considering the blast radius in a Kubernetes clusters and Docker container, we can have several dimensions. As in the IaaS case there is the whole sequence of attacks with stolen API keys. However, in addition to protecting the API keys for accessing the cluster one has to consider the whole pipeline of application delivery and look at the blast radius of the different components:
• Container registry
• Container images
• Kubernetes credentials
• Container runtime
Let’s explore the risks in each of these dimensions:
Before we even consider the security of the cluster itself, we need to understand the security of the container registry. In order for users or systems to be able to push/update containers in the Docker registry they need credentials. Authentication and authorization in the registry is performed through OAUTH 2.0 tokens. These tokens have a limited expiration and they must be configured with the shortest possible parameters. However, if an attacker gets ahold of an OAUTH token for the registry, they can easily push an image update to the registry that is not necessarily seen. This image update can include spyware or coin-mining software that
is not going to be visible for a while.
There has been a lot of discussion about vulnerability scanning in containers, both for private and public registries. It makes big headlines to disclose that a large number of images have vulnerabilities in the registry and people tend to pay attention to this. Unfortunately, most vulnerability scanning is very much limited to basic OS vulnerabilities. No vulnerability scanner will be able to identity a coin-mining daemon buried deep in the code of a popular application image. These types of attack techniques have already shown up inside popular frameworks and they can easily appear in popular images.
So, when it comes to minimizing damage by container images one has to first consider the attack surface of the registry itself and strictly control who can push image updates to the registry. Limiting this capability to a CI/CD pipeline might reduce the attack surface, but one has to carefully manage access credentials for the pipeline itself.
It’s turtles all the way down. Who gives credentials to the CI/CD pipeline, and it they are hardwired, the pipeline becomes a vulnerability itself.
Same as with registry, IaaS platforms and so on, Kubernetes has the concept of user and service account that it boils down to a pretty static token. Any leak of the token, and a third party can get direct access to your Kubernetes cluster. Similar to gcloud, if you read these credentials in the clear from a default file (~/.kube/config). If a user loses these credentials, the attacker has the same access to the cluster as the original user. And because there is an automatic renewal process, the attacker will continue to have this access for a while. In other words, the attack is not just something that can be limited in time over 24 hours and until the credentials expire.
Last but not least, one needs to worry about the blast radius of attacks on the container runtime. Lots has been written about this and we will not spend too much time on the topic. The bottom line is that with proper configuration one can minimize the attack surface. Using user namespaces, proper controls on seccomp profiles and so on.
It’s All About Identity Protection
As you can see in all the above examples, the blast radius and attack surface in cloud deployments is not the boundary of the network. It is not a machine, a subnet, a VPC. The weakest link in all these attacks is actually identity and credentials for users and applications.
Without proper management of identities and access control in these platforms, a single attacker can create havoc without actually anything fancier than stealing the user credentials of finding an API key in Github. It is well known that Amazon AWS is actually scanning Github for API keys, but this sensitive information can actually leak in many different ways.
The problem is amplified with programmatic access to APIs that brings out the power of these platforms. This programmatic access becomes the weakest link.
What Should Users Do?
Most of these platforms provide very sophisticated mechanisms for controlling access to APIs through robust authorization mechanisms. They have to be rather weak on the token validity front since they do not want to compromise usability with the overhead of security. That is the main reason that the AWS access token is valid for a year. Addressing this problem requires a different way of thinking:
Interested in securing microservices and containers? Read the Microservices Security: Best Practices white paper.