Build things! But also break them!

What I would like to write is: I got my first CVE.

But I guess it's a lot more appropriate to say: I stumbled upon a "vulnerability" that would become CVE2024-24747.

Let's unpack this!

When the challenge mini0day dropped during RealWorldCTF 2024, I immediately jumped on it since I had some experience with minio from various projects I am involved in. This post is not going to be a full writeup for that challenge, but I will give a short summary of the relevant parts for context:

Summary of minio0day ​

Minio is an object storage with an S3-compatible API. In other words: A server you can run yourself and use just like AWS' S3 api to store and retrieve files via HTTP.

The challenge description mentioned "old CVEs" and came with a copy of the remote setup, leaking an access key and revealing the deployed minio version. That particular version of minio had some improper privilege management (CVE-2023-28434), which allowed an access key with write permissions to overwrite files in the so-called metadata-bucket, where minio stores its access policies, access keys, etc. To solve the challenge, we exploited the aforementioned vulnerability to attach a policy to our leaked access key, giving it administrative privileges. From there we leveraged a disabled signature check to "update" the minio binary to one with a built-in reverse-shell, which finally allowed us to take the flag.

But wait, there is more! ​

I am quite confident that we solved the challenge the way the author intended. The interesting thing actually happened after the CTF: Astrid from Kalmarunionen shared their solution for the challenge which was a lot simpler:

Instead of exploiting aforementioned CVE, they simply plugged the leaked access key into minios cli and issued a command to escalate the keys own policy.

It would look something like this:

$ mcli admin user svcacct edit --policy admin-policy.json <server-alias> <access-key-name>

where admin-policy.json would roughly have these contents:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:*"],
      "Resource": ["arn:aws:s3:::*"]
    },
    {
      "Effect": "Allow",
      "Action": ["admin:*"],
      "Resource": []
    }
  ]
}

There were some comments on discord that this was likely an unintended solve and a lot of facepalming on our team, but apparently none of the roughly 550 people in the channel considered it noteworthy beyond that.

I learned about this unintended solve the next day and had a complete epiphany: Of course that works! But in order to understand why, we will need a bit more context about permissions and access control in minio and AWS.

Many trees, but only one root ​

AWS has many APIs, but access control for all of those is centralized in IAM. The three important concepts of IAM we need to digest the issue at hand are actions, policies and service accounts. You have already seen a JSON-policy in the snippet above. Actions, such as storing or deleting objects in S3 are identified by strings such as s3:PutObject or s3:DeleteObject, similarly, starting and stopping EC2 instances is represented via ec2:StartInstances or ec2:StopInstances. A policy denies or allows a set of actions on a set of resources which are also identified with a similar url-like, hierarchical naming scheme. So we have a tree of actions and a tree of resources. Crucially, access control itself is not exempt from this scheme. iam:CreatePolicy is simply an action of IAM, performed on an IAM resource (a policy). Sensibly, for service accounts, all actions are denied by default.

In order to be as S3-compatible as possible, minio implements the same mechanisms, but only for the s3:*-subtree of actions. However, there is some functionality in minio that AWS doesn't have; specifically: all the actions that you need to manage your minio cluster. To control access to these actions, minio opted to simply create another sub-tree of actions: admin:, containing e.g.: admin:ServerUpdate, which we needed to perform our malicious update during the CTF. So what about iam:? How do we manage the permissions for managing permissions? Well, they ended up in admin: as well, for example: admin:UpdateServiceAccount.

The last tree we need to plant for the complete picture is the implicit hierarchy of service accounts. Access keys (service accounts) inherit the policies (permissions) attached to their parent. In a simple minio deployment that hierarchy has exactly one root: The minio root user, which - well - may do everything.

So let's tie it all together:

• The actions of the S3 API and the actions for managing permissions are in completely separate subtrees:

  • iam: in AWS
  • admin: in minio.

• In AWS, service accounts are denied all actions by default - you need to explicitly allow them with policies.

• In the typical minio setup, all service accounts are descendants of the root user, inheriting its policies.

So unless you somewhere in your service account hierarchy explicitly deny admin: actions, access keys can simply modify their own policy to something giving them access to everything. Exacerbating the footgunyness of this constellation is that specifying policies denying s3: actions for access keys perfectly seems to get the job done at first. If you set a policy that allows all actions, but only for arn:aws:s3:::mypublicbucket, that access key can only access mypublicbucket but only until that access key decides to go rogue and overwrite its own policy. Additionally, it doesn't seem to work like this for all actions, admin:ServerUpdate for example needs to be allowed explicitly.

So is that a "vulnerability"? ​

I was contemplating this for quite a while. Essentially, all the information is there. One just needs to make the deductions laid out above. But it wasn't explicit in minio's user interface, documentation or any of minio's blog posts. My most important considerations when deciding whether to write a report where these:

Unsafe default behaviors like this are not unheard of, minio's default admin credentials are the closest example. Considering those a vulnerability would be silly, but compared to inheritance of policies, they are a lot more "on the nose".

I had access to several minio deployments and discovered them to be "vulnerable" like that. With minio having more than 1B pulls on dockerhub and more than 40k stars on Github, it was only a matter of statistics that there are a lot more vulnerable deployments.

"Is this really a vulnerability?" might really be the wrong question to ask here. Maybe I should ask: "What can a report achieve here?". I was hoping that they would perhaps update the UI to indicate admin privileges on access keys, or perhaps add a warning during key creation, or at least update the documentation.

Finally, I decided to submit a report to the minio maintainers and let them decide. The rest is contained in the advisory.

I didn't really find it - but neither did anyone else... ​

Saying that I "found" this vulnerability would be an overstatement. I was aware that the svcacct edit functionality existed, but I never got the idea to use it to escalate a key's own privileges. On the other hand, I apparently was the first or only person to realize the implications of the unintended solve. At the end of the day, I am really glad Github has a role called Reporter for advisories, making me feel a lot less like an imposter. 😃

Building vs. breaking stuff ​

Overall, I believe this anecdote supports a hypothesis I have been considering for quite some time now:

Building things helps you understand security problems, but it may also give you blind spots.

As much as I enjoy breaking things. I also love building stuff: I dabbled in programming even before attending university and started working as a developer during the second semester of my bachelors. I also took on a major part in hosting saarctf23 after playing less than a dozen CTFs myself.

These years of experience are usually quite helpful; they have etched many principles into my thinking that help me intuitively navigate complex systems and challenges. Sometimes I observe some behavior on a piece of software and can practically see the faulty code in front of my inner eye. Similarly, I sometimes read a description of a security issue and the entire process of how it was developed fast-forwards through my brain.

On the other hand, these strong intuitions for how to "correctly" implement something occasionally stop me from asking these very simple but ever so important: "What if I just ..."-questions. What if I just try to overwrite my own policy? If you are really comfortable inside the box, it takes more effort to step out of it.

The implications of the unintended solve where immediately clear to me because I had worked with that system before, but I would have never gotten the idea to try that, because it just felt too wrong to even consider. So go on, build things! But don't forget to break 'em!