Introducing the k8s-aws-ebs-tagger

https://stock.adobe.com/images/silhouettes-of-taggers-drawing-graffiti/67460951

EDIT: The k8s-aws-ebs-tagger was renamed to k8s-pvc-tagger as the scope of the project was expanded to include more than just aws-ebs volumes. Don’t worry, it’s still backwards compatible.

The k8s-aws-ebs-tagger brings tagging to the AWS EBS volumes created by Kubernetes PersistentVolumeClaims (PVC). This new utility enables you to set arbitrary tags on the EBS volume so that you can better categorize and report on the state of your AWS resources. Having proper cost control tags can help you keep a handle on your AWS billing and resource utilization.

Let’s dive into how to install and use it.

Install the k8s-aws-ebs-tagger

The container images are released both on DockerHub and GitHub Container Registry and are built for both linux/amd64 and linux/arm64.

The first thing needed is an AWS IAM Role that is allowed to add & delete tags from EBS volumes. I recommend using kube2iam for assigning the role to the Pod(s) instead of using AWS access key/secrets.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:volume/*"
            ]
        }
    ]
}

Once you have the IAM Role you can get the app running via its Helm chart. Be sure to check the default values and adjust as appropriate for your environment.

helm repo add mtougeron https://mtougeron.github.io/helm-charts/
helm repo update
helm install k8s-aws-ebs-tagger mtougeron/k8s-aws-ebs-tagger

If you want it to only watch a single namespace you can set the watchNamespace value for the chart but it still needs a ClusterRole in order to get the volume ID from the PersistentVolume. Currently it only supports watching a single namespace or all namespaces (#9) but I plan on updating this soon.

Configuring the tags to set

The first approach is to use the (optional) --default-tags command line flag that takes a json encoded string of key/value pairs. It uses these tags as the base set of tags to add to all EBS Volumes when a PersistentVolumeClaim is added or updated. This is useful if you always want to add a fixed tag to all EBS volumes created in the cluster (or namespace). For example, you may want all volumes to have the tag Environment=Production.

The default tags can be extended by the aws-ebs-tagger/tags annotation on the PVC. This annotation also takes a json encoded string of key/value pairs and uses them for tags on the volume. This can be used to extend the list of tags you want set as well as override the default values.

Take for example, this deployment and PVC

apiVersion: apps/v1
kind: Deployment
metadata:
name: k8s-aws-ebs-tagger
spec:
...
  template:
    spec:
      containers:
        - name: k8s-aws-ebs-tagger
          args:
            - --default-tags={"Environment": "Production"}
...
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example1
  annotations:
    aws-ebs-tagger/tags: |
      {"Database": "true"}
...

The resulting EBS volume will have the tags Environment=Production and Database=true.

Let’s say that for databases you have a different Environment tag. You could use the same Deployment as above but on the Database PVCs you override the Environment tag in the annotation.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db1
  annotations:
    aws-ebs-tagger/tags: |
      {"Database": "true", "Environment": "DBProduction"}
...

And the result tags will be Environment=DBProduction and Database=true

If you have a PVC that you don’t want tagged at all you can use the aws-ebs-tagger/ignore annotation and no tags will be processed for that volume.

Currently you can only use fixed values for the tags. However I’m working on updating that in the near future to allow for templated tag values (#15).

JSON vs multiple annotations vs comma delimited values

I went back and forth over this a lot when I was first thinking about writing this app. There are pros & cons to each approach but the deciding factor was that I needed a tag name with a : in it and I couldn’t use that in an annotation name. I also considered using a comma delimited list of tags but that made it difficult to allow the , in the value of the tag. With those restrictions the greatest flexibility was to use a json string of key/value pairs. That unfortunately leaves the user with the hassle of hand-coding json key/value pairs for their tags (which I really hate) but it’s the best I could think of.

Thoughts on the future

While the CSI Drivers are the latest & future of storage on Kubernetes there are still a lot of us not using them yet. Even if you are using the aws-ebs-csi-driver it still has an open issue to allow adding arbitrary tags. Until the CSI driver is updated this utility app should provide the desired tagging.

Completing #9 & #15 will allow handling of the last scenarios I can think of to make this usable by the masses. However, if you have any suggestions for improvement I’m happy to hear them!