It’s all about the data; a journey into Kubernetes CSI on AWS

Mike Tougeron
GrepMyMind
Published in
10 min readMar 15, 2021

Over the last several weeks I’ve taken a trip into the world of Kubernetes storage; both the Container Storage Interface (CSI) and Container Attached Storage (CAS). I’ve talked with folks in the CAS space before but for whatever reason the power of it never really settled into my brain until recently. The idea of this journey started picking up steam when I realized that the in-tree storage plugins were deprecated and no new enhancements were being made to them starting with Kubernetes 1.20. When I discovered that simply switching from gp2 to gp3 volumes meant I had to start using the AWS CSI Driver I realized I was behind the times. This desire for a simple change opened the door and the next thing I knew I was on an adventure of potentially significant impact.

photo of containers and trucks, boat, planes carrying them

The journey started with the new AWS EBS volume types, but then sped into some code trying to fix an open issue in the aws-ebs-csi-driver, jumped up into VolumeSnapshots, spun around to creating PVCs from snapshots, and rounded the corner into doing an OpenEBS proof of concept. By the time I was done I was exhausted but full of excitement for the future possibilities.

aws-ebs-csi-driver

Please note that all references are assuming that you are running Kubernetes 1.17+ though I’ve only been running this configuration on a 1.19 cluster.

What is it?

The aws-ebs-csi-driver is a CSI storage plugin that replaces to the in-tree storage plugin for AWS EBS volumes. These plugins are what are used when you request a new PVC and the EBS volumes get created behind the scenes. In the beginning the storage plugins were part of the base Kubernetes repo/app but over time that has evolved. The creation of the CSI spec has opened the door not only for new CAS providers but also the cloud providers. The speed at which storage drivers can be iterated on can grow significantly because it is out of band from the core Kubernetes releases.

If you’re looking at it for the first time it can be daunting to understand what each component of the app does or even why you need to run it at all! At a very high level, you run a Deployment for the controller (with its many sidecars), a DaemonSet (also with sidecars) on every node, and optionally the snapshot controller.

The ebs-csi-controller has several sidecars in its deployment but main container is the ebs-plugin. This is where the code lives that interacts with the AWS APIs to create, delete, resize, etc the EBS volumes when a Persistent Volume is created. This is the code you’ll see when you go to https://github.com/kubernetes-sigs/aws-ebs-csi-driver.

The other sidecars essentially contain boilerplate logic that handle the communication and coordination between the ebs-plugin and the Kubernetes API. For example, the csi-resizer sidecar watches for PVC edits and notifies the ebs-plugin, over a socket using grpc, with the necessary data so that it can resize the EBS volume via the AWS API. The sidecars allow the ebs-plugin driver to focus on just the storage volume functionality and not have to re-implement a lot of the same Kubernetes API interactions. I highly recommend reading the Kubernetes CSI Sidecar Containers portion of the documentation to get a better idea of what each one does. I don’t think you necessarily need to know them in-depth but a general understanding of what each piece does is really helpful.

Along with the controller there is the ebs-csi-node that runs as a DaemonSet one each node. It is responsible for mounting and unmounting a volume from the node when requested by the kubelet. This is what makes the volume available for the Pods to use.

How to set it up

The aws-ebs-csi-driver requires AWS API access in order to manage the EBS volumes. I recommend something like kube2iam to handle this and to not use access keys. The official documentation has an example IAM policy and it looks like this.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteSnapshot",
"ec2:DeleteTags",
"ec2:DeleteVolume",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInstances",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumes",
"ec2:DescribeVolumesModifications",
"ec2:DetachVolume",
"ec2:ModifyVolume"
],
"Resource": "*"
}
]
}

Once you have the IAM role configured you can launch the controllers via the Helm chart.

helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driverhelm repo updatehelm upgrade --install aws-ebs-csi-driver \
--namespace kube-system \
--set enableVolumeScheduling=true \
--set enableVolumeResizing=true \
--set 'podAnnotations.iam\.amazonaws\.com/role'=ROLE_ARN \
--set 'node.podAnnotations.iam\.amazonaws\.com/role'=ROLE_ARN \
aws-ebs-csi-driver/aws-ebs-csi-driver

After it has been applied you’ll see the pods running in the kube-system namespace.

NAME                                  READY STATUS    RESTARTS AGE
ebs-csi-controller-85bc6d8897-lt5xk 6/6 Running 0 3m7s
ebs-csi-controller-85bc6d8897-v542j 6/6 Running 0 3m7s
ebs-csi-node-66dt6 3/3 Running 0 3m7s
ebs-csi-node-9424k 3/3 Running 0 3m7s
ebs-csi-node-b9mnd 3/3 Running 0 3m7s
ebs-csi-node-gd6d6 3/3 Running 0 3m7s
ebs-csi-node-hr4qt 3/3 Running 0 3m7s
ebs-csi-node-jjbcj 3/3 Running 0 3m7s

You will also see the CSIDriver installed on your cluster.

$> kubectl get csidriverNAME              ATTACHREQUIRED   PODINFOONMOUNT   MODES        AGE
ebs.csi.aws.com true false Persistent 21m

The CSIDriver is what you use when creating the StorageClass so that Kubernetes knows which CSI storage plugin should be used. This means that you can have more than one storage plugin running on your cluster at the same time! For example, in my case I have the in-tree storage plugins, the aws-ebs-csi-driver plugin, and OpenEBS (from the POC that I’ll discuss in a future blog post) all running nicely together.

How to use it

Now that the controller and node pods are running and the CSIDriver is created you can create the StorageClass(es) your users will use.

apiVersion: storage.k8s.io/v1
kind: StorageClass
provisioner: ebs.csi.aws.com # <-- The same name as the CSIDriver
metadata:
name: gp3
parameters: # <-- parameters for this CSIDriver
encrypted: "true"
type: gp3
allowVolumeExpansion: true
volumeBindingMode: Immediate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
provisioner: ebs.csi.aws.com
metadata:
name: gp3-6000iops
parameters:
encrypted: "true"
type: gp3
throughput: 250
iops: 6000 # <-- For volumes 1TB-2TB in size or needing more iops
allowVolumeExpansion: true
volumeBindingMode: Immediate

From an end-user perspective, the new gp3 storage class is used just like they’ve been used to doing with the in-tree storage plugins.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: touge-pvc
spec:
storageClassName: gp3
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi

Let’s follow the process and inspect the results.

$> kubectl apply -f touge-pvc.yaml
persistentvolumeclaim/touge-pvc created
$> kubectl get pvc touge-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
touge-pvc Bound pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec 10Gi RWO gp3 10s
$> kubectl describe pv pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec
Name: pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: gp3
Status: Bound
Claim: default/touge-pvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: topology.ebs.csi.aws.com/zone in [us-west-2c]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: ebs.csi.aws.com
FSType: ext4
VolumeHandle: vol-0f06e363f467b87bd
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1615144050357-8081-ebs.csi.aws.com
Events: <none>
$> aws ec2 describe-volumes --volume-ids vol-0f06e363f467b87bd --region us-west-2
{
"Volumes": [
{
"Attachments": [],
"AvailabilityZone": "us-west-2c",
"CreateTime": "2021-03-07T21:42:30.268Z",
"Encrypted": true,
"KmsKeyId": "arn:aws:kms:us-west-2:REDACTED:key/REDACTED",
"Size": 10,
"SnapshotId": "",
"State": "available",
"VolumeId": "vol-0f06e363f467b87bd",
"Iops": 3000,
"Tags": [
{
"Key": "kubernetes.io/created-for/pv/name",
"Value": "pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec"
},
{
"Key": "kubernetes.io/created-for/pvc/namespace",
"Value": "default"
},
{
"Key": "kubernetes.io/created-for/pvc/name",
"Value": "touge-pvc"
},
{
"Key": "CSIVolumeName",
"Value": "pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec"
}
],
"VolumeType": "gp3",
"MultiAttachEnabled": false,
"Throughput": 125
}
]
}

VolumeSnapshots

VolumeSnapshots are a pretty cool feature that’s possible with CSI. You can do things like taking a snapshot of a volume and then restore the PVC with that snapshot if your data becomes corrupt. An interesting use-case would be to create a nightly snapshot of your dev database and allow users to create a PersistentVolumeClaim (PVC) from that snapshot to use in their personal testing. The snapshot doesn’t even need to come from inside Kubernetes!

Enabling VolumeSnapshots

It’s an add-on to the default setup so the first thing you need to do is install the CSI Snapshotter CRDs. After installing the Snapshotter CRDs you can add --set enableVolumeSnapshot=true to the Helm install command from above and a new StatefulSet, ebs-snapshot-controller, will be running.

It uses a VolumeSnapshotClass to know which CSI Plugin the snapshot requests go to so let’s create one.

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
name: ebs-csi-aws
driver: ebs.csi.aws.com # <-- The CSIDriver we defined previously
deletionPolicy: Delete

Creating VolumeSnapshots

To create a new VolumeSnapshot create a resource on the cluster for it.

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
name: touge-snapshot
spec:
volumeSnapshotClassName: ebs-csi-aws
source:
persistentVolumeClaimName: touge-pvc

This will trigger the snapshotting process, the aws-ebs-csi-driver will be notified, and it will create a snapshot in AWS for the EBS volume that is backing the PVC. Once again, let’s follow the process and inspect the results.

$> kubectl apply -f touge-snapshot.yaml 
volumesnapshot.snapshot.storage.k8s.io/touge-snapshot created
$> kubectl describe volumesnapshot touge-snapshot
Name: touge-snapshot
Namespace: default
Labels: <none>
Annotations: API Version: snapshot.storage.k8s.io/v1beta1
Kind: VolumeSnapshot
Metadata:
Creation Timestamp: 2021-03-07T21:50:40Z
Finalizers:
snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
Generation: 1
Resource Version: 135554
Self Link: /apis/snapshot.storage.k8s.io/v1beta1/namespaces/default/volumesnapshots/touge-snapshot
UID: 7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Spec:
Source:
Persistent Volume Claim Name: touge-pvc
Volume Snapshot Class Name: ebs-csi-aws
Status:
Bound Volume Snapshot Content Name: snapcontent-7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Creation Time: 2021-03-07T21:51:13Z
Ready To Use: true
Restore Size: 10Gi

If we look at the Status we will see Bound Volume Snapshot Content Name: snapcontent-7d32eca6–2015–4a6e-a5b6–3ec48ca68005. This tells us which VolumeSnapshotContent is created for our VolumeSnapshot. The VolumeSnapshotContent is a resource that is created by the snapshot controller that represents the data the CSI Plugin created.

$> kubectl describe volumesnapshotcontents snapcontent-7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Name: snapcontent-7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Namespace:
Labels: <none>
Annotations: <none>
API Version: snapshot.storage.k8s.io/v1beta1
Kind: VolumeSnapshotContent
Metadata:
Creation Timestamp: 2021-03-07T21:50:40Z
Finalizers:
snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
Generation: 1
Resource Version: 135553
Self Link: /apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents/snapcontent-7d32eca6-2015-4a6e-a5b6-3ec48ca68005
UID: 0049e16f-2196-456b-9460-319ee24b3a15
Spec:
Deletion Policy: Delete
Driver: ebs.csi.aws.com
Source:
Volume Handle: vol-0f06e363f467b87bd
Volume Snapshot Class Name: ebs-csi-aws
Volume Snapshot Ref:
API Version: snapshot.storage.k8s.io/v1beta1
Kind: VolumeSnapshot
Name: touge-snapshot
Namespace: default
Resource Version: 133849
UID: 7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Status:
Creation Time: 1615153873000000000
Ready To Use: true
Restore Size: 10737418240
Snapshot Handle: snap-05bc0ec2f3a65b7be

Here is where we see Snapshot Handle: snap-05bc0ec2f3a65b7be that tells us the SnapshotID in AWS.

$> aws ec2 describe-snapshots --snapshot-ids snap-05bc0ec2f3a65b7be --region us-west-2
{
"Snapshots": [
{
"Description": "Created by AWS EBS CSI driver for volume vol-0f06e363f467b87bd",
"Encrypted": true,
"KmsKeyId": "arn:aws:kms:us-west-2:REDACTED:key/REDACTED",
"OwnerId": "REDACTED",
"Progress": "100%",
"SnapshotId": "snap-05bc0ec2f3a65b7be",
"StartTime": "2021-03-07T21:51:13.115Z",
"State": "completed",
"VolumeId": "vol-0f06e363f467b87bd",
"VolumeSize": 10,
"Tags": [
{
"Key": "CSIVolumeSnapshotName",
"Value": "snapshot-7d32eca6-2015-4a6e-a5b6-3ec48ca68005"
}
]
}
]
}

Creating a PVC from an Existing AWS Snapshot

Let’s go through the example use-case of taking an existing AWS snapshot and creating a PVC from it for someone to use inside Kubernetes.

First we need to create the VolumeSnapshotContent that references the AWS snapshot. Using the AWS console I created snap-002e544b538087ec1 from an EBS volume that I had. To show the power of this, the volume & snapshot were created outside of Kubernetes.

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotContent
metadata:
name: my-imported-snapshot
spec:
volumeSnapshotRef:
kind: VolumeSnapshot
name: my-imported-snapshot
namespace: default
source:
snapshotHandle: snap-002e544b538087ec1 # <-- snapshot to import
driver: ebs.csi.aws.com
deletionPolicy: Delete
volumeSnapshotClassName: ebs-csi-aws

Then we need to create the VolumeSnapshot that uses that VolumeSnapshotContent.

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
name: my-imported-snapshot
namespace: default
spec:
volumeSnapshotClassName: ebs-csi-aws
source:
volumeSnapshotContentName: my-imported-snapshot

And apply it to the cluster.

$> kubectl apply -f touge-import-snapshot.yaml 
volumesnapshotcontent.snapshot.storage.k8s.io/my-imported-snapshot created
volumesnapshot.snapshot.storage.k8s.io/my-imported-snapshot created
$> kubectl get volumesnapshotcontent
NAME AGE
my-imported-snapshot 4m31s
$> kubectl get volumesnapshot
NAME AGE
my-imported-snapshot 4m56s

The VolumeSnapshot is now available to be used to create the PersistentVolumeClaim.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-imported-snapshot-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 10Gi
dataSource:
name: my-imported-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io

This is then applied to the cluster and we will have a new PVC that we can mount on our Pod.

$> kubectl apply -f touge-pvc-from-snapshot.yaml 
persistentvolumeclaim/my-imported-snapshot-pvc created
$> kubectl get pvc my-imported-snapshot-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-imported-snapshot-pvc Bound pvc-1ff63250-5a4f-442f-9907-171b69569c2b 10Gi RWO gp3 26s
$> kubectl describe pv pvc-1ff63250-5a4f-442f-9907-171b69569c2b
Name: pvc-1ff63250-5a4f-442f-9907-171b69569c2b
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: gp3
Status: Bound
Claim: default/my-imported-snapshot-pvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: topology.ebs.csi.aws.com/zone in [us-west-2a]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: ebs.csi.aws.com
FSType: ext4
VolumeHandle: vol-03b42ee74d7fd4f4e
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1615144050357-8081-ebs.csi.aws.com

Where I plan to go from here…

Once I had played around with the aws-ebs-csi-driver for a few days I went ahead and implemented it in the production clusters. I haven’t yet gotten around to migrating the in-tree volumes to the new CSI based ones but there’s a bit of time for that.

Following my excitement I ended up doing a proof of concept using OpenEBS. I’m really excited about the possibilities there and will be writing about that setup soon.

Sign up to discover human stories that deepen your understanding of the world.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

Published in GrepMyMind

Feel free to grep & grok your way through my thoughts on Kubernetes, programming, tech & other random bits of knowledge. My randomness is my own & not those of any company I might be working for. I may be right, I may be wrong, but as Deep Thought said, “42.”

Written by Mike Tougeron

Lead SRE @Adobe , #kubernetes fan & gamer (board & video). he/him. Remember, reality is all in your head…

No responses yet

Write a response