It’s all about the data; a journey into Kubernetes CSI on AWS
Over the last several weeks I’ve taken a trip into the world of Kubernetes storage; both the Container Storage Interface (CSI) and Container Attached Storage (CAS). I’ve talked with folks in the CAS space before but for whatever reason the power of it never really settled into my brain until recently. The idea of this journey started picking up steam when I realized that the in-tree storage plugins were deprecated and no new enhancements were being made to them starting with Kubernetes 1.20. When I discovered that simply switching from gp2
to gp3
volumes meant I had to start using the AWS CSI Driver I realized I was behind the times. This desire for a simple change opened the door and the next thing I knew I was on an adventure of potentially significant impact.

The journey started with the new AWS EBS volume types, but then sped into some code trying to fix an open issue in the aws-ebs-csi-driver
, jumped up into VolumeSnapshots, spun around to creating PVCs from snapshots, and rounded the corner into doing an OpenEBS proof of concept. By the time I was done I was exhausted but full of excitement for the future possibilities.
aws-ebs-csi-driver
Please note that all references are assuming that you are running Kubernetes 1.17+ though I’ve only been running this configuration on a 1.19 cluster.
What is it?
The aws-ebs-csi-driver is a CSI storage plugin that replaces to the in-tree storage plugin for AWS EBS volumes. These plugins are what are used when you request a new PVC and the EBS volumes get created behind the scenes. In the beginning the storage plugins were part of the base Kubernetes repo/app but over time that has evolved. The creation of the CSI spec has opened the door not only for new CAS providers but also the cloud providers. The speed at which storage drivers can be iterated on can grow significantly because it is out of band from the core Kubernetes releases.
If you’re looking at it for the first time it can be daunting to understand what each component of the app does or even why you need to run it at all! At a very high level, you run a Deployment
for the controller (with its many sidecars), a DaemonSet
(also with sidecars) on every node, and optionally the snapshot controller.
The ebs-csi-controller
has several sidecars in its deployment but main container is the ebs-plugin
. This is where the code lives that interacts with the AWS APIs to create, delete, resize, etc the EBS volumes when a Persistent Volume is created. This is the code you’ll see when you go to https://github.com/kubernetes-sigs/aws-ebs-csi-driver.
The other sidecars essentially contain boilerplate logic that handle the communication and coordination between the ebs-plugin
and the Kubernetes API. For example, the csi-resizer
sidecar watches for PVC edits and notifies the ebs-plugin
, over a socket using grpc, with the necessary data so that it can resize the EBS volume via the AWS API. The sidecars allow the ebs-plugin
driver to focus on just the storage volume functionality and not have to re-implement a lot of the same Kubernetes API interactions. I highly recommend reading the Kubernetes CSI Sidecar Containers portion of the documentation to get a better idea of what each one does. I don’t think you necessarily need to know them in-depth but a general understanding of what each piece does is really helpful.
Along with the controller there is the ebs-csi-node
that runs as a DaemonSet one each node. It is responsible for mounting and unmounting a volume from the node when requested by the kubelet
. This is what makes the volume available for the Pods to use.
How to set it up
The aws-ebs-csi-driver requires AWS API access in order to manage the EBS volumes. I recommend something like kube2iam to handle this and to not use access keys. The official documentation has an example IAM policy and it looks like this.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteSnapshot",
"ec2:DeleteTags",
"ec2:DeleteVolume",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInstances",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumes",
"ec2:DescribeVolumesModifications",
"ec2:DetachVolume",
"ec2:ModifyVolume"
],
"Resource": "*"
}
]
}
Once you have the IAM role configured you can launch the controllers via the Helm chart.
helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driverhelm repo updatehelm upgrade --install aws-ebs-csi-driver \
--namespace kube-system \
--set enableVolumeScheduling=true \
--set enableVolumeResizing=true \
--set 'podAnnotations.iam\.amazonaws\.com/role'=ROLE_ARN \
--set 'node.podAnnotations.iam\.amazonaws\.com/role'=ROLE_ARN \
aws-ebs-csi-driver/aws-ebs-csi-driver
After it has been applied you’ll see the pods running in the kube-system
namespace.
NAME READY STATUS RESTARTS AGE
ebs-csi-controller-85bc6d8897-lt5xk 6/6 Running 0 3m7s
ebs-csi-controller-85bc6d8897-v542j 6/6 Running 0 3m7s
ebs-csi-node-66dt6 3/3 Running 0 3m7s
ebs-csi-node-9424k 3/3 Running 0 3m7s
ebs-csi-node-b9mnd 3/3 Running 0 3m7s
ebs-csi-node-gd6d6 3/3 Running 0 3m7s
ebs-csi-node-hr4qt 3/3 Running 0 3m7s
ebs-csi-node-jjbcj 3/3 Running 0 3m7s
You will also see the CSIDriver
installed on your cluster.
$> kubectl get csidriverNAME ATTACHREQUIRED PODINFOONMOUNT MODES AGE
ebs.csi.aws.com true false Persistent 21m
The CSIDriver
is what you use when creating the StorageClass
so that Kubernetes knows which CSI storage plugin should be used. This means that you can have more than one storage plugin running on your cluster at the same time! For example, in my case I have the in-tree storage plugins, the aws-ebs-csi-driver plugin, and OpenEBS (from the POC that I’ll discuss in a future blog post) all running nicely together.
How to use it
Now that the controller and node pods are running and the CSIDriver is created you can create the StorageClass(es)
your users will use.
apiVersion: storage.k8s.io/v1
kind: StorageClass
provisioner: ebs.csi.aws.com # <-- The same name as the CSIDriver
metadata:
name: gp3
parameters: # <-- parameters for this CSIDriver
encrypted: "true"
type: gp3
allowVolumeExpansion: true
volumeBindingMode: Immediate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
provisioner: ebs.csi.aws.com
metadata:
name: gp3-6000iops
parameters:
encrypted: "true"
type: gp3
throughput: 250
iops: 6000 # <-- For volumes 1TB-2TB in size or needing more iops
allowVolumeExpansion: true
volumeBindingMode: Immediate
From an end-user perspective, the new gp3
storage class is used just like they’ve been used to doing with the in-tree storage plugins.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: touge-pvc
spec:
storageClassName: gp3
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
Let’s follow the process and inspect the results.
$> kubectl apply -f touge-pvc.yaml
persistentvolumeclaim/touge-pvc created$> kubectl get pvc touge-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
touge-pvc Bound pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec 10Gi RWO gp3 10s$> kubectl describe pv pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec
Name: pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: gp3
Status: Bound
Claim: default/touge-pvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: topology.ebs.csi.aws.com/zone in [us-west-2c]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: ebs.csi.aws.com
FSType: ext4
VolumeHandle: vol-0f06e363f467b87bd
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1615144050357-8081-ebs.csi.aws.com
Events: <none>$> aws ec2 describe-volumes --volume-ids vol-0f06e363f467b87bd --region us-west-2
{
"Volumes": [
{
"Attachments": [],
"AvailabilityZone": "us-west-2c",
"CreateTime": "2021-03-07T21:42:30.268Z",
"Encrypted": true,
"KmsKeyId": "arn:aws:kms:us-west-2:REDACTED:key/REDACTED",
"Size": 10,
"SnapshotId": "",
"State": "available",
"VolumeId": "vol-0f06e363f467b87bd",
"Iops": 3000,
"Tags": [
{
"Key": "kubernetes.io/created-for/pv/name",
"Value": "pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec"
},
{
"Key": "kubernetes.io/created-for/pvc/namespace",
"Value": "default"
},
{
"Key": "kubernetes.io/created-for/pvc/name",
"Value": "touge-pvc"
},
{
"Key": "CSIVolumeName",
"Value": "pvc-a2cc33c6-f5d5-425f-bd1e-0902b82bbcec"
}
],
"VolumeType": "gp3",
"MultiAttachEnabled": false,
"Throughput": 125
}
]
}
VolumeSnapshots
VolumeSnapshots
are a pretty cool feature that’s possible with CSI. You can do things like taking a snapshot of a volume and then restore the PVC with that snapshot if your data becomes corrupt. An interesting use-case would be to create a nightly snapshot of your dev database and allow users to create a PersistentVolumeClaim
(PVC) from that snapshot to use in their personal testing. The snapshot doesn’t even need to come from inside Kubernetes!
Enabling VolumeSnapshots
It’s an add-on to the default setup so the first thing you need to do is install the CSI Snapshotter CRDs. After installing the Snapshotter CRDs you can add --set enableVolumeSnapshot=true
to the Helm install command from above and a new StatefulSet
, ebs-snapshot-controller
, will be running.
It uses a VolumeSnapshotClass
to know which CSI Plugin the snapshot requests go to so let’s create one.
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
name: ebs-csi-aws
driver: ebs.csi.aws.com # <-- The CSIDriver we defined previously
deletionPolicy: Delete
Creating VolumeSnapshots
To create a new VolumeSnapshot
create a resource on the cluster for it.
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
name: touge-snapshot
spec:
volumeSnapshotClassName: ebs-csi-aws
source:
persistentVolumeClaimName: touge-pvc
This will trigger the snapshotting process, the aws-ebs-csi-driver
will be notified, and it will create a snapshot in AWS for the EBS volume that is backing the PVC. Once again, let’s follow the process and inspect the results.
$> kubectl apply -f touge-snapshot.yaml
volumesnapshot.snapshot.storage.k8s.io/touge-snapshot created$> kubectl describe volumesnapshot touge-snapshot
Name: touge-snapshot
Namespace: default
Labels: <none>
Annotations: API Version: snapshot.storage.k8s.io/v1beta1
Kind: VolumeSnapshot
Metadata:
Creation Timestamp: 2021-03-07T21:50:40Z
Finalizers:
snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
Generation: 1
Resource Version: 135554
Self Link: /apis/snapshot.storage.k8s.io/v1beta1/namespaces/default/volumesnapshots/touge-snapshot
UID: 7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Spec:
Source:
Persistent Volume Claim Name: touge-pvc
Volume Snapshot Class Name: ebs-csi-aws
Status:
Bound Volume Snapshot Content Name: snapcontent-7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Creation Time: 2021-03-07T21:51:13Z
Ready To Use: true
Restore Size: 10Gi
If we look at the Status
we will see Bound Volume Snapshot Content Name: snapcontent-7d32eca6–2015–4a6e-a5b6–3ec48ca68005
. This tells us which VolumeSnapshotContent
is created for our VolumeSnapshot
. The VolumeSnapshotContent
is a resource that is created by the snapshot controller that represents the data the CSI Plugin created.
$> kubectl describe volumesnapshotcontents snapcontent-7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Name: snapcontent-7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Namespace:
Labels: <none>
Annotations: <none>
API Version: snapshot.storage.k8s.io/v1beta1
Kind: VolumeSnapshotContent
Metadata:
Creation Timestamp: 2021-03-07T21:50:40Z
Finalizers:
snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
Generation: 1
Resource Version: 135553
Self Link: /apis/snapshot.storage.k8s.io/v1beta1/volumesnapshotcontents/snapcontent-7d32eca6-2015-4a6e-a5b6-3ec48ca68005
UID: 0049e16f-2196-456b-9460-319ee24b3a15
Spec:
Deletion Policy: Delete
Driver: ebs.csi.aws.com
Source:
Volume Handle: vol-0f06e363f467b87bd
Volume Snapshot Class Name: ebs-csi-aws
Volume Snapshot Ref:
API Version: snapshot.storage.k8s.io/v1beta1
Kind: VolumeSnapshot
Name: touge-snapshot
Namespace: default
Resource Version: 133849
UID: 7d32eca6-2015-4a6e-a5b6-3ec48ca68005
Status:
Creation Time: 1615153873000000000
Ready To Use: true
Restore Size: 10737418240
Snapshot Handle: snap-05bc0ec2f3a65b7be
Here is where we see Snapshot Handle: snap-05bc0ec2f3a65b7be
that tells us the SnapshotID in AWS.
$> aws ec2 describe-snapshots --snapshot-ids snap-05bc0ec2f3a65b7be --region us-west-2
{
"Snapshots": [
{
"Description": "Created by AWS EBS CSI driver for volume vol-0f06e363f467b87bd",
"Encrypted": true,
"KmsKeyId": "arn:aws:kms:us-west-2:REDACTED:key/REDACTED",
"OwnerId": "REDACTED",
"Progress": "100%",
"SnapshotId": "snap-05bc0ec2f3a65b7be",
"StartTime": "2021-03-07T21:51:13.115Z",
"State": "completed",
"VolumeId": "vol-0f06e363f467b87bd",
"VolumeSize": 10,
"Tags": [
{
"Key": "CSIVolumeSnapshotName",
"Value": "snapshot-7d32eca6-2015-4a6e-a5b6-3ec48ca68005"
}
]
}
]
}
Creating a PVC from an Existing AWS Snapshot
Let’s go through the example use-case of taking an existing AWS snapshot and creating a PVC from it for someone to use inside Kubernetes.
First we need to create the VolumeSnapshotContent
that references the AWS snapshot. Using the AWS console I created snap-002e544b538087ec1
from an EBS volume that I had. To show the power of this, the volume & snapshot were created outside of Kubernetes.
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotContent
metadata:
name: my-imported-snapshot
spec:
volumeSnapshotRef:
kind: VolumeSnapshot
name: my-imported-snapshot
namespace: default
source:
snapshotHandle: snap-002e544b538087ec1 # <-- snapshot to import
driver: ebs.csi.aws.com
deletionPolicy: Delete
volumeSnapshotClassName: ebs-csi-aws
Then we need to create the VolumeSnapshot
that uses that VolumeSnapshotContent
.
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
name: my-imported-snapshot
namespace: default
spec:
volumeSnapshotClassName: ebs-csi-aws
source:
volumeSnapshotContentName: my-imported-snapshot
And apply it to the cluster.
$> kubectl apply -f touge-import-snapshot.yaml
volumesnapshotcontent.snapshot.storage.k8s.io/my-imported-snapshot created
volumesnapshot.snapshot.storage.k8s.io/my-imported-snapshot created$> kubectl get volumesnapshotcontent
NAME AGE
my-imported-snapshot 4m31s$> kubectl get volumesnapshot
NAME AGE
my-imported-snapshot 4m56s
The VolumeSnapshot
is now available to be used to create the PersistentVolumeClaim
.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-imported-snapshot-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 10Gi
dataSource:
name: my-imported-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
This is then applied to the cluster and we will have a new PVC that we can mount on our Pod.
$> kubectl apply -f touge-pvc-from-snapshot.yaml
persistentvolumeclaim/my-imported-snapshot-pvc created$> kubectl get pvc my-imported-snapshot-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-imported-snapshot-pvc Bound pvc-1ff63250-5a4f-442f-9907-171b69569c2b 10Gi RWO gp3 26s$> kubectl describe pv pvc-1ff63250-5a4f-442f-9907-171b69569c2b
Name: pvc-1ff63250-5a4f-442f-9907-171b69569c2b
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: gp3
Status: Bound
Claim: default/my-imported-snapshot-pvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 10Gi
Node Affinity:
Required Terms:
Term 0: topology.ebs.csi.aws.com/zone in [us-west-2a]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: ebs.csi.aws.com
FSType: ext4
VolumeHandle: vol-03b42ee74d7fd4f4e
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1615144050357-8081-ebs.csi.aws.com
Where I plan to go from here…
Once I had played around with the aws-ebs-csi-driver
for a few days I went ahead and implemented it in the production clusters. I haven’t yet gotten around to migrating the in-tree volumes to the new CSI based ones but there’s a bit of time for that.
Following my excitement I ended up doing a proof of concept using OpenEBS. I’m really excited about the possibilities there and will be writing about that setup soon.