Kubernetes clusters for everyone using vcluster
I recently started playing around with a powerful Kubernetes tool called vcluster from Loft Labs. vcluster provides an easy way of creating virtual Kubernetes clusters inside of a regular cluster but scoped within a namespace. What’s really neat is that the resources created can still be restricted by the host cluster’s RBAC, quotas and other security policies. While I’ve only started to touch the surface of what vcluster can do I can already see some long-term, high-impact use-cases.

Custom Operators use-case
In my environment tenant users are not allowed to create or modify CustomResourceDefinitions (CRDs) that Custom Operators use (they can run the operator but not manage the CRDs). They have to go through a ticketing & deployment process which adds overhead for the SRE team to review and delays for the development team. It’s unfortunately a lose-lose situation but the multi-tenancy restrictions and requirements must be respected at all times. It’s even more difficult to develop for a new Custom Operator where the CRD hasn’t been fully designed yet. Using tools like kind are useful for doing development on these operators but they don’t always let you test with the application full stack running along side it.
But this is where vcluster can come to the rescue! There can be a cluster where the tenant team has a namespace, with appropriate RBAC permissions, from which they can launch a virtual Kubernetes cluster using vcluster. The team can then do their development using that virtualized cluster. Where things get interesting is if this is taken to the next level and the production application is run inside of the vcluster. Suddenly the development team can run their custom operator however they’d like and it would still respect the multi-tenant nature of the host cluster. For me running production workloads with vcluster is still a long ways off but the potential for it has me super excited. Even if running the entire application stack isn’t feasible, running the custom operator could be.
Development environment use-case
Another powerful option for vcluster is to manage developer workflows. Like git allows you to easy toggle between branches vcluster could allow you to toggle between virtual clusters per feature branch. Each feature branch could be fully deployed in isolation from each other. When the feature branch work is finish the virtual cluster is removed and all resources are automatically cleaned up. Because of the way that pods are created on the nodes the host cluster’s namespace quotas still apply. This protects against the ever frustrating resource creep that can happen.
Install and run vcluster
Setting up a virtual cluster using vcluster is wickedly straight-forward. There’s a little bit of prep work from the cluster operator team and then the tenants can manage the vcluster themselves.
Setting up the namespace
Start by creating a new namespace, in this case we’ll call it team-touge
, and give the tenant the necessary access.
kubectl create namespace team-touge
Create a RoleBinding
that gives users admin access to the team-touge
namespace. Depending on your environment you may want to use something more restrictive than admin but for my use-case it is appropriate.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: team-admin
namespace: team-touge
subjects:
- kind: Group
name: some-team-group-name
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: admin
apiGroup: rbac.authorization.k8s.io
After access to the host cluster namespace has been given the tenants can take over and manage everything from this point forward.
Setup vcluster
After the vcluster cli is installed we need to create a yaml config file that tells vcluster as run as non-root.
# vcluster.yaml
securityContext:
runAsUser: 12345
runAsNonRoot: true
We then use this config file when launching the virtual cluster called touge
inside of the team-touge
namespace.
$> vcluster create touge -n team-touge -f vcluster.yaml
[info] execute command: helm upgrade touge vcluster --repo https://charts.loft.sh --version 0.5.0-alpha.7 --kubeconfig /var/folders/rn/hrzkvjz5325dvtxz2ztzyf480000gp/T/1416406882 --namespace team-touge --install --repository-config='' --values /var/folders/rn/hrzkvjz5325dvtxz2ztzyf480000gp/T/3591452462 --values vcluster.yaml
[done] √ Successfully created virtual cluster touge in namespace team-touge. Use 'vcluster connect touge --namespace team-touge' to access the virtual cluster
After a few seconds we now have some new pods running in the team-touge
namespace.
$> kubectl get pods -n team-touge
coredns-7bbd4f6c46-pvqcg-x-kube-system-x-touge 1/1 Running 0 1m16s
touge-0 2/2 Running 0 3m30s
The touge-0
pod is vcluster with the k3s control-plane and the coredns-7bbd4f6c46-pvqcg-x-kube-system-x-touge
pod is the coredns
deployment for inside the virtual cluster. Now that vcluster is running we can connect to that virtual cluster via port-forwarding. This means that you can configure RBAC rules for whether or not a user is allowed to connect to it at all. You could also setup an Ingress for connectivity if you’d prefer.
$> vcluster connect touge -n team-touge
[done] √ Virtual cluster kube config written to: ./kubeconfig.yaml. You can access the cluster via `kubectl --kubeconfig ./kubeconfig.yaml get namespaces`
[info] Starting port-forwarding at 8443:8443
Forwarding from 127.0.0.1:8443 -> 8443
Forwarding from [::1]:8443 -> 8443
In another window I configure my environment to use this newly generated kube config and get a list of pods in the kube-system
namespace.
$> export KUBECONFIG=$(pwd)/kubeconfig.yaml
$> kubectl get pods -n kube-systemNAME READY STATUS RESTARTS AGE
coredns-7bbd4f6c46-pvqcg 1/1 Running 0 7m31s
If you look back, you’ll see this pod name matches to the coredns-7bbd4f6c46-pvqcg-x-kube-system-x-touge
that we saw earlier. vcluster will sync the pods (but not the deployment) and delegate scheduling to the host cluster. That way it ends up on a real node and can serve real workloads.
Inspecting the virtual cluster
An interesting aspect of the way the vcluster works is how it creates “fake” nodes that the workloads run on. Once a pod is scheduled on the host cluster’s node it will appear inside of the virtual cluster as well with partial data. For example, the IP address of the node will be different and the node Conditions are specific to the virtual cluster’s management of the node. The only pods it shows as running on the node are those that are running inside the virtual cluster. This keeps the data segmented from the host cluster and any other vcluster’s that may be running.
$> kubectl get nodes
NAME STATUS ROLES AGE VERSION
vmss-agent-worker-touge-cmixj000000 Ready <none> 10m v1.20.11+k3s2$> kubectl describe node vmss-agent-worker-touge-cmixj000000
Name: vmss-agent-worker-touge-cmixj000000
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=fake-vmss-agent-worker-touge-cmixj000000
kubernetes.io/os=linux
vcluster.loft.sh/fake-node=true
<snip>
Non-terminated Pods: (1 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-7bbd4f6c46-pkdgp 100m (0%) 1 (6%) 70Mi (0%) 170Mi (0%) 10m
Let’s create a secret in the virtual cluster’s kube-system
namespace.
$> kubectl create secret generic my-secret -n kube-system --from-literal=TEST=foo
secret/my-secret created$> kubectl get secret -n kube-system my-secret
NAME TYPE DATA AGE
my-secret Opaque 1 20s
But if we try to access this secret from host cluster it won’t exist.
$> kubectl get secret -n kube-system my-secret
Error from server (NotFound): secrets "my-secret" not found$> kubectl get secret -n team-touge my-secret
Error from server (NotFound): secrets "my-secret" not found
Only resources such as pods, services and ingresses are sync’d with the host cluster by default. Additional resources can be sync’d between the host and virtual cluster but it may need additional RBAC permissions that a standard user won’t have.
Installing Vault Secrets Operator
As I’ve talked about before, I love the Vault Secrets Operator so let’s install it on the new virtual cluster.
If you remember, in the host cluster, we gave users the ability to admin the team-touge
namespace but we didn’t give them access to create new namespaces. However, they do have admin access to the virtual cluster (more on that later) so a new namespace can still be created. This new namespace exists only within the virtual cluster.
$> kubectl create namespace vault-secrets-operator
namespace/vault-secrets-operator created
Next we’ll install the chart repositories and the charts. PLEASE NOTE that this is NOT how you would install Vault in a production environment.
$> helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories
$> helm repo add ricoberger https://ricoberger.github.io/helm-charts
"ricoberger" has been added to your repositories$> helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ricoberger" chart repository
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈$> helm install --namespace vault-secrets-operator vault hashicorp/vault# Follow the steps from https://learn.hashicorp.com/tutorials/vault/kubernetes-minikube?in=vault/kubernetes#initialize-and-unseal-vault to unseal Vault$> helm install --namespace vault-secrets-operator --set environmentVars\[0\].name=VAULT_TOKEN --set environmentVars\[0\].value=your-vault-token vault-secrets-operator ricoberger/vault-secrets-operator
We can now see that the application is installed along with the CRDs.
$> kubectl get crd vaultsecrets.ricoberger.de
NAME CREATED AT
vaultsecrets.ricoberger.de 2021-12-19T23:35:48Z
If we go back to the host cluster, you can see that the CRD is not installed there.
$> unset KUBECONFIG
$> kubectl get crd vaultsecrets.ricoberger.de
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "vaultsecrets.ricoberger.de" not found
A note on security
I’m still digging into this but one of the things about vcluster that isn’t clear to me is the cluster access permissions. It appears to me that all users, using the vcluster connect
, have admin access. You can restrict access to the vcluster by restricting who can port-forward or access the kubeconfig secret in the host namespace but that blanket access/restriction is limiting in scope. While for a demo/testing environment this may acceptable, you’ll want to make sure you have something more robust before using it for anything beyond that. There may be something already in place for this but I’m not seeing it at first glance.
On the plus side with security, because the virtualized cluster still has pods running within the host cluster, any rules you may have in place with a PodSecurityPolicy (note those are now deprecated) or Open Policy Agent are still enforced. If you don’t allow a pod to be deployed on a control-plane node via an OPA policy, that pod will fail to launch when vcluster creates it.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning SyncError 17s (x14 over 59s) pod-syncer Error syncing to physical cluster: admission webhook "mutating-webhook.openpolicyagent.org" denied the request: The Pod spec.tolerations[] contains a toleration for a control plane Node taint, but the Pod is not within an approved control plane Namespace
Where to next?
In this post I focused on the custom operators use-case but there are a few others that I’m playing around with. I’d really like to be able to deploy, as close as possible, the deployments from my core cluster into a virtual cluster for CI/CD. That would greatly speed up my testing as the cloud resources wouldn’t need to be created each time in order to test with a clean environment. It would also be nice to have teams test deploying their full application stack inside a virtual cluster for things like canary or blue/green deployments. Being able to fully isolate and test different versions of the applications could provide a lot of value to the application teams.
I’ll definitely be watching this project closely over the coming months!