One of the nags many people get from running Kubernetes CIS security benchmarks is not using proper CA validation for kubelet serving certificates.
Rule 1.1.21: Verify kubelet's certificate before establishing connection.
Why is that important? By default, each kubelet will generate its own serving certificate as a self-signed certificate. The nagging is good as using self-signed certificates makes it easy to have MITM type of breaches.
Enabling Automatic Certificate Bootstrapping
So luckily this is something that is kinda designed into Kubernetes system components. So we can enable
serverTLSBootstrap: true in kubelet config file and everything will fall in place automagically. Almost, but no cigar. From kubelet point of view this is all we need as it will make kubelet to create a proper CSR waiting to be approved and once approved it will take the properly signed certificate and use it as the serving certificate. But the problem is that there's nobody doing the approving automatically. :(
Currently Kubernetes controller manager does NOT approve kubelet serving certificate CSRs automatically. This is mentioned in the documentation:
To use RotateKubeletServerCertificate operators need to run a custom approving controller, or manually approve the serving certificate requests.
Manually approve? No thanks. So we need some "custom approving controller".
Rubber Stamper to the Rescue
As we, as in all users of Pharos, want to have capability to run Pharos in dynamic environments so there needs to be some controller that automatically approves the certificate requests. This is very important e.g. for environments where nodes can be dynamically created by autoscaling or other such functionality.
So we've created kubelet-rubber-stamp as a fully automated kubelet serving CSR approver. The process looks something like this:
The validation has few of steps:
- Recognize that the CSR request is for kubelet serving cert
- CSR has been created to adhere to kubelet certificate "rules"
- has proper addresses
- has proper x509 usage policies defined
- has proper names for both org and common name
- SubjectAccessReview passes, the given node/kubelet has proper RBAC rules bound.
These checks are actually pretty similar checks what the out-of-box kubelet client certificate approver does.
In a cluster with kubelet-rubber-stamper active, this is what happens during the cluster bootstrap:
$ kubectl get csr NAME AGE REQUESTOR CONDITION csr-5rg65 10m system:node:demo-pharos-worker-0 Approved,Issued csr-7lh9z 10m system:node:demo-pharos-master-0 Approved,Issued csr-qt86h 10m system:node:demo-pharos-worker-2 Approved,Issued csr-wrh4x 10m system:node:demo-pharos-worker-1 Approved,Issued
All the nodes (actually kubelet on the node) has requested a certificate and rubber-stamper has auto-approved those. Neat.
Custom Controller, lot of Work?
In all honesty, much of the "heavy-lifting" in our custom CSR approving controller is actually managed by the fantastic operator-sdk framework. It makes creating these kind of custom controllers really straight-forward. In this case we only need to wire it up so that we watch
CertificateSigningRequest objects in
certificates.k8s.io/v1beta1 and act upon new CSRs.
As this operator is Golang based we just bake it into a slim image and deploy it onto a cluster. One thing to note of course is that the deployment needs to create also proper RBAC rules so that the rubber-stamper has enough privileges to actually approve the CSRs.
Starting from Pharos 2.2.0 kubelet-rubber-stamp is integrated into the Kubernetes bootstrapping and configuration process. So all your kubelets now come with fully automated certificate bootstrapping for both client and serving certificates.