Managing access to Kube-API via service accounts
In the world of Kubernetes, we have two types of accounts that can interact with the Kube-API. We have the user accounts, these are us humans that will authenticate using a kube-config file, and then we have service accounts which are processes, and these guys run inside our pods. In the case where your application, which is running on a Kubernetes cluster, needs to interact with other Kubernetes objects, then our application can use this service account and its accompanying token, to retrieve the required information about the objects from the API. More on this token later.
But why would you need to interact with other Kubernetes objects in the clusters; what would be the use case? Well, there are a few that come to mind. Suppose you have a config-reloader application that needs to monitor when a configMap is updated, for it to reload the main application. In that case, the config-reloader application will need a service account (and the appropriate RBAC rules) which will allow it to monitor the relevant configMap. Another use-case will be pod monitoring or logging tools that need to retrieve the current running pods, for example. However, the bottom line is, that the majority of our apps that run on Kubernetes don’t need to interact with the Kube-API.
Here’s the interesting part
It would seem that Kubernetes assumes otherwise. Kubernetes has an AdmissionController, quite appropriately named, The ServiceAccount Admission Controller - which injects some additional parameters into the pod spec before finally creating the object. If you do not specify a service account for your pod, this ServiceAccount AdmissionController will assign it the default service account. Any service account is also issued a token, which the admission controller mounts inside your pod. This token can be used to access the Kubernetes API. Now, the access of this default token is quite limited, but still, it is considered best practice only to provide access where required.
To see this in action; if we export the pod specification of any running pod in our cluster, we’ll notice a volume mount that we never specified:
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-cchkp
readOnly: true
And its accompanying volume; looks something like this:
volumes:
- name: kube-api-access-cchkp
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
As mentioned, the access might be limited, which we’ll cover shortly, and on top of that, the token does expire. In newer Kubernetes versions (1.20) the token now has a field expirationSeconds, and the Kubelet will rotate this token periodically. So this is not all bad news. Built-in security in Kubernetes is improving.
The Token
In the event of your application being compromised, the first thing the attacker will probably do is look for the service account token to try to do more damage to the cluster.
Here’s a Dockerfile that we’ll be using for our investigation.
> cat Dockerfile.h2c
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y curl
CMD /bin/bash
Once we’ve built and pushed that to a repo that our cluster can reach, let’s start it up in our cluster and start digging around.
> kubectl -n h2c run -it --rm testpod --image=<accountNum>.dkr.ecr.<region>.amazonaws.com/ubuntu16.04:test01 -- /bin/bash
Let’s take a look at that token and cert that Kubernetes mounted on our behalf:
root@testpod:/# ls -lah /var/run/secrets/kubernetes.io/serviceaccount/
total 0
drwxrwxrwt 3 root root 140 Jul 8 08:03 .
drwxr-xr-x 3 root root 28 Jul 8 08:03 ..
drwxr-xr-x 2 root root 100 Jul 8 08:03 ..2022_07_08_08_03_13.561282802
lrwxrwxrwx 1 root root 31 Jul 8 08:03 ..data -> ..2022_07_08_08_03_13.561282802
lrwxrwxrwx 1 root root 13 Jul 8 08:03 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 16 Jul 8 08:03 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 12 Jul 8 08:03 token -> ..data/token
The Kubernetes API is exposed internally to the cluster via a service, called kubernetes in the default namespace, and its parameters are injected into each running pod, via environment variables
# printenv | grep KUBERNETES
KUBERNETES_PORT=tcp://192.168.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_HOST=192.168.0.1
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_ADDR=192.168.0.1
KUBERNETES_PORT_443_TCP=tcp://192.168.0.1:443
Incorporating all of this info into a CURL command and query the API server from within the pod.
root@testpod:/# export TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
root@testpod:/# export CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
root@testpod:/# curl -s https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/apis --header "Authorization: Bearer $TOKEN" --cacert $CACERT
{
"kind": "APIGroupList",
"apiVersion": "v1",
"groups": [
{
"name": "apiregistration.k8s.io",
"versions": [
{
"groupVersion": "apiregistration.k8s.io/v1",
"version": "v1"
}
],
"preferredVersion": {
"groupVersion": "apiregistration.k8s.io/v1",
"version": "v1"
}
},
...
Keep in mind that it’s the token that can be used to access the Kube-API server, however, RBAC policies (roles and rolebinding) dictate what can be accessed via the API. And our service account isn’t able to do much at this point. Here we can see we’re trying to access the events API endpoint which we don’t have access to, and we’re getting a 403 Forbidden error.
root@testpod:/# curl -s https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/events.k8s.io/v1 --header "Authorization: Bearer $TOKEN" --cacert $CACERT
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:serviceaccount:h2c:default\" cannot get path \"/events.k8s.io/v1\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
[...]
The Solution
Of course, we don’t want our Kube-API accessed unnecessarily, and why have credentials for our API lying around inside our pods for no reason. To prevent access tokens from being mounted in our pods that don’t need them we can define a custom service account and set the automountServiceAccountToken field to false on our deployment or in the service account yaml.
automountServiceAccountToken
We can specify the following parameter in the pod spec of all our workloads that don’t need to access the API.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: testpod
name: testpod
namespace: h2c
spec:
replicas: 1
selector:
matchLabels:
app: testpod
template:
metadata:
labels:
app: testpod
spec:
automountServiceAccountToken: false <<-----
containers:
- command:
- sleep
- "30000"
image: <accountNum>.dkr.ecr.<region>.amazonaws.com/testpod:001
name: test-app
Once we’ve started our testpod, and exec into the pod, we can see if we can find the API token again.
root@testpod-7df66fb447-d82tm:/# ls -lah /var/run/secrets/kubernetes.io/serviceaccount/
ls: cannot access '/var/run/secrets/kubernetes.io/serviceaccount/': No such file or directory
We can see that we don’t have a service account folder created inside our pod. If we look at the pod spec you’ll notice that the volumeMount and volume weren’t injected into our pod. Progress!
[...]
spec:
automountServiceAccountToken: false <<--- token not mounted
containers:
- command:
- sleep
- "30000"
image: awsAccountID.dkr.ecr.af-south-1.amazonaws.com/testpod:001
imagePullPolicy: IfNotPresent
name: test-app
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
enableServiceLinks: true
[...]
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: testpod <<--- non-default SA
serviceAccountName: testpod
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
status:
conditions:
- lastProbeTime: null
[...]
Alternatively, we can also specify the same parameter in the service account definition.
apiVersion: v1
kind: ServiceAccount
metadata:
name: testpod
namespace: h2c
automountServiceAccountToken: false
Conclusion
Ensuring we follow the principle of least privilege, we should prevent unnecessary access to the Kube-API server, and thus prevent access tokens from being injected into our pods.
It is a small, yet important improvement to our overall application security stance. Improving our security even marginally will put us in a better position than what we were in yesterday, and constantly making even tiny security improvements will ensure our cluster and the workloads that run within are kept as secure as possible.
Keep in mind that nothing in security is of a “set-it-and-forget-it” nature, and there is no silver bullet either, you need to monitor constantly, and fill the gaps where needed, with the right tool for the job.