The 4Cs of Kubernetes Security
To effectively secure Kubernetes from threats, approach security from the perspective of the 4Cs model. The 4Cs model is code, container, cluster, and cloud security, and is a roadmap for addressing security concerns within Kubernetes architecture.
Code Security
Code refers to the applications executed by containers. Code presents a significant attack surface in Kubernetes environments, not only from inadvertent introduction of bugs resulting in vulnerabilities, but also from potentially vulnerable third-party libraries.
To secure code running on the Kubernetes platform, start by preventing unauthorized access to code with basic network security and access controls as the first line of defense. Implement secure coding practices, execute regular scans and tests with dedicated tooling, and adhere to OWASP coding guidelines.
Container Security
Containers contain images, which themselves consist of the base image, operating system, container configuration, dependencies, and runtime necessary to execute application code.
Containers running in pods must use trustworthy base images and runtime environments, each of which may be targeted by malicious actors. Verify the originating image repository is secure, minimize the code base to limit the number of third-party libraries in use, scan container images for vulnerabilities, and secure pods with access controls and network policies to restrict communication between pods.
Cluster Security
Kubernetes architecture is organized into clusters. Kubernetes clusters consist of pods, and each pod contains one or more containers operating on the same local network. The design of cluster security hinges on careful design of access policies and security configurations.
Cluster security involves securing containers and applications running within, the control plane (the API, scheduler, datastore, and controllers), and the broader network in which the cluster runs.
Cloud Security
The cloud layer is the physical datacenter or cloud infrastructure which runs Kubernetes, commonly infrastructure as code (IaC) platforms or managed Kubernetes services.
Cloud providers offer guidelines and security best practices for implementing proper access controls. Reduce risks to the cloud infrastructure by implementing least privilege procedures on resources, scanning for cloud vulnerabilities or misconfigurations, and access controls to restrict which users can interact with Kubernetes.
Generally, administrators can expect to implement security measures across all stages of the application lifecycle: from initial coding and testing, production deployment, and ongoing operations within the cluster.
Key Principles for Kubernetes Security
The following principles and security countermeasures can help ensure the resilience of containerized applications running on Kubernetes:
Role-Based Access Control (RBAC)
In Kubernetes, RBAC controls access to resources within the cluster, and defines the actions a user or group is allowed or not allowed to perform. RBAC is used to isolate team access, restrict operations, control admin access, and manage service account permissions.
Roles are used to grant access to resources within a single namespace, while cluster roles are broader in scope and define access across namespaces. Clearly, not every user should have full, unrestricted access to all resources. In evaluating Kubernetes roles and permissions, refer to the principle of least privilege (PoLP) as a general guideline.
When using RBAC, administrators should tend to prefer namespace-specific permissions instead of cluster-wide permissions, specifying access controls for every Kubernetes object and namespace. Allow access only when necessary for specific tasks, and nothing more.
Network Policies Enforcement
Containers communicate with external services and with each other over the network. Containerized applications often use cluster networks extensively. To better understand how applications interact with other systems, and to identify anomalous communications, implement monitoring for active network traffic and compare it to the traffic allowed by Kubernetes network policy.
To secure network connectivity, organizations must enforce network policies that limit communication to necessary services, preferring the minimum required for workloads to function correctly. This recommendation applies to both inbound and outbound traffic to the cluster, and traffic within the cluster.
Encrypt network traffic using virtual private networks (VPNs) and TLS. To enhance container security, deploy firewalls within the environment to add another protective layer, and implement network segmentation and isolation of resources to reduce the attack surface and contain breaches.
Enforce Pod Security Admission (PSA)
PSA, the successor of the Pod Security Policy (PSP), enforces security policies, known as Pod Security Standards (PSS) within Kubernetes. As a built-in feature, PSA eliminates the need for third-party tools, simplifying security by defining and adhering to a set of standards. It places restrictions on unsecure configurations, reducing the attack surface.
The PSS enforced by Pod Security Admission defines three security profiles for workloads. The Privileged profile enforces no restrictions, and should not be used unless absolutely necessary. The Baseline profile provides a minimum amount of security for applications, while Restricted enforces security best practices.
PSA verifies that pods are configured according to these profiles, enforcing compliance. In general, administrators should set either the Baseline or Restricted policy for pods to ensure the security of the cluster.
Secure the Control Plane
The Kubernetes control plane is responsible for controlling the cluster. It manages the cluster state, health, and configuration data, ensuring the containers run with all necessary resources. Because of its importance and complexity, the control plane is considered somewhat difficult to configure, and has consequently become a prime target for attackers.
The control plane consists of the following components:
- etcd: The etcd key-value database stores configuration information and cluster state data. Ensure that etcd has encryption enabled, that communication is restricted to the API server unless absolutely necessary, and that clients use certificate-based authentication.
- kube-controller-manager: The Controller Manager daemon runs clusters and controls its functions. Securing the Controller Manager involves restricting network access, implementing TLS for all API traffic, limiting resource usage on the cluster, and minimizing the privileges used by containers.
- kube-scheduler: The Scheduler manages and provisions new containers. Security steps include deactivating its profiling capability to reduce its attack surface, and verifying that the Scheduler’s IP address is not bound to an insecure IP.
- kube-apiserver: The API Server acts as a frontend, handling internal and external requests. In brief, secure the API Server by restricting access by external IPs, enforcing strong authentication mechanisms, and applying TLS encryption. Refer to the “Secure the Kubernetes API” section below for more detail.
Secure the Kubernetes API Server
External users access the Kubernetes control plane via the API, so ensuring its security and limiting access to unauthorized users is extremely important.
To secure the Kubernetes API, start by regulating requests targeting the server to make sure that unauthorized API requests don’t successfully access the cluster. Consider using third-party authentication providers, which will enable multi-factor authentication (MFA). Use OAuth 2.0 connectors or OpenID Connect (OIDC) providers to secure cluster access.
Limit access to public networks, and implement Transport Layer Security (TLS) encryption for data in transit through the API server. Also confirm that the Kubernetes datastore, etcd, which directly communicates with the API server, is itself appropriately secured.
Image Security Scanning
Images form the basis of clusters and pods created. These images must be regularly scanned for security vulnerabilities, ensuring containers created from the images do not inherit the image’s security flaws.
Image scanning tools may be used to identify vulnerabilities in the base image upon which containers are built, and applications for libraries included within container images. This is done by scanning the base image and all packages against a vulnerability database.
Ensure that access to image registries is restricted to prevent tampering, and scan all images throughout continuous integration/continuous deployment (CI/CD) pipeline stages. Adding scanning in the CI/CD pipeline helps to ensure that containers are properly configured, updated, and free from malware.
Encrypt Secrets and Communication
Secrets are a form of sensitive information. In Kubernetes, the most common secrets are typically passwords, OAuth tokens, SSH keys, or other credentials. Secrets stored in cleartext – in YAML configuration files, container images, or within documents on containers – represent a critical risk, and endanger the security of the entire cluster.
Kubernetes offers a built-in object, the Kubernetes Secret, to store this data securely. Using Secret objects, administrators can separate confidential information from application code.
Because Kubernetes Secrets handle privileged information, they are frequently targeted by hackers. Secrets must be encrypted at rest, and access to them should be restricted to only the components and users that require them. Implement secret scanning practices or systems to identify and remediate accidental secret exposure.
Limit Node Exposure
Kubernetes Nodes come in two main types: master nodes, and worker nodes. Master nodes run the basic cluster services, including the API server, scheduler, controller, and the etcd datastore. Worker nodes run the applications within the cluster.
Kubernetes clusters are composed of nodes which run either Linux or Windows operating systems (OS). Many hardening techniques that generally apply to hardening these OSes are valid in the context of Kubernetes nodes:
- Install only the applications and libraries required for the cluster to operate correctly, and nothing more.
- Restrict access to administrative / root accounts, using them only when necessary.
- Deploy real-time monitoring tools to detect security breaches in progress.
- Linux nodes can be hardened with tools like SELinux or AppArmor.
Furthermore, the Center for Internet Security (CIS) provides a set of security benchmarks for both master and worker nodes. Using a tool like Kube-bench, administrators can scan their clusters and evaluate them against CIS benchmarks. These scans commonly provide recommendations to remediate configurations that fall outside of CIS best practices.
Enable Audit Logs
Audit logs are important to maintain both operations and security. These logs offer actionable insights into cluster activity, and allow for swift detection of anomalous activity. While Kubernetes includes audit logging capabilities as a built-in feature, they are not enabled by default. Upon activation, all actions are recorded within the cluster.
Cluster administrators must configure an audit policy that defines the events to log, and external tools for storage, management, and analysis of those logs. This ensures that security issues are rapidly detected, while providing information needed for incident response (IR) teams to investigate any breaches that occur.
Run Workloads with Minimal Privileges
Kubernetes workloads can be complex and difficult to adequately secure. Workloads are dynamic, transitioning between on-premises and cloud environments, each with their own network security controls. Moreover, automated CI/CD processes frequently deploy new services or new versions on cluster nodes, intensifying the complexity.
Generally, administrators should never trust workloads. Apply fine-grained security policies to restrict communication between workloads and third-party applications. This reduces the likelihood of lateral movement of threats, and helps to maintain compliance.
Network security definitions should be built into workloads, ensuring they are portable across Kubernetes distributions and data centers. This way, wherever the workload runs, it carries along the security definitions.
Lastly, configure perimeter security tools to continuously monitor the IP addresses and ports used by workloads, allowing for real-time identification of suspicious behavior.
Feedback