Advanced Kubernetes

Kubernetes is a container orchestration tool for automating the deployment, scaling and management of containerized applications. Kubernetes (K8s) has undoubtedly become the cornerstone of modern cloud infrastructure. In the previous Kubernetes post we went through the basics of Kubernetes, now lets dive into some advance Kubernetes topics.

Kubernetes Operator

Kubernetes manages the complete life cycle of stateless application in fully automated way without any extra knowledge to create/update/delete resources. Kubernetes uses the core control loop mechanism were it Observes the state, Checks for the differences between current & desired state and takes Action to update to desired state. For stateful application were storage resources such as database is present, the Kubernetes control loop process doesn't work. The stateful application can have database replicas for example which have different state and identity, so these replicas need to be synchronized and sequentially updated for database consistency. This process would vary for various databases from MySQL to Postgres. Kubernetes Operators as used by stateful application to automate these complex stateful operations to update the resources.

Kubernetes Operators use the core control loop mechanism to watch for changes in the application state and make updates. It uses Kubernetes Custom Resource Definition (CRD) which extend existing K8s API and uses app specific knowledge to automate the lifecycle of the application it operates. Each resource in Kubernetes such as Pods, Services Deployments etc is an endpoint on Kubernetes API. This endpoint stores the collection of Kubernetes objects. The Kubernetes API is used to create, update, delete and get these Kubernetes objects. Kubernetes CRD is a way of extending the Kubernetes API to develop custom resources and install in any Kubernetes cluster. Once the resource is installed in Kubernetes cluster, we can create its object using Kubernetes CLI (kubectl). CRD only allows to store and fetch the data, but when combined with custom controller we can have a custom declarative API, which allows the keep the object's current state in sync with object's desired state.

Custom operator are created for each applications and can be found in OperatorHub.io. Operator SDK allows developers easily Kubernetes native applications.

Operator consists of three parts, a controller, a state and a resource.

The controller manages the (internal or external) resources, and contains the logic to get from current state to the desired state. It manages the resources using the Kubernetes loop mechanism: Observe, Check and Adjust the state. State is the state of the resource called as CRD. The controller uses the state that is stored in Kubernetes control plane (Etcd) to ensure that the resource is at the requested state. State is described as CRD with declarative description in form of YAML. Customer Kubernetes Operator is an application which defines an API for its state, stores the state in Etcd (Control plane) and uses the Kubernetes API to manage, update and delete the desired resource. The resource can be Kubernetes native resource e.g. a Pod or any other external resource outside Kubernetes e.g. Printer.

Create a new PDFDocument CustomResourceDefinition.

$ kubectl apply -f pdf-crd.yaml

$ kubectl get pdfdocument

$ kubectl get pdf

$ kubectl api-resources | grep pdf

$ kubectl proxy --port=8080

$ curl localhost:8080/apis/k8s.startkubernetes.com/v1/namespaces/default/pdfdocuments

$ kubectl get crd

Create a custom controller using KubeBuilder.

$ go mod init k8s.startkubernetes.com/v1

$ kubebuilder init

$ kubebuilder init --domain dev.emprovise --repo=github.com/pranav-patil/sample-oprator

$ kubebuilder create api --group k8s.startkubernetes.com --version v1 --kind PdfDocument

Admission Webhooks

In Kubernetes any events e.g. create/delete pods, scale deploy etc are requested through the API Server. Admission Webhooks allows us to intercept these requests at different stages. There are two types of Admission web hooks, Mutating web hooks and Validation web hooks.

1. Mutating web hooks intercepts the requests (object/YAML) before it hits the API server and allows to inject changes to the request object.
2. Validation web hooks allows to accept or reject the request to the API service, example for policy enforcement.

https://github.com/marcel-dempers/docker-development-youtube-series/tree/master/kubernetes/admissioncontrollers/introduction

inject changes or after the request is being validated by the API server.

Pod Security Admission

Pod security admission is an implementation of the Kubernetes pod security standards. Use pod security admission to restrict the behavior of pods. Pods that do not comply with the pod security admission defined globally or at the namespace level are not admitted to the cluster and doesn't run. Globally, the privileged profile is enforced, and the restricted profile is used for warnings and audits.

Kubernetes v1.21 and later use the Pod Security Standards (PSS) and Pod Security Admission (PSA) controls to manage Pod security, instead of the older PodSecurityPolicy. Pod Security Policy (PSP) was deprecated in Kubernetes v1.21, and removed from Kubernetes in v1.25. Pod Security Standards (PSS) and Pod Security Admission (PSA) were turned on by default in Kubernetes v1.23, and replace Pod Security Policies (PSP) in Kubernetes v1.25 and above. For Kubernetes v1.22 PodSecurity has to be enabled manually, add --feature-gates=PodSecurity=true to below configuration:

• Kubelet level: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
• Kube-API level: /etc/Kubernetes/manifest/kube-apiserver.yaml

Use the below command to check the configuration in API server. There are multiple feature gates which can be enabled in API server config.

$ kubectl describe pod -n kube-system kube-apiserver-master-node

The PodSecurity restrictions can be applied at Namespace, Label or Cluster level using AdmissionConfig file.

apiVersion: v1
kind: Pod
kind: Namespace
metadata:
 name: dev
 labels:
   pod-security.kubernetes.io/{{MODE}}={{LEVEL}}

Below are the Pod Security Levels to be defined.

• Privileged - It is unrestricted policy, full admin permissions, widely open and used for privilege escalations.
• Baseline - Allows the default minimally and is specified at Pod configuration. The Allow privileged escalation must be false.
• Restricted - Heavily restricted policy and Pod hardening.

All the Pod Security levels work at the Security context level under spec.containers[*].securityContext.

Below are the Pod Security Modes to be applied.

• Enforce - It's a rule, if violated Pod creation is rejected.
• Audit - If violated, it records event in audit.log, Pod creation allowed
• Warn - If violated then warning message, Pod creation allowed
  

Policy application is controlled based on labels on the namespace. The following labels are supported:

• pod-security.kubernetes.io/enforce: <policy level>
• pod-security.kubernetes.io/enforce-version: <policy version>
• pod-security.kubernetes.io/audit: <policy level>
• pod-security.kubernetes.io/audit-version: <policy version>
• pod-security.kubernetes.io/warn: <policy level>
• pod-security.kubernetes.io/warn-version: <policy version>

The two controllers work independently using the following processes to enforce security policies:

• The security context constraint controller may mutate some security context fields per the pod’s assigned SCC (Security context constraints). The security context constraint controller also validates the pod’s security context against the matching SCC.
• The pod security admission controller validates the pod’s security context against the pod security standard assigned to the namespace.

Kyverno

Kyverno is the policy management