We already know that Kubernetes is the No. 1 orchestration platform for container-based applications, automating the deployment and scaling of these apps, and streamlining maintenance operations. However, Kubernetes comes with its own complexity challenges. So how can an enterprise take advantage of containerization to tackle complexity and not end up with even more complexity? This article provides some of the best practices that you can implement to adopt Kubernetes.

Cross Posted from Container Journal

#1.Keep a tab on policies

Define appropriate policies for cluster access controls, service access controls, resource utilization controls, and secret access controls. By default, containers run with unbounded compute resources on a Kubernetes cluster. To limit or restrict you have to implement appropriate policies

• Use NetworkPolicy resources labels to select pods and define rules that specify what traffic is allowed to the selected pods.
• Kubernetes scheduler has default limits on the number of volumes that can be attached to a Node. To define the maximum number of volumes that can be attached to a Node for various cloud providers, use Node-specific Volume Limits.
• To enforce constraints on resource usage, use Limit Range option for appropriate resource in the namespace
• To limit aggregate resource consumption per namespace, use below Resource Quotas
  • Compute Resource Quota
  • Storage Resource Quota
  • Object Count Quota
  • Limits the number of resources based on scope defined in Quota Scopes option
  • Requests vs Limits – Each container can specify a request and a limit value for either CPU or memory.
  • Quota and cluster capacity – Expressed in absolute units
  • Limit Priority Class consumption by default – For example, restrict usage of certain high priority pods
• To allow/deny fine-grained permissions, use RBAC (Role-Based Access Control) and rules can be defined to allow/deny fine-grained permissions.
• To define & control security aspects of Pods, use Pod Security Policy (available from v1.15) to enable fine-grained authorization of pod creation and updates.
  • Running of privileged containers
  • Usage of host namespaces
  • Usage of host networking and ports
  • Usage of volume types
  • Usage of the host filesystem
  • Restricting escalation to root privileges
  • The user and group IDs of the container
  • AppArmor or seccomp or sysctl profile used by containers
• Use any of the tools like Open Policy Agent Gatekeeper policy engine to manage, author the policies.

#2.Manage Resources wisely 

Use resource utilization (resource quota) guidelines to ensure the containerized applications co-exist without being eliminated due to resource violations at runtime. To enforce constraints on resource usage, use Limit Range option for appropriate resources in the namespace.

To limit aggregate resource consumption per namespace, use below Resource Quotas

• Compute Resource Quota
• Storage Resource Quota
• Object Count Quota
• Limits the number of resources based on scope defined in Quota Scopes option
• Requests vs Limits – Each container can specify a request and a limit value for either CPU or memory.
• Quota and cluster capacity – Expressed in absolute units
• Limit Priority Class consumption by default – For example, restrict usage of certain high priority pods

#3.Focus on comprehensive observability of the cluster

Currently, the Kubernetes ecosystem provides two add-ons for aggregating and reporting monitoring data from your cluster: (1) Metrics Server and (2) kube-state-metrics.

Metrics Server is a cluster add-on that collects resource usage data from each node and provides aggregated metrics through the Metrics API.kube-state-metrics service provides additional cluster information that Metrics Server does not.

Below are the key metrics and alerts that are required to monitor your Kubernetes cluster.

What to monitor?	Metrics to monitor	Alert Criteria
Cluster state	Monitor the aggregated resources usage across all nodes in your cluster. Node status Desired pods Current pods Available pods Unavailable pods	Node status Desired vs. current pods Available and unavailable pods
Node resources	For each of the node monitor : Memory requests Memory limits Allocatable memory Memory utilization CPU requests CPU limits Allocatable CPU CPU utilization Disk utilization	If the node’s CPU or memory usage drops below a desired threshold. Memory limits per pod vs. memory utilization per pod Memory utilization Memory requests per node vs. allocatable memory per node Disk utilization CPU requests per node vs. allocatable CPU per node CPU limits per pod vs. CPU utilization per pod CPU utilization
Missing pod	Health and availability of your pod deployments. Available pods Unavailable pods	If the number of available pods for a deployment falls below the number of pods you specified when you created the deployment.
Pods that are not running	If a pod isn’t running or even scheduled, there could be an issue with either the pod or the cluster, or with your entire Kubernetes deployment. Pod status	Alerts should be based on the status of your pods (“Failed,” ”Pending,” or “Unknown” for the period of time you specify)
Container restarts	Container restarts could happen when you’re hitting a memory limit (ex.Out of Memory kills) in your containers. Also, there could be an issue with either the container itself or its host.	Kubernetes automatically restarts containers,  but setting up an alert will give you an immediate notification later you can analyze and set the proper limits
Container resource usage	Monitor container resource usage for containers in case you’re hitting resource limits, spikes in resource consumption,	Alerts to check if container CPU and memory usage and on limits are based on thresholds.
Storage volumes	Monitor storage to Ensure your application has enough disk space so pods don’t run out of space. Volume usage and adjust either the amount of data generated by the application or the size of the volume according to usage.	Alerts to check if available bytes, capacity crosses your thresholds. Identify persistent volumes and apply a different alert threshold or notification for these volumes, which likely hold important application data.
Control Plane – Etcd	Monitor etcd for the below parameters: Leader existence and change rate Committed, applied, pending, and failed proposals. gRPC performance.	Alerts to check if any pending or failed proposals or reach inappropriate thresholds.
Control Plane – API Server	Monitor the API server for below parameters : Rate / number of HTTP requests Rate/number of apiserver requests	Alerts to check if the rate or number of HTTP requests crosses a desired threshold.
Control Plane – Scheduler	Monitor the scheduler for the below parameters Rate, number, and latency of HTTP requests. Scheduling latency. Scheduling attempts by result. End-to-end scheduling latency (sum of scheduling).	Alerts to check if the rate or number of HTTP requests crosses a desired threshold.
Control Plane – Controller Manager	Monitor the scheduler for the below parameters: Work queue depth Number of retries handled by the work queue	Alerts to check if requests to the work queue exceed a maximum threshold.
Kubernetes events	Collecting events from Kubernetes and from the container engine (such as Docker) allows you to see how pod creation, destruction, starting, or stopping affects the performance of your infrastructure.	Any failure or exception should need to be alerted.

Consider integrating with any of the commercial monitoring tools to consume probe-generated metrics and platform-generated metrics to have comprehensive observability of the cluster.

#4.Container security management must be part of your DevOps pipeline

Continuous security must be included as part of the DevOps pipeline to ensure containers are well-managed. Use any of the below static analysis tools to identify vulnerabilities in application containers while building images for containers.

• Clair
• Trivy
• kube-bench
• Falco
• Notary

#5.Audit and compliance your cluster routinely

Routinely audit the platform for Kubernetes patch levels, secret stores, compliance against the security vulnerabilities, encryption of secret stores, storage volumes, cluster policies, role binding policies, RBAC, and user management controls.

#6.Chaos test your cluster

Proactively chaos tests your platform to ensure the robustness of the cluster. It also helps to test the stability of the containerized applications and the impact of crashing these containers. There are a wide range of the open-source tools + commercial that can be used, few of them are listed below

• Chaosblade
• Chaos Mesh
• PowerfulSeal
• chaoskube
• Chaos Toolkit
• Litmus
• Gremlin

#7.Archive and backup your cluster

Kubernetes uses etcd as its internal metadata management store to manage the objects across clusters. It is necessary to define a backup strategy for etcd and any other dependent persistent stores used within the Kubernetes clusters.

Use Velero or any of the open-source tools to backup Kubernetes resources and application data so that in cases of recovery from disaster, it can reduce the time for recovery.

#8.Manage your deployment manifests

Kubernetes follows declaration-based management hence every object or resource or instruction is described only through YAML declarative manifests. It is necessary to leverage SCM tools or create custom utilities to manage these manifests.

#9.Continuous deployment of services

kubectl style of deployments would not be possible in a large-scale production setup. Instead, you have to use some of the established open-source frameworks For e.g., Helm is specifically built for Kubernetes to manage seamless deployments via the CI-CD pipeline.

Helm uses Charts that define the set of Kubernetes resources that together define an application. You can think of charts as packages of pre-configured Kubernetes resources. Charts help you to define, install, and upgrade even the most complex Kubernetes application. These charts can describe a single resource, such as a Redis pod, or a full stack of a web application: HTTP servers, databases, and caches.

In the recent release of Helm, Releases will be managed inside of Kubernetes using Release Objects and Kubernetes Secrets. All modifications such as installing, upgrading, downgrading releases will end in having a new version of that Kubernetes Secret.

#10.Use Service mesh

Service mesh offers consistent discovery, security, tracing, monitoring, and failure handling without the need for a shared asset such as an API gateway. So if you have service mesh on your cluster, you can achieve all the below items without making changes to your application code.

• Automatic Load balancing
• Fine-grained control of traffic behavior with routing rules, retries, failovers, etc.,
• Pluggable policy layer
• Configuration API supporting access controls, rate limits, and quotas
• Service discovery
• Service monitoring with automatic metrics, logs, and traces for all traffic
• Secure service to service communication

Currently, service mesh is being offered by Linkerd, Istio, and Conduit providers.

It is necessary to choose an appropriate service mesh that is compatible with the Kubernetes cluster as well as the underlying infrastructure.

Conclusion

This article covers the key best practices that you can implement for Kubernetes adoption. However, operating Kubernetes clusters is not without its challenges.

Summary

Article Name

Kubernetes Adoption in 2024: Key Statistics

Description

Discover the latest statistics on Kubernetes adoption in 2024 and what they mean for your business.

Author

Karthik

Publisher Name

Upnxtblog

Publisher Logo