We already know that Kubernetes is the No. 1 orchestration platform for container-based applications, automating the deployment and scaling of these apps, and streamlining maintenance operations. However, Kubernetes comes with its own complexity challenges. So how can an enterprise take advantage of containerization to tackle complexity and not end up with even more complexity? This article provides some of the best practices that you can implement to adopt Kubernetes.

Cross Posted from Container Journal

#1.Keep a tab on policies

Define appropriate policies for cluster access controls, service access controls, resource utilization controls, and secret access controls. By default, containers run with unbounded compute resources on a Kubernetes cluster. To limit or restrict you have to implement appropriate policies

• Use NetworkPolicy resources labels to select pods and define rules that specify what traffic is allowed to the selected pods.
• Kubernetes scheduler has default limits on the number of volumes that can be attached to a Node. To define the maximum number of volumes that can be attached to a Node for various cloud providers, use Node-specific Volume Limits.
• To enforce constraints on resource usage, use Limit Range option for appropriate resource in the namespace
• To limit aggregate resource consumption per namespace, use below Resource Quotas
  • Compute Resource Quota
  • Storage Resource Quota
  • Object Count Quota
  • Limits the number of resources based on scope defined in Quota Scopes option
  • Requests vs Limits – Each container can specify a request and a limit value for either CPU or memory.
  • Quota and cluster capacity – Expressed in absolute units
  • Limit Priority Class consumption by default – For example, restrict usage of certain high priority pods
• To allow/deny fine-grained permissions, use RBAC (Role-Based Access Control) and rules can be defined to allow/deny fine-grained permissions.
• To define & control security aspects of Pods, use Pod Security Policy (available from v1.15) to enable fine-grained authorization of pod creation and updates.
  • Running of privileged containers
  • Usage of host namespaces
  • Usage of host networking and ports
  • Usage of volume types
  • Usage of the host filesystem
  • Restricting escalation to root privileges
  • The user and group IDs of the container
  • AppArmor or seccomp or sysctl profile used by containers
• Use any of the tools like Open Policy Agent Gatekeeper policy engine to manage, author the policies.

#2.Manage Resources wisely

Use resource utilization (resource quota) guidelines to ensure the containerized applications co-exist without being eliminated due to resource violations at runtime. To enforce constraints on resource usage, use Limit Range option for appropriate resources in the namespace.

To limit aggregate resource consumption per namespace, use below Resource Quotas

• Compute Resource Quota
• Storage Resource Quota
• Object Count Quota
• Limits the number of resources based on scope defined in Quota Scopes option
• Requests vs Limits – Each container can specify a request and a limit value for either CPU or memory.
• Quota and cluster capacity – Expressed in absolute units
• Limit Priority Class consumption by default – For example, restrict usage of certain high priority pods

#3.Focus on comprehensive observability of the cluster

Currently, the Kubernetes ecosystem provides two add-ons for aggregating and reporting monitoring data from your cluster: (1) Metrics Server and (2) kube-state-metrics.

Metrics Server is a cluster add-on that collects resource usage data from each node and provides aggregated metrics through the Metrics API.kube-state-metrics service provides additional cluster information that Metrics Server does not.

Below are the key metrics and alerts that are required to monitor your Kubernetes cluster.

Consider integrating with any of the commercial monitoring tools to consume probe-generated metrics and platform-generated metrics to have comprehensive observability of the cluster.

#4.Container security management must be part of your DevOps pipeline

Continuous security must be included as part of the DevOps pipeline to ensure containers are well-managed. Use any of the below static analysis tools to identify vulnerabilities in application containers while building images for containers.

• Clair
• Trivy
• kube-bench
• Falco
• Notary

#5.Audit and compliance your cluster routinely

Routinely audit the platform for Kubernetes patch levels, secret stores, compliance against the security vulnerabilities, encryption of secret stores, storage volumes, cluster policies, role binding policies, RBAC, and user management controls.

#6.Chaos test your cluster

Proactively chaos tests your platform to ensure the robustness of the cluster. It also helps to test the stability of the containerized applications and the impact of crashing these containers. There are a wide range of the open-source tools + commercial that can be used, few of them are listed below

• Chaosblade
• Chaos Mesh
• PowerfulSeal
• chaoskube
• Chaos Toolkit
• Litmus
• Gremlin

#7.Archive and backup your cluster

Kubernetes uses etcd as its internal metadata management store to manage the objects across clusters. It is necessary to define a backup strategy for etcd and any other dependent persistent stores used within the Kubernetes clusters.

Use Velero or any of the open-source tools to backup Kubernetes resources and application data so that in cases of recovery from disaster, it can reduce the time for recovery.

#8.Manage your deployment manifests

Kubernetes follows declaration-based management hence every object or resource or instruction is described only through YAML declarative manifests. It is necessary to leverage SCM tools or create custom utilities to manage these manifests.

#9.Continuous deployment of services

kubectl style of deployments would not be possible in a large-scale production setup. Instead, you have to use some of the established open-source frameworks For e.g., Helm is specifically built for Kubernetes to manage seamless deployments via the CI-CD pipeline.

Helm uses Charts that define the set of Kubernetes resources that together define an application. You can think of charts as packages of pre-configured Kubernetes resources. Charts help you to define, install, and upgrade even the most complex Kubernetes application. These charts can describe a single resource, such as a Redis pod, or a full stack of a web application: HTTP servers, databases, and caches.

In the recent release of Helm, Releases will be managed inside of Kubernetes using Release Objects and Kubernetes Secrets. All modifications such as installing, upgrading, downgrading releases will end in having a new version of that Kubernetes Secret.

#10.Use Service mesh

Service mesh offers consistent discovery, security, tracing, monitoring, and failure handling without the need for a shared asset such as an API gateway. So if you have service mesh on your cluster, you can achieve all the below items without making changes to your application code.

• Automatic Load balancing
• Fine-grained control of traffic behavior with routing rules, retries, failovers, etc.,
• Pluggable policy layer
• Configuration API supporting access controls, rate limits, and quotas
• Service discovery
• Service monitoring with automatic metrics, logs, and traces for all traffic
• Secure service to service communication

Currently, service mesh is being offered by Linkerd, Istio, and Conduit providers.

It is necessary to choose an appropriate service mesh that is compatible with the Kubernetes cluster as well as the underlying infrastructure.

Conclusion

This article covers the key best practices that you can implement for Kubernetes adoption. However, operating Kubernetes clusters is not without its challenges.