Top Kubernetes Security Best Practices: Securing Kubernetes Workloads with OPA & OPA Gatekeeper in Amazon EKS

Introduction

As the adoption of container orchestration platforms like Kubernetes increases, so does the need for robust security measures. Open Policy Agent (OPA) and OPA Gatekeeper are powerful tools that help enforce policy-based security and governance in Kubernetes clusters. In this blog post, we will explore what OPA and OPA Gatekeeper are, their benefits, and how to implement them in Amazon Elastic Kubernetes Service (EKS) to enhance the security of your workloads.

What is Open Policy Agent (OPA)?

Open Policy Agent (OPA) is an open-source, general-purpose policy engine that enables policy enforcement across various domains, including cloud infrastructure, microservices, and Kubernetes. OPA uses the Rego language to define policies that express complex logic and rules governing access, configuration, and behavior.

OPA Gatekeeper

OPA Gatekeeper is a Kubernetes-native implementation of OPA that provides a policy enforcement mechanism for Kubernetes resources. It acts as an admission controller, intercepting requests to the Kubernetes API server before they are persisted in the cluster. This allows Gatekeeper to evaluate custom policies and enforce them in real-time, preventing the creation of non-compliant resources.

OPA decouples policy decision-making from policy enforcement. When your software needs to make policy decisions it queries OPA and supplies structured data (e.g., JSON) as input. OPA accepts arbitrary structured data as input.In the context of a development platform running on Amazon EKS, platform teams and administrators need a way of being able to set policies to adhere to governance and security requirements for all workloads and teams working on the same cluster. Examples of standard use cases for using policies via OPA Gatekeeper are listed below:

• Which users can access which resources.
• Which subnets egress traffic is allowed to.
• Which clusters a workload must be deployed to.
• Which registries binaries can be downloaded from.
• Which OS capabilities a container can execute with.
• Which times of day the system can be accessed at.

OPA generates policy decisions by evaluating the query input and against policies and data. OPA and Rego are domain-agnostic so you can describe almost any kind of invariant in your policies. Policy decisions are not limited to simple yes/no or allow/deny answers. Like query inputs, your policies can generate arbitrary structured data as output.

Key Terminology

• OPA Constraint Framework – Framework that enforces CRD-based policies and allow declaratively configured policies to be reliably shareable
• Constraint – A Constraint is a declaration that its author wants a system to meet a given set of requirements. Each Constraint is written with Rego, a declarative query language used by OPA to enumerate instances of data that violate the expected state of the system. All Constraints are evaluated as a logical AND. If one Constraint is not satisfied, then the whole request is rejected.
• Enforcement Point – Places where constraints can be enforced. Examples are Git hooks, Kubernetes admission controllers, and audit systems.
• Constraint Template – Templates that allows users to declare new constraints
• Target – Represents a coherent set of objects sharing a common identification and/or selection scheme, generic purpose, and can be analyzed in the same validation context

Benefits of OPA & OPA Gatekeeper

1. Policy-based Security: OPA allows you to define custom policies tailored to your organization’s security requirements. These policies can enforce rules on resources, such as Pods, Deployments, ConfigMaps, and more, ensuring that only approved configurations are deployed.

2. Compliance & Governance: With OPA Gatekeeper, you can ensure that your Kubernetes cluster complies with industry standards and best practices. This helps maintain governance across multiple teams and projects, reducing the risk of security breaches and misconfigurations.

3. Real-time Enforcement: As an admission controller, OPA Gatekeeper enforces policies in real-time. This means that any non-compliant resource requests are blocked before they impact the cluster, reducing the attack surface and improving overall cluster security.

Implementing OPA & OPA Gatekeeper in Amazon EKS

Step 1: Set up an Amazon EKS Cluster

Before implementing OPA and OPA Gatekeeper, you need an Amazon EKS cluster up and running. You can create a cluster using the AWS Management Console or AWS CLI, following the official documentation. Following are the steps:

 

 

Step 2: Install OPA Gatekeeper

To install OPA Gatekeeper, you can use Kubernetes manifests or the Gatekeeper Helm chart. I’m going to deploy a released version of Gatekeeper in your cluster with a prebuilt image using the following command:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

To validate if Gatekeeper has been successfully installed, use the following command

kubectl get pods -n gatekeeper-system

If you notice the gatekeeper-audit-6584df88df-nsf28 pod is created when we deploy the OpaGatekeeperAddOn. The audit functionality enables periodic evaluations of replicated resources against the Constraints enforced in the cluster to detect pre-existing misconfigurations. Gatekeeper stores audit results as violations listed in the status field of the relevant Constraint. The gatekeeper-controller-manager is simply there to manage the OpaGatekeeperAddOn.

Once OPA Gatekeeper pods are in ‘Running’ state, monitor Audit controller and Controller manager component logs for webhook requests that are being issued by the Kubernetes API server.

Step 3: Define and Enforce Policies

With Gatekeeper installed, you can define custom policies in Rego language and apply them to your EKS cluster. Policies can range from simple resource whitelisting to complex access control rules. For instance, you can create policies to enforce specific labels, limit resource requests, or ensure that only approved container images are used.

Gatekeeper uses the OPA Constraint Framework to describe and enforce policy. Lets take an example of enforcing labels on the objects. You must first define a ConstraintTemplate, which describes both the Rego that enforces the constraint and the schema of the constraint.

Example constraint template that requires all labels described by the constraint to be present:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

Let’s install above ConstraintTemplate with the following command:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/basic/templates/k8srequiredlabels_template.yaml

Once the ConstraintTemplate is created,Constraints are then used to inform Gatekeeper that the admin wants a ConstraintTemplate to be enforced, and how.

Below constraint uses the K8sRequiredLabels constraint template above to make sure the gatekeeper label is defined on all namespaces

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-gk
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["gatekeeper"]

Let’s install above Constraint with the following command:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/basic/constraints/all_ns_must_have_gatekeeper.yaml

Step 4: Test Policies and Iteratively Refine

After defining policies, it’s crucial to test them thoroughly in a staging environment before applying them to production. This iterative approach allows you to identify false positives and ensure that policies do not inadvertently disrupt legitimate workflows.

Lets try out the below sample

cat > example.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: test-opa
spec: {}
EOF
kubectl create -f example.yaml

Output :

The request was denied by the Kubernetes API because it did not comply with the constraint imposed by OPA Gatekeeper that all namespace objects created must have a value set for the owner label.

Additionally, check the Controller manager logs to see the webhook requests sent by the Kubernetes API server for validation and mutation, as well as the Audit logs to check for policy compliance on objects that already exist in the cluster.

Step 5: Monitor and Maintain

Continuously monitor the OPA Gatekeeper logs and cluster behavior to detect any policy violations or potential issues. Regularly update and refine policies as your application and infrastructure evolve.

Audit component performs periodic evaluations of existing resources against constraints, detecting pre-existing misconfigurations.

Sample Audit Logs :

Sample Controller Manager logs:

Conclusion

Congratulations !!! We learnt how to leverage OPA Gatekeeper to implement fine-grained policies in Kubernetes clusters, enhancing overall security while also simplifying compliance and audit requirements.

By incorporating OPA and OPA Gatekeeper into your Amazon EKS clusters, you gain a powerful security toolset to enforce custom policies, strengthen governance, and reduce the risk of misconfigurations. The flexibility of OPA and its seamless integration with Kubernetes make it an excellent choice for security-conscious organizations. Take advantage of OPA and OPA Gatekeeper to create a secure and compliant Kubernetes environment in your AWS EKS clusters.