AWS Elastic Kubernetes Service (EKS)  sandbox 

Feb 15, 2024 / Notes / 2 min
Table of Contents

Workshop - https://www.eksworkshop.com/docs/introduction/

Update kubeconfig to access EKS

aws eks update-kubeconfig --name my-cluster --region us-east-1

If cluster admin role is different from current role used to aut

aws eks update-kubeconfig --name my-cluster --role-arn arn:aws:iam::123456789000:role/eksrole --region us-east-1

More here - https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

Workload AWS access

Provide access to AWS workloads like S3 buckets, SNS topics etc via short lived temporary credentials.

IRSA (IAM roles for service accounts)

Pods can use service accounts which was backed by IAM roles and their permissions. Needs an annotation on k8s service account which refers to the IAM role arn. Principal in the role’s trust policy is the EKS cluster’s oidc provider.

You can add conditions to lock down which service accounts are able to use this role.

annotation: eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role

Example trust policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:default:my-service-account",
                    "oidc.eks.region-code.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

More here https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html

Pod Identities

Pods can assume an IAM role directly via k8s service account, without the need of an annotation on the service account. It need a pod identity association at cluster level (needs namespace, service account name). An agent runs on each node which makes this possible. The role’s trust policy is different than when using IRSA. The principal here is service: pods.eks.amazonaws.com

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

Following cdk code snippet (typescript) creates such a role, and adds a pod identity association.

  const podIdentityServicePrincipal = new iam.SessionTagsPrincipal(new iam.ServicePrincipal("pods.eks.amazonaws.com"))
  const workloadPodIdentityRole = new iam.Role(this, "WorkloadPodIdentityRole", {
      assumedBy: podIdentityServicePrincipal,
    });
    workloadPolicy.attachToRole(WorkloadPodIdentityRole);
    new eks.CfnPodIdentityAssociation(this, "WorkloadPodIdentityRoleAssociation", {
      clusterName: cluster.clusterName,
      roleArn: workloadPodIdentityRole.roleArn,
      namespace: "workload",
      serviceAccount: "workload-podidentity",
    });

More here https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-eks-podidentityassociation.html

Networking

https://aws.github.io/aws-eks-best-practices/networking/

https://github.com/aws/amazon-vpc-cni-k8s

Advanced Networking

https://aws.amazon.com/blogs/containers/eks-vpc-routable-ip-address-conservation/

eks-vpc-secondary-cidr

  • can use both Fargate and EC2 clusters to launch nodes.
  • STS endpoint for the deployment region must be enabled.
  • worker nodes must have a private DNS entry or a node not found error will occur.
  • node IAM role must be present in the aws-auth-cm.yaml file when using self-managed Amazon Linux nodes.

Ref: https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html, https://docs.aws.amazon.com/eks/latest/userguide/eks-connector.html

Add-ons

Kubecost