How to enable Kubernetes cluster autoscaling in AWS

In this blog post, you will learn how to configure autoscaling for a Kubernetes cluster running in AWS.

In the most common scenarios web applications are not always under the same workload. The number of users varies depending on the time of a day, week, or even year, so to save resources and money it’s mandatory to scale the underlying infrastructure as per needs. When your application runs on Kubernetes you may want to use Autoscaler, which is a software component that ensures all pods have enough memory and CPU to run and there are no unneeded nodes in a cluster.

In the next sections, you will see step by step how to install it on a Kubernetes cluster running in AWS.

Prerequisites

In order to accomplish the task, you would need the following command-line tools to be installed on your local computer. Please refer to the documentation for detailed installation and configuration instructions.

1. helm: https://helm.sh/docs/intro/install/

2. AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

3. eksctl: https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html

AWS configuration

Before we start let’s set up a few environment variables, which will be used in further command-line commands:

$ export AWS_REGION=eu-central-1
$ export CLUSTER_NAME=arn:aws:eks:$AWS_REGION:<ACCOUNT-ID>:cluster/<NAME>
$ export EKS_CLUSTER_NAME=<NAME>

The former sets the default AWS region name for all AWS CLI commands and the latter name of our EKS cluster (remember to replace your AWS account id and actual name).

First, we need to create a policy that will allow cluster-autoscaler to access information about Auto Scaling Groups (ASG) that are managing nodes in our cluster.

The policy should look more or less as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}

Let’s save it to a file as-iam-policy.json and create a policy in our AWS account:

$ export ROLE_ARN=$(aws iam create-policy \
    --policy-name K8SAutoscalerPolicy \
    --policy-document file://as-iam-policy.json \
    | jq -r .Policy.Arn)

From the command output note down the printed role ARN, it will be required in the next step.

Now, we must create an IAM role which will have attached the above policy and can be assumed by the Kubernetes service account, so that it can call AWS APIs without a need to provide credentials to it (that required OIDC Federated Authentication, so make sure Kubernetes cluster has this feature enabled). To create a role we will use eksctl and the following command:

$ eksctl create iamserviceaccount \
    --cluster $CLUSTER_NAME \
    --namespace=kube-system \
    --name=cluster-autoscaler-sa \
    --attach-policy-arn=$ROLE_ARN \
    --override-existing-serviceaccounts \
    --approve

Replace <CLUSTER-NAME> with the name of your AWS EKS cluster and <ROLE-ARN> with role ARN obtained previously.

Helm Chart Deployment

If everything on the AWS side of things is prepared we can go ahead and deploy the Helm chart with cluster-autoscaler. Before we do that, we must first ensure we have added the chart repository:

$ helm repo add autoscaler https://kubernetes.github.io/autoscaler

Now we can execute the following command:

$ helm upgrade --install cluster-autoscaler autoscaler/cluster-autoscaler \
    --set "autoDiscovery.clusterName=$EKS_CLUSTER_NAME" \
    --set "awsRegion=$AWS_REGION" \
    --set "cloudProvider=aws" \
    --set "rbac.serviceAccount.create=false" \
    --set "rbac.serviceAccount.name=cluster-autoscaler-sa" \
    --namespace=kube-system

Passing the rbac.serviceAccount.create=false parameter will disable creating service account defined in chart template and setting rbac.serviceAccount.name=cluster-autoscaler-sa ensures cluster-autoscaler will use our manually created service account.

If everything went well you should be able to verify if cluster-autoscaler is running by querying your cluster for its pods using the following command:

$ kubectl --namespace=kube-system get pods -l "app.kubernetes.io/name=aws-cluster-autoscaler,app.kubernetes.io/instance=cluster-autoscaler"

And that’s it. Now your nodes will be added or removed depending on the load. One thing to keep in mind is that Autoscaler will respect ASG settings, so it won’t create (or remove) nodes above (or below) configured threshold.