Money Forward Developers Blog

Cross-account KEDA deployment

What is KEDA? 🤔

KEDA is a Kubernetes-based Event Driven Autoscaler.

With KEDA, you can drive the scaling of any deployment in Kubernetes based on events, and custom metrics and also scale down the deployment to 0 if needed. You can find more descriptions here.

Why do we need KEDA? 😃

Generally in Kubernetes, we can control the scaling of a workload only based on the metrics released by the metrics server like CPU usage, memory usage, or some application-related metrics.

In some cases we might need event-driven scaling of workloads, that’s when KEDA comes into the picture. 🥂

To further simplify it, you can think about AWS Lambda which triggers on events, similarly, KEDA triggers scaling on events. These events can message queue events and CloudWatch metrics, and KEDA also lets you define custom metrics for scaling.

Deployment environment 🌇

We have two AWS accounts, one with our EKS cluster and the other containing the application-specific AWS resources like SQS, S3, etc. We want KEDA to scale our workloads based on the number of messages present in the SQS queue. Please look at the below image for a better understanding.

In this article, we are going to use a Kubernetes ServiceAccount and IRSA(IAM roles for service accounts) to grant our workloads cross-account access to the AWS resources.

Understanding AWS permissions 🔐

There are two things to be considered while creating an AWS role, The first one is the Trust relationship and the second thing is the policy.

Trust relationship defines, what entities can assume the created role, whereas policy defines the resources that can be accessible.

In the case of cross-account access, we need to add AssumeRole permission to both account roles to enable the cross-account access. More about this here.

Understanding KEDA deployment 😕

KEDA is deployed in two parts.

  • KEDA operator
  • KEDA resources

KEDA has its custom resources so we need to deploy the KEDA operator in our EKS cluster before the resources so that we have the definition of those resources in the cluster.

Before starting I would also like to mention some things that go on under the hood in KEDA for permissions.

There are two types of KEDA resources we are going to deploy,

  • TriggerAuthentication → Specifies the type of authentication used. In the case of AWS, it can use the role specified in the service account of deployment to authenticate.
  • ScaledObject → Specifies the deployment to be scaled, the resource to be used for detecting the scaling event and the TriggerAuthentication resource to be used.

KEDA operator is responsible for detecting the scaling events and notifying the ScaledObject to upscale or downscale the deployment. Now we encounter a problem here, how will the KEDA operator gain access to the AWS resources in Account B?

The first approach can be creating a user in Account B with access to the resources and letting KEDA use that user, but it is not a good practice to create a user when accessing AWS resources from our EKS.

So we are going to create some IAM Roles and Kubernetes ServiceAccounts to enable KEDA to access AWS resources.

Deploying the KEDA operator 🚙

Before deploying the KEDA operator, we will create an IAM role for the KEDA operator with the following config. Let’s name our role keda-accountA-role for now.

Trust relationship:-

This enables the service account to access the role using the OIDC.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": ""
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          ":sub": "system:serviceaccount::"
        }
      }
    }
  ]
}

Policy:-

Just allows anyone to assume this role but no other permissions.

{
  "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ],
}

Now we will deploy the KEDA operator using helm, For those who don’t know, helm is a package manager for Kubernetes, more details here.

Make a helm manifest(values.yaml) to direct the KEDA to use the role we just created.

podIdentity:
  aws:
    irsa:
      enabled: true
      audience: "sts.amazonaws.com"
      roleArn: ""

Apply your helm manifest:-

# Run this command if you haven't added the KEDA repository to your local helm config
$ helm repo add kedacore https://kedacore.github.io/charts
$ helm repo update

# KEDA is deployed in a separate namespace
$ kubectl create namespace keda

# Install or upgrade KEDA
$ helm upgrade --install --namespace keda --version  -f  keda kedacore/keda

That’s it 👯, now we have our operator running, using our IAM role.

I won’t dive in too much depth but to debug any problem you can always use kubectl describe to check the KEDA deployments.

Deploying the KEDA resources

For KEDA resources we will create an IAM role in Account B, which will enable access to the SQS. Let’s name it keda-sqs-role.

Trust relationship:-

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": ""
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          ":aud": "sts.amazonaws.com",
          ":sub": "system:serviceaccount::"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": ""
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Policy:-

{
  "Statement": [
    {
      "Action": [
        "sqs:GetQueueAttributes",
      ],
      "Effect": "Allow",
      "Resource": [
        "",
      ]
    },
  ],
}

Now our IAM role has been created we can deploy our KEDA resources using this role.

First, we will create a Kubernetes manifest for deploying the TriggerAuthentication resource, which uses the aws-eks provider to fetch the service account from the pods of deployment we want to scale.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-auth
spec:
  podIdentity:
    provider: aws-eks

Next, create the ScaledObject which will use the above TriggerAuthentication resource.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: aws-scaledobject
spec:
  scaleTargetRef:
    name: 
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: keda-auth
      metadata:
        queueURL: 
        queueLength: ""
        awsRegion: ""
  minReplicaCount: 
  maxReplicaCount:

You can apply these two resources to your cluster and it’s done. Now you can sit back and enjoy the scaling based on the AWS SQS queue length 😄.

Thank you for reading… :).

Published-date