How to deploy Helm charts to an EKS cluster through AWS CloudFormation

Problem statement

There are several ways to create an EKS cluster in AWS:

• Web console or CLI

• EKSctl tool

• Terraform, CloudFormation or other IaC tools

• Third-party products

In most cases an empty kubernetes cluster is not enough. We still may need an Ingress Controller, Cluster autoscaler, External DNS, Prometheus, etc. included in a default cluster set.

As you probably know, when an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator (with system:masters permissions). Initially, only that IAM user can make calls to the Kubernetes API server.

For example, in the Terraform we can use a Kubernetes/Helm provider or even “local-exec”, that will deploy required Kubernetes resources into the cluster, patch “aws-auth” ConfigMap in order to add extra IAM entities for a cluster administration. But, unfortunately, there is no such basic functionality in AWS CloudFormation. If we have a multi-account (multi-region) AWS environment, we have to use CloudFormation Stack Sets. In this case we can not use any things other than CloudFormation templates. We can create an EKS cluster itself, a managed node group, and the Fargate profile, but we can not manipulate with Kubernetes entities like pods, deployments, configmaps, services, etc. Potentially this can be solved via Lambda-backed Custom Resource, but there is another way.

The proposed solution

AWS CloudFormation has a registry, where we can find Amazon or Third-party public extensions. The AWS Quick Start team has developed and published several CloudFormation extensions related to Kubernetes:

• AWSQS::EKS::Cluster - github link

• AWSQS::Kubernetes::Get - link

• AWSQS::Kubernetes::Helm - link

• AWSQS::Kubernetes::Resource - link

Before using an extension you need to activate it in the relevant AWS region.

You have to choose an appropriate Execution role, created in advance. Every CloudFormation extension has a template for the Execution role with required policies (example here). You can also configure logging for the extension, it may be useful for debugging if anything goes wrong with Stack provisioning. You can enable or disable automatic extension updates.

As I mentioned before, we use CloudFormation Stack Sets for multi-account (multi-region) provisioning, so we’ve created a separate CloudFormation template that deploys needed IAM execution roles + activates needed extensions.

// omitted code
Resources:
  EKSClusterExtensionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: [resources.cloudformation.amazonaws.com, cloudformation.amazonaws.com, lambda.amazonaws.com]
            Action: sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: ResourceTypePolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - "eks:CreateCluster"
                  - "eks:DeleteCluster"
                  - "iam:PassRole"
                  - "sts:AssumeRole"    
                  - "lambda:InvokeFunction"
                  - "lambda:CreateFunction"
                  // other required permissions 
                Resource: "*"

  EKSClusterExtension:
    Type: AWS::CloudFormation::TypeActivation
    DependsOn: EKSClusterExtensionRole
    Properties: 
      AutoUpdate: false
      ExecutionRoleArn: !GetAtt EKSClusterExtensionRole.Arn
      PublicTypeArn: !Sub "arn:aws:cloudformation:${AWS::Region}::type/resource/408988dff9e863704bcc72e7e13f8d645cee8311/AWSQS-EKS-Cluster"
// omitted code

The same procedure will be for other required extensions, e.g. AWSQS::Kubernetes::Helm. Once the extension is activated we can use it to create EKS cluster with many more options compared with default CloudFormation resource for EKS, e.g. extra IAM roles/users added to aws-auth configMap, control plane logging configuration and API endpoint configuration.

// omitted code
Resources:
  AdminEKSRole:
    Type: "AWS::IAM::Role" # Extra role added to aws-auth
    Properties:
    // omitted code

  EksKMSKey:
    Type: "AWS::KMS::Key" # KMS key for envelope encryption 
    // omitted code


  EKSCluster:
    Type: "AWSQS::EKS::Cluster"
    Properties:
      Name: !Sub "${ProjectName}-${ProductName}-${Env}-${ClusterName}"
      Version: !Ref KubernetesVersion
      RoleArn: !GetAtt serviceRole.Arn 
      LambdaRoleName: !ImportValue …… # extra role needed if EKS API endpoint is private (check docs)
       
      ResourcesVpcConfig:
        SubnetIds:
          - !ImportValue ...
           // omitted code
        SecurityGroupIds:
          - !Ref EKSSecurityGroup
        EndpointPrivateAccess: true
        EndpointPublicAccess: false
      EnabledClusterLoggingTypes: !If [ LoggingEnabled, !Ref EKSClusterLoggingTypes, !Ref "AWS::NoValue" ]
      KubernetesApiAccess: # Extra roles/users added to aws-auth configmap
        Roles:
          - Arn: !GetAtt AdminEKSRole.Arn
            Username: "AdminRole"
            Groups: ["system:masters"]
          - Arn: !ImportValue ... # Execution role for Helm extension
            Username: "DeployRole"
            Group: ["system:masters"]
      EncryptionConfig: !If
        - EnableEncryption
        - - Resources: [ secrets ]
            Provider:
              KeyArn:  // omitted code
 
  MetricsServer: # Example for applying kubernetes manifest
    Type: "AWSQS::Kubernetes::Resource"
    Properties:
      ClusterName: !Ref EKSCluster
      Namespace: kube-system
      Url: https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml

  KubeStateMetrics: # Example for deploying Helm chart
    Type: "AWSQS::Kubernetes::Helm"
    Properties:
      ClusterID: !Ref EKSCluster
      Name: kube-state-metrics
      Namespace: kube-state-metrics
      Repository: https://prometheus-community.github.io/helm-charts
      Chart: prometheus-community/kube-state-metrics
      ValueYaml: |
        prometheus:
          monitor:
            enabled: true

Conclusion

AWS CloudFormation extensions described in this post help us deploy EKS clusters with all necessary Kubernetes resources in a fully automated way across all AWS accounts/OUs within an AWS Landing Zone where EKS is required. Whole configuration is stored in the GIT repository and deployed by AWS CodePipeline + AWS CloudFormation Stack Sets.