top of page
premier.png
Search

Retired third-party CloudFormation extensions. Registering a private extension.

Problem statement

A long time ago, we used public third-party CloudFormation extensions to deploy the EKS cluster with deployed Helm charts as part of a single CloudFormation template. AWS introduced many cool things since then, for example, EKS add-ons, so such an approach may not be entirely relevant nowadays. But in our case, the customer returned with the request to deploy the same code in a different AWS Region. For context, we used a Control Tower Customization Pipeline and consequently deployed an extensive set of AWS resources. First of all, third-party public CloudFormation extensions:

### omitted code

Resources:
  EKSClusterExtensionRole:
    Type: AWS::IAM::Role
    Properties:

### omitted code

   EKSClusterExtension:
    Type: AWS::CloudFormation::TypeActivation
    DependsOn: EKSClusterExtensionRole
    Properties: 
      AutoUpdate: false
      ExecutionRoleArn: !GetAtt EKSClusterExtensionRole.Arn
      PublicTypeArn: !Sub "arn:aws:cloudformation:${AWS::Region}::type/resource/408988dff9e863704bcc72e7e13f8d645cee8311/AWSQS-EKS-Cluster"

### omitted code

  HelmResourceExecutionRole:
    Type: AWS::IAM::Role

### omitted code

  HelmCustomResource:
    Type: AWS::CloudFormation::TypeActivation
    DependsOn: HelmResourceExecutionRole
    Properties:
      AutoUpdate: false
      ExecutionRoleArn: !GetAtt HelmResourceExecutionRole.Arn
      PublicTypeArn: !Sub "arn:aws:cloudformation:${AWS::Region}::type/resource/408988dff9e863704bcc72e7e13f8d645cee8311/AWSQS-Kubernetes-Helm"
      Type: "RESOURCE"

Once they were deployed, we could see them in the extensions list:


And could proceed with the custom EKS stack:

AWSTemplateFormatVersion: '2010-09-09'

Resources:
### omitted code

  EKSCluster:
    Type: "AWSQS::EKS::Cluster"
    Properties:
      Name: !Sub "${ProjectName}-${ClusterName}"
      Version: !Ref KubernetesVersion
      RoleArn: !GetAtt serviceRole.Arn
      LambdaRoleName: !ImportValue
        'Fn::Sub': "${ProjectName}-EKSClusterExtensionRoleName"
      ResourcesVpcConfig:
        SubnetIds:
          - !ImportValue
            'Fn::Sub': "${ProjectName}-PrivateSubnetA"
          - !ImportValue
            'Fn::Sub': "${ProjectName}-PrivateSubnetB"
          - !ImportValue
            'Fn::Sub': "${ProjectName}-PrivateSubnetC"
        SecurityGroupIds:
          - !Ref EKSSecurityGroup
        EndpointPrivateAccess: true
        EndpointPublicAccess: false
      EnabledClusterLoggingTypes: !If [ ControlPlaneLoggingEnabled, !Ref EKSClusterLoggingTypes, !Ref "AWS::NoValue" ]
      KubernetesApiAccess:
        Roles:
          - Arn: !GetAtt AdminEKSRole.Arn
            Username: "AdminRole"
            Groups: ["system:masters"]
          - Arn: !ImportValue
              'Fn::Sub': "${ProjectName}-${Env}-HelmResourceExecutionRoleARN"
            Username: "DeployRole"
            Groups: ["system:masters"]
          - Arn: !Sub "{{resolve:ssm:/iam/deploy-role/arn}}"
            Username: JenkinsRole
            Groups: ["system:masters"]
          - Arn: !GetAtt NodeInstanceRole.Arn
            Username: "system:node:{{EC2PrivateDNSName}}"
            Groups:
              - "system:nodes"
              - "system:bootstrappers"

### omitted code

  NginxIngress:
    DependsOn:
      - Listener
      - EKSCluster
    Type: "AWSQS::Kubernetes::Helm"
    Condition: DeployNginxIngress
    Properties:
      ClusterID: !Sub "${ProjectName}-${ClusterName}"
      Namespace: ingress-nginx
      Chart: ingress-nginx
      Repository: "https://kubernetes.github.io/ingress-nginx"
      ValueYaml: !Sub |
        controller:
          service:
            enabled: true
            type: NodePort
            nodePorts:
              http:  ${HttpIngressPort}
              https: ${HttpsIngressPort}

So, long story short, CloudFormation extensions were deprecated, and our template stopped working with the following error:

Template contains errors.: Template format error: Unrecognized resource types: [AWSQS::EKS::Cluster, AWSQS::Kubernetes::Helm]

Investigation

So the first place where we go to check the issue is a CloudFormation console, Activated extensions. Every extension should contain a valid Schema, which looks like this (example for "AWSQS::EKS::Cluster"):


But we saw this (Schema is not valid anymore, and we can not use this extension):


And the GitHub repository is archived:


Proposed solution

  1. The best solution would be to remove extensions and rewrite the CloudFormation stack, but this one did not fit due to time constraints.

  2. Alternative - as was said in the edited Schema above: "Fork the code and use it as a private resource type (at your own risk)."


So we chose the second one as a fast and temporary solution. We did not build extensions from sources; we just took an artifact from the public S3 bucket and created a Private CloudFormation Extension via the CF template:

---
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  ProjectName:
    Type: String
    AllowedPattern: "^[a-zA-Z][a-zA-Z0-9_-]*$"

Resources:
  HelmResourceExecutionRole:
    Type: AWS::IAM::Role
    Metadata:
      cfn_nag:
        rules_to_suppress:
          - id: F38
            reason: 'official documentation https://github.com/aws-quickstart/quickstart-amazon-eks-cluster-resource-provider/blob/main/execution-role.template.yaml'
    Properties:
      MaxSessionDuration: 8400
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - "lambda.amazonaws.com"
                - "resources.cloudformation.amazonaws.com"
            Action: sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: ResourceTypePolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - "secretsmanager:GetSecretValue"
                  - "kms:Decrypt"
                  - "eks:DescribeCluster"
                  - "s3:GetObject"
                  - "sts:AssumeRole"
                  - "iam:PassRole"
                  - "ec2:CreateNetworkInterface"
                  - "ec2:DescribeNetworkInterfaces"
                  - "ec2:DeleteNetworkInterface"
                  - "ec2:DescribeVpcs"
                  - "ec2:DescribeSubnets"
                  - "ec2:DescribeRouteTables"
                  - "ec2:DescribeSecurityGroups"
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                  - "lambda:UpdateFunctionConfiguration"
                  - "lambda:DeleteFunction"
                  - "lambda:GetFunction"
                  - "lambda:InvokeFunction"
                  - "lambda:CreateFunction"
                  - "lambda:UpdateFunctionCode"
                  - "cloudformation:ListExports"
                  - "ecr:GetAuthorizationToken"
                  - "ecr:BatchCheckLayerAvailability"
                  - "ecr:GetDownloadUrlForLayer"
                  - "ecr:BatchGetImage"
                Resource: "*"

  HelmCustomResource:
    Type: AWS::CloudFormation::ResourceVersion
    Properties:
      TypeName: AWSQS::Kubernetes::Helm
      SchemaHandlerPackage: s3://aws-quickstart/quickstart-helm-resource-provider/awsqs-kubernetes-helm.zip
      ExecutionRoleArn: !GetAtt HelmResourceExecutionRole.Arn

  HelmCustomResourceDefaultVersion:
    Type: AWS::CloudFormation::ResourceDefaultVersion
    Properties:
      TypeVersionArn: !Ref HelmCustomResource

  EKSClusterExtensionRole:
    Type: AWS::IAM::Role
    Metadata:
      cfn_nag:
        rules_to_suppress:
          - id: F38
            reason: 'official documentation https://github.com/aws-quickstart/quickstart-amazon-eks-cluster-resource-provider/blob/main/execution-role.template.yaml'
    Properties:
      MaxSessionDuration: 8400
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: [resources.cloudformation.amazonaws.com, cloudformation.amazonaws.com, lambda.amazonaws.com]
            Action: sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: ResourceTypePolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - "sts:GetCallerIdentity"
                  - "eks:CreateCluster"
                  - "eks:DeleteCluster"
                  - "eks:DescribeCluster"
                  - "eks:ListTagsForResource"
                  - "eks:UpdateClusterVersion"
                  - "eks:UpdateClusterConfig"
                  - "eks:TagResource"
                  - "eks:UntagResource"
                  - "iam:PassRole"
                  - "sts:AssumeRole"
                  - "lambda:UpdateFunctionConfiguration"
                  - "lambda:DeleteFunction"
                  - "lambda:GetFunction"
                  - "lambda:InvokeFunction"
                  - "lambda:CreateFunction"
                  - "lambda:UpdateFunctionCode"
                  - "ec2:DescribeVpcs"
                  - "ec2:DescribeSubnets"
                  - "ec2:DescribeSecurityGroups"
                  - "ec2:CreateNetworkInterface"
                  - "ec2:DeleteNetworkInterface"
                  - "ec2:DescribeNetworkInterfaces"
                  - "kms:CreateGrant"
                  - "kms:DescribeKey"
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:DescribeLogGroups"
                  - "logs:DescribeLogStreams"
                  - "logs:PutLogEvents"
                  - "cloudwatch:ListMetrics"
                  - "cloudwatch:PutMetricData"
                Resource: "*"

  EKSClusterExtension:
    Type: AWS::CloudFormation::ResourceVersion
    Properties:
      TypeName: AWSQS::EKS::Cluster
      SchemaHandlerPackage: s3://aws-quickstart/quickstart-amazon-eks-cluster-resource-provider/awsqs-eks-cluster.zip
      ExecutionRoleArn: !GetAtt EKSClusterExtensionRole.Arn

  EKSClusterExtensionDefaultVersion:
    Type: AWS::CloudFormation::ResourceDefaultVersion
    Properties:
      TypeVersionArn: !Ref EKSClusterExtension

So, the old CloudFormation stack was changed from subscription to the public third-party extension to the creation of the Private CF extension:


Extensions names were used the same for further compatibility:


Conclusion

Cloudformation extensions are not used frequently, so if you face an issue with them, finding a solution is not an ordinary task. If you use a public third-party CloudFormation extension, it can be retired at any moment, and your IaC will stop working. We used extensions for deploying the EKS cluster with Helm charts (the story is here). So, as a fast solution with minimal changes in infrastructure, we took the code and created the same CloudFormation extension, but Privately registered. We used a standard CloudFormation resource, but you can also use AWS CLI. I would avoid using the CloudFormation extension without a special need, but if you use it, remember what was written in this post.

Comments

bottom of page