Overview of third-party addons for EKS (Kubecost, Dynarace, Istio)
- Oleksii Bebych
- Feb 21, 2023
- 7 min read
Updated: Mar 7, 2023
AWS introduced EKS addons in the EKS v1.20. Just a few add-ons appeared back then, e.g. VPC CNI plugin, CoreDNS, and Kube-proxy. All Amazon EKS add-ons include the latest security patches, bug fixes, and are validated by AWS to work with Amazon EKS. Amazon EKS add-ons allow you to consistently ensure that your Amazon EKS clusters are secure and stable and reduce the amount of work that you need to do in order to install, configure, and update add-ons.
Later AWS added Amazon EBS CSI driver and AWS Distro for OpenTelemetry and at the end of 2022 third-party addons were officially presented in the AWS re:Invent 2022. In this post, we will take a look at several available add-ons, their capabilities and value.
EKS add-ons and Marketplace
If you chose EKS v1.24, three add-ons will be automatically installed with a cluster.
You can select different versions according to your requirements. Later you can easily update it.
There are several add-ons available at the time of writing, but this list is constantly growing.
Addons may require permissions to use AWS API. They can use either IAM role of a node where they run, or assume IAM role using IRSA approach that is preferable.
Add-on update process
The purpose was to check how smoothly the update is going. I chose Amazon VPI CNI and wanted to make sure that the update process would not brake the network and that all pods continue running.
You can select newer and older version
I monitor all pods and nodes during the VPC CNI update
The update is started
We can see that only the target application "VPC CNI" (aws-node daemonSet) was recreated and all other pods are stable and all nodes are "Ready"
Kubecost
Kubecost started in early 2019 as an open-source tool to give developers visibility into Kubernetes spend. Kubecost provides real-time cost visibility and insights by uncovering patterns that create overspending on infrastructure to help teams prioritize where to focus optimization efforts. By identifying root causes for negative patterns, customers using Kubecost save 30-50% or more of their Kubernetes cloud infrastructure costs.
You can try it for free and install it as an EKS add-on.
The only thing that you need to start is a subscription in AWS Marketplace and install the add-on.
Kubecost comes bundled with a Prometheus installation. However, if you wish to integrate with an external Prometheus deployment, provide your local Prometheus service address with this format http://..svc. Note: integrating with an existing Prometheus is only officially supported under Kubecost paid plans and requires some extra configurations on your Prometheus
$ kubectl get po -n kubecost
NAME READY STATUS RESTARTS AGE
kubecost-cost-analyzer-74955f9d46-g2m4n 2/2 Running 0 43h
kubecost-prometheus-server-f4dd75668-82whb 1/1 Running 0 43h
$ kubectl port-forward --namespace kubecost deployment/kubecost-cost-analyzer 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090Then you can visit http://localhost:9090 via your web browser
The Kubecost Cost Allocation dashboard allows you to quickly see allocated spend across all native Kubernetes concepts, e.g. namespace, k8s label, and service. It also allows for allocating cost to organizational concepts like team, product/project, department, or environment.
Here you can aggregate cost by namespace, deployment, service, and other native Kubernetes concepts. While selecting Single Aggregation, you will only be able to select one concept at a time. While selecting Multi Aggregation, you will be able to filter for multiple concepts at the same time.
The Kubecost Assets view shows Kubernetes cluster costs broken down by the individual backing assets in your cluster (e.g. cost by node, disk, and other assets). It’s used to identify spend drivers over time and to audit Allocation data. This view can also optionally show out-of-cluster assets by service, tag/label, etc.
Kubecost automatically generates recommendations you can use to save 30-50% or more on infrastructure spend
The health score starts at 100. Penalties reduce the score. There are three penalty types:
SevereErrorPenalty = 50
ErrorPenalty = 15
WarningPenalty = 3WarningPenalty is applied when:
Single Cluster (Master exists on Cluster - for kops based kubernetes deployments on AWS)
Single Region
Predictive Disk Growth crosses a 90% threshold
ErrorPenalty is applied:
Any Nodes in the Cluster are Not Ready
Any Nodes are under MemoryPressure
SevereErrorPenalty is applied:
Memory Usage exceeds 90% of Available Memory on the Cluster
Kubecost alerts allow teams to receive updates on real-time Kubernetes spend. They are configurable via the Kubecost UI or Helm values. They can be sent via email, Slack, and Microsoft Teams using Kubecost Helm chart values.
Alerts are either created to monitor specific data sets and trends, or they must be toggled on or off. The following alert types are supported:
Allocation Budget: Sends an alert when spending crosses a defined threshold
[Beta] Allocation Efficiency: Detects when a Kubernetes tenant is operating below a target cost-efficiency threshold
Allocation Recurring Update: Sends an alert with cluster spending across all or a subset of kubernetes resources.
Allocation Spend Change: Sends an alert reporting unexpected spend increases relative to moving averages
Asset Budget: Sends an alert when spend for a particular set of assets crosses a defined threshold.
Cloud Report: Sends an alert with asset spend across all or a subset of cloud resources.
Monitor Cluster Health: Used to determine if the cluster's health score changes by a specific threshold.
Monitor Kubecost Health: Used for production monitoring for the health of Kubecost itself.
Dynatrace
Dynatrace provides software intelligence to simplify cloud complexity and accelerate digital transformation. With advanced observability, AI, and complete automation, the all-in-one platform provides answers, not just data, about the performance of applications, the underlying infrastructure, and the experience of all users.
With Dynatrace, you can:
Monitor your full stack with no manual configuration. End-to-end monitoring of your AWS applications and infrastructure
Automatically discover all EC2 instances running in Availability Zones by leveraging CloudWatch API
Migrate into AWS faster with automation and intelligence
Optimize delivery pipeline with an AI-driven DevOps methodology
Improve mean time to resolution with precise root cause analysis showing causation and correlation
Analyze highly complex and dynamic ecosystems and billions of events in real-time
Out-of-the-box, Dynatrace works with Amazon EC2, Elastic Container Service, Elastic Kubernetes Service, Fargate, and serverless solutions like Lambda.
This add-on just deploys a container agent (https://github.com/dynatrace/dynatrace-operator).
$ kubectl get all -n dynatrace
NAME READY STATUS RESTARTS AGE
pod/dynatrace-operator-6d6457bc86-g5hdl 1/1 Running 1 (6h39m ago) 9h
pod/dynatrace-webhook-5fb848c58f-h8dzc 1/1 Running 0 9h
pod/dynatrace-webhook-5fb848c58f-r2cst 1/1 Running 0 9h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dynatrace-webhook ClusterIP 172.20.138.152 <none> 443/TCP 9h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dynatrace-operator 1/1 1 1 9h
deployment.apps/dynatrace-webhook 2/2 2 2 9h
NAME DESIRED CURRENT READY AGE
replicaset.apps/dynatrace-operator-6d6457bc86 1 1 1 9h
replicaset.apps/dynatrace-webhook-5fb848c58f 2 2 2 9hExtra steps should be performed for the complete configuration. First of all, you need to sign-up, you can try Dynatrace for 15 days free of charge:
There are many integrations for different clouds and workloads, e.g. you can connect AWS accounts via IAM user or IAM role:
AWS workloads appears after that:
For the Kubernetes cluster you need to create token and apply the provided manifest:
Here is an example of the downloaded dynakube.yaml, we provide token and API url for connection:
apiVersion: v1
data:
apiToken: ZHQwYzAxLlEy*********TZVTTJQQkY1
dataIngestToken: ZHQwYzAxLjVLQkIyNUNEUEM0TEV********xZRk40
kind: Secret
metadata:
name: demo
namespace: dynatrace
type: Opaque
---
apiVersion: dynatrace.com/v1beta1
kind: DynaKube
metadata:
name: demo
namespace: dynatrace
annotations:
feature.dynatrace.com/automatic-kubernetes-api-monitoring: "true"
spec:
apiUrl: https://a******5.live.dynatrace.com/api
skipCertCheck: true
oneAgent:
classicFullStack:
image:
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
env:
- name: ONEAGENT_ENABLE_VOLUME_STORAGE
value: "false"
activeGate:
capabilities:
- routing
- kubernetes-monitoring
- dynatrace-api
image: ""
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 1000m
memory: 1.5GiNew objects appear after that:
$ kubectl get all -n dynatrace
NAME READY STATUS RESTARTS AGE
pod/demo-activegate-0 1/1 Running 0 2m52s
pod/demo-oneagent-csk6m 1/1 Running 0 2m55s
pod/demo-oneagent-swmdz 1/1 Running 0 2m55s
pod/dynatrace-operator-6d6457bc86-g5hdl 1/1 Running 1 (6h39m ago) 9h
pod/dynatrace-webhook-5fb848c58f-h8dzc 1/1 Running 0 9h
pod/dynatrace-webhook-5fb848c58f-r2cst 1/1 Running 0 9h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/demo-activegate ClusterIP 172.20.128.207 <none> 443/TCP,80/TCP 2m55s
service/dynatrace-webhook ClusterIP 172.20.138.152 <none> 443/TCP 9h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/demo-oneagent 2 2 2 2 2 <none> 2m56s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dynatrace-operator 1/1 1 1 9h
deployment.apps/dynatrace-webhook 2/2 2 2 9h
NAME DESIRED CURRENT READY AGE
replicaset.apps/dynatrace-operator-6d6457bc86 1 1 1 9h
replicaset.apps/dynatrace-webhook-5fb848c58f 2 2 2 9h
NAME READY AGE
statefulset.apps/demo-activegate 1/1 2m53sKubernetes cluster, nodes and applications appears in the Dynatarce console:
Logs are also available with filtering capabilities:
Smartscape is a map for your application topology. As the Dynatrace OneAgent discovers all the components and dependencies in your application environment, Smartscape technology simultaneously builds an interactive map of how everything is interconnected:
Visualizations get built dynamically and automatically without any need for manual configuration, additional instrumentation, or scripts.
Intuitive infographics make it easy to understand the complexities of your application stack and delivery chain.
Smartscape provides 100% end-to-end observability into all application components and dependencies up, down, and across all tiers of your stack—no gaps or blind spots.
Moreover, Dynatrace has the capabilities of:
Message queue and databases monitoring
and more than 70 extensions for monitoring other popular products
Pricing is:
This is quite an interesting product with many functions and features that can be a worthy competitor to other solutions for monitoring and tracing.
Tetrate Istio Distro
Tetrate Istio Distro is an open-source project from Tetrate that provides vetted builds of Istio tested against all major cloud platforms. TID provides extended Istio version support beyond upstream Istio (release date plus 14 months). It also includes the GetMesh lifecycle and change management CLI.
The TID Istio distributions are hardened and performant and are full distributions of the upstream Istio project.
Nothing special happens, it just installs an Istio control plane and you are ready to configure your service mesh.
$ kubectl get po -n istio-system
NAME READY STATUS RESTARTS AGE
istiod-7997d87f64-t5ms7 1/1 Running 0 32h
$ kubectl api-resources | grep istio
NAME SHORTNAMES APIGROUP NAMESPACED KIND
wasmplugins extensions.istio.io true WasmPlugin
istiooperators iop,io install.istio.io true IstioOperator
destinationrules dr networking.istio.io true DestinationRule
envoyfilters networking.istio.io true EnvoyFilter
gateways gw networking.istio.io true Gateway
proxyconfigs networking.istio.io true ProxyConfig
serviceentries se networking.istio.io true ServiceEntry
sidecars networking.istio.io true Sidecar
virtualservices vs networking.istio.io true VirtualService
workloadentries we networking.istio.io true WorkloadEntry
workloadgroups wg networking.istio.io true WorkloadGroup
authorizationpolicies security.istio.io true AuthorizationPolicy
peerauthentications pa security.istio.io true PeerAuthentication
requestauthentications ra security.istio.io true RequestAuthentication
telemetries telemetry telemetry.istio.io true TelemetryConclusion
In this post, we looked at EKS add-ons from AWS Marketplace, such as Kubecost, Dynatrace, and Istio. It looks very interesting and convenient to install and manage. Of course, add-ons are already supported by Terraform and CloudFormation and in the next posts we will check others.
Comments