Skip to content

Tamedia Kubernetes as a Service (KaaS) Terraform Module

Opinionated batteries included Terraform module to deploy Kubernetes in AWS. Includes:

Managed Addons:

  • EBS CSI
  • VPC CNI
  • CoreDNS
  • KubeProxy

Core components (installed by default):

Additional components (optional):

Integrations (optional):

  • Okta
  • PagerDuty
  • Slack

Requirements

The module needs some resources to be deployed in order to operate correctly:

IAM service-linked roles

Release new kubernetes version

important Each new kubernetes version needs it's own release. This is due to the fact that we should not skip kubernetes versions during a cluster upgrade.

To release a new Kubernetes version, follow these steps:

  1. Update the version file:
  2. Open the K8S_VERSION file located in the root of the repository.

  3. Update the version number to the next Kubernetes version.

  4. Commit the Changes:

  5. Commit the changes to the K8S_VERSION file with a meaningful commit message following the release proces. For example: 

    git add K8S_VERSION
git commit -m "feat! update Kubernetes version to 1.30"

  6. Push the Changes:

  7. Push the changes to the main branch, the release workflow will automatically run. This workflow will:

    • Read the updated Kubernetes version from the K8S_VERSION file.
    • Determine the new module version based on the commit message.
    • Create a new release with the updated module version and the kubernetes version as metadata. The format would be X.Y.Z+A.B where X.Y.Z is the module version and A.B is the kubenetes control plane version.

  8. Verify the Release:

  9. Check the GitHub Actions page to ensure the release workflow completed successfully.
  10. Verify that the new module version is available in the Terraform Registry.

Usage

module "k8s_platform" {
  source = "tx-pts-dai/kubernetes-platform/aws"
  # Pin this module to a specific version to avoid breaking changes
  # version = "0.0.0"

  name = "example-platform"

  vpc = {
    enabled = true
  }

  tags = {
    Environment = "sandbox"
    GithubRepo  = "terraform-aws-kubernetes-platform"
  }
}

See the Examples below for more use cases

Explanation and description of interesting use-cases

Why this module?

  • To provide an AWS account with a K8s cluster with batteries included so that you can start deploying your workloads on a well-built foundation
  • To encourage standardization and common practices
  • To ease maintenance

Reloader

The Stakater Reloader is a Kubernetes controller that automatically watches for changes in ConfigMaps and Secrets and triggers rolling restarts of the associated deployments, statefulsets, or daemonsets when these configurations are updated. This functionality ensures that applications deployed within a Kubernetes cluster always reflect the latest configuration without manual intervention.

When an application relies on configuration data or sensitive information stored in ConfigMaps or Secrets, and these resources are modified, Reloader automates the process of applying these changes by updating the relevant pods. Without Reloader, such changes would require a manual pod restart or redeployment to take effect.

Reloader is deployed by default on the cluster but is used as on demand via annotations.

Considering this kubernetes deployment and the required annotation: 

kind: Deployment
metadata:
  name: foo
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    metadata:

Reloader will now watch for updates and manage rolling restart of pods for this specific deployment.

Examples

  • Complete - Includes creation of VPC, k8s cluster, addons and all the optional features.
  • Datadog - EKS deployment with Datadog Operator integration
  • Disable-Addons - EKS + Karpenter deployment with all addons disabled
  • Lacework - EKS deployment with Lacework integration
  • Network - VPC deployment with custom subnets for kubernetes
  • Simple - Simplest EKS deployment with default VPC, addons, ... creation

Cleanup example deployments

Destroy Workflow - This manual workflow destroys deployed example deployments by selection the branch and the example to destroy.

Contributing

< issues and contribution guidelines for public modules >

Pre-Commit

Installation: install pre-commit and execute pre-commit install. This will generate pre-commit hooks according to the config in .pre-commit-config.yaml

Before submitting a PR be sure to have used the pre-commit hooks or run: pre-commit run -a

The pre-commit command will run:

  • Terraform fmt
  • Terraform validate
  • Terraform docs
  • Terraform validate with tflint
  • check for merge conflicts
  • fix end of files

as described in the .pre-commit-config.yaml file

Requirements

Name Version
terraform >= 1.7.0
aws ~> 5.42
helm ~> 2.12
kubectl >= 2.0.2
kubernetes >= 2.27
time >= 0.11

Providers

Name Version
aws ~> 5.42
helm ~> 2.12
kubernetes >= 2.27
time >= 0.11

Modules

Name Source Version
acm terraform-aws-modules/acm/aws 5.1.1
addons aws-ia/eks-blueprints-addons/aws 1.21.0
amp terraform-aws-modules/managed-service-prometheus/aws 3.0.0
argocd ./modules/argocd n/a
downscaler tx-pts-dai/downscaler/kubernetes 0.3.1
ebs_csi_driver_irsa terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks 5.55.0
eks terraform-aws-modules/eks/aws 20.36.0
fluent_operator ./modules/addon n/a
grafana ./modules/addon n/a
karpenter terraform-aws-modules/eks/aws//modules/karpenter 20.36.0
karpenter_security_group ./modules/security-group n/a
network ./modules/network n/a
okta_secrets ./modules/addon n/a
pagerduty_secrets ./modules/addon n/a
prometheus_irsa terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks 5.55.0
prometheus_operator_crds ./modules/addon n/a
prometheus_stack ./modules/addon n/a
slack_secrets ./modules/addon n/a
ssm ./modules/ssm n/a
vpc_cni_irsa terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks 5.55.0

Resources

Name Type
aws_cloudwatch_log_group.fluentbit resource
aws_iam_policy.fluentbit resource
aws_route_table_association.karpenter resource
aws_security_group_rule.eks_control_plane_ingress resource
aws_subnet.karpenter resource
helm_release.cluster_secret_store resource
helm_release.karpenter_crd resource
helm_release.karpenter_release resource
helm_release.karpenter_resources resource
helm_release.reloader resource
kubernetes_annotations.monitoring resource
time_sleep.wait_on_destroy resource
time_static.timestamp_id resource
aws_availability_zones.available data source
aws_iam_policy_document.fluentbit data source
aws_iam_roles.iam_cluster_admins data source
aws_iam_roles.sso data source
aws_region.current data source
aws_route53_zone.base_domain_zone data source
aws_route_tables.private_route_tables data source

Inputs

Name Description Type Default Required
acm_certificate ACM certificate configuration for the domain(s). Controls domain name, alternative domain names, wildcard configuration, and validation behavior.
Options include:
- domain_name: Primary domain name for the certificate. If not provided, uses base_domain from other configuration.
- subject_alternative_names: List of additional domain names to include in the certificate.
- wildcard_certificates: When true, adds a wildcard prefix (*.) to all domains in the certificate.
- prepend_stack_id: When true, prepends the stack identifier to each domain name. Only works after random_string is created.
- wait_for_validation: When true, Terraform will wait for certificate validation to complete before proceeding.		 
object({
    domain_name               = optional(string)
    subject_alternative_names = optional(list(string), [])
    wildcard_certificates     = optional(bool, false)
    prepend_stack_id          = optional(bool, false)
    wait_for_validation       = optional(bool, false)
  })
 {} no
argocd Argo CD configurations 
object({
    # Hub specific
    enable_hub        = optional(bool, false)
    namespace         = optional(string, "argocd")
    hub_iam_role_name = optional(string, "argocd-controller")

    helm_values = optional(list(string), [])
    helm_set = optional(list(object({
      name  = string
      value = string
    })), [])

    # Spoke specific
    enable_spoke = optional(bool, false)

    hub_iam_role_arn  = optional(string, null)
    hub_iam_role_arns = optional(list(string), null)

    # Common
    tags = optional(map(string), {})
  })
 {} no
aws_load_balancer_controller AWS Load Balancer Controller configurations any {} no
base_domain Base domain for the platform, used for ingress and ACM certificates string null no
cert_manager Cert Manager configurations any {} no
cluster_admins Map of IAM roles to add as cluster admins. Only exact matching role names are returned 
map(object({
    role_name         = string
    kubernetes_groups = optional(list(string))
  }))
 {} no
create_addons Create the platform addons. if set to false, no addons will be created bool true no
downscaler Downscaler configurations any {} no
eks Map of EKS configurations any {} no
enable_acm_certificate Enable ACM certificate bool false no
enable_amp Enable AWS Managed Prometheus bool false no
enable_argocd Enable Argo CD bool false no
enable_aws_load_balancer_controller Enable AWS Load Balancer Controller bool true no
enable_cert_manager Enable Cert Manager bool false no
enable_downscaler Enable Downscaler bool false no
enable_external_dns Enable External DNS bool true no
enable_external_secrets Enable External Secrets bool true no
enable_fargate_fluentbit Enable Fargate Fluentbit bool true no
enable_fluent_operator Enable fluent operator bool false no
enable_grafana Enable Grafana bool false no
enable_ingress_nginx Enable Ingress Nginx bool false no
enable_metrics_server Enable Metrics Server bool true no
enable_okta Enable Okta integration bool false no
enable_pagerduty Enable PagerDuty integration bool false no
enable_prometheus_stack Enable Prometheus stack bool false no
enable_reloader Enable Reloader bool true no
enable_slack Enable Slack integration bool false no
external_dns External DNS configurations any {} no
external_secrets External Secrets configurations any {} no
fargate_fluentbit Fargate Fluentbit configurations any {} no
fluent_cloudwatch_retention_in_days Number of days to keep logs in cloudwatch string "7" no
fluent_log_annotation Pod Annotation required to enable fluent bit logging. Setting name to empty string will disable annotation requirement. 
object({
    name  = optional(string, "fluentbit.io/include")
    value = optional(string, "true")
  })
 {} no
fluent_operator Fluent configurations any {} no
grafana Grafana configurations, used to override default configurations any {} no
ingress_nginx Ingress Nginx configurations any {} no
karpenter [Deprecated] Karpenter configurations any {} no
karpenter_helm_set List of Karpenter Helm set values 
list(object({
    name  = string
    value = string
  }))
 [] no
karpenter_helm_values List of Karpenter Helm values list(string) [] no
karpenter_resources_helm_set List of Karpenter Resources Helm set values 
list(object({
    name  = string
    value = string
  }))
 [] no
karpenter_resources_helm_values List of Karpenter Resources Helm values list(string) [] no
metadata Metadata for the platform 
object({
    environment = optional(string, "")
    team        = optional(string, "")
  })
 {} no
metrics_server Metrics Server configurations any {} no
name The name of the platform, a timestamp will be appended to this name to make the stack_name. If not provided, the name of the directory will be used. string "" no
okta Okta configurations 
object({
    base_url                    = optional(string, "")
    secrets_manager_secret_name = optional(string, "")
    kubernetes_secret_name      = optional(string, "okta")
  })
 {} no
pagerduty PagerDuty configurations 
object({
    secrets_manager_secret_name = optional(string, "")
    kubernetes_secret_name      = optional(string, "pagerduty")
  })
 {} no
prometheus_stack Prometheus stack configurations any {} no
reloader Reloader configurations any {} no
slack Slack configurations 
object({
    secrets_manager_secret_name = optional(string, "")
    kubernetes_secret_name      = optional(string, "slack")
  })
 {} no
tags Default tags to apply to all resources map(string) {} no
vpc Map of VPC configurations any {} no

Outputs

Name Description
argocd Map of attributes for the ArgoCD module
eks Map of attributes for the EKS cluster
karpenter Map of attributes for the Karpenter module
network Map of attributes for the VPC module

Authors

Module is maintained by Alfredo Gottardo, David Beauvererd, Davide Cammarata, Francisco Ferreira, Roland Bapst and Samuel Wibrow

License

Apache 2 Licensed. See LICENSE for full details.