Introduction
One of the most common sources of confusion in modern infrastructure engineering is not understanding any single tool - it is understanding how four tools that are frequently mentioned together actually relate to each other. Terraform, Docker, Ansible, and Kubernetes each have distinct, well-defined responsibilities. They are not alternatives to one another; they are not even fully separable. They compose into a layered system where each tool operates at a different level of abstraction, and the value of the stack comes from understanding those layers clearly enough to assign work to the right tool at the right level.
The confusion is understandable. All four tools touch "infrastructure." All four can be used to deploy software. All four have some capability overlap with the others - Terraform has a Kubernetes provider, Ansible can manage Docker containers, Kubernetes can provision storage. This overlap creates a genuine architectural question: when two tools can both do a thing, which one should? Answering that question well requires not just knowing what each tool does but why it was designed the way it was, and what design pressures it optimizes for. A Kubernetes cluster manifest applied via Terraform and the same manifest applied via kubectl produce the same running workload, but they carry very different implications for state management, team workflow, and operational posture.
This article establishes a clear mental model of what each tool is responsible for, maps the integration points between them, walks through a complete infrastructure pipeline that uses all four, and addresses the places where the boundaries genuinely blur. The intended audience is engineers who have used at least one or two of these tools in isolation and are now building systems where they need to work together. This is not a getting-started guide for any single tool - it is an architectural thinking piece for engineers who are past that stage.
The Problem: Infrastructure Has Layers That Require Different Mental Models
The root of the confusion between these four tools is that "infrastructure" describes at least four distinct engineering concerns, each with its own state model, change cadence, and failure mode. Cloud resources - VPCs, subnets, managed database instances, IAM roles, Kubernetes clusters - are provisioned through cloud provider APIs and have a lifecycle measured in months or years. Operating system configuration - installed packages, security hardening, service definitions, user management - changes less frequently than applications but more frequently than cloud topology, and its state lives on individual machines. Container images - the immutable, versioned artifacts that package application code and dependencies - are built per commit and have a lifecycle tied to the software development cycle. Running workloads - the scheduled, replicated, load-balanced execution of container images on a cluster - change with every deployment, potentially many times per day.
These four concerns have fundamentally different state semantics. Cloud resources have state that lives in a cloud provider's control plane - modifying that state requires an API call, and the consequences of untracked changes (drift) are severe and hard to detect. OS configuration is stateful but addressable - Ansible's idempotency model works precisely because you can inspect and correct OS state incrementally. Container images are immutable once built - there is no state to manage, only artifact versions. Running workloads in Kubernetes have state managed by the Kubernetes control plane itself - the desired state is declared, and the controller reconciliation loop continuously works to achieve it. Reaching for a single tool to manage all four concerns is not just awkward - it fundamentally mismatches the state model of each concern.
The Four Roles, Clearly Defined
Terraform: Cloud Resource Provisioning
Terraform's job is to provision and manage the infrastructure resources your systems run on. It speaks to cloud provider APIs - AWS, GCP, Azure, and dozens of others - and models their resources as a directed acyclic graph of dependencies. Its state model is explicit: Terraform writes the known state of all managed resources to a state file, compares the actual resource state against the desired state declared in HCL, and computes a plan of creates, updates, and destroys needed to reconcile them. This plan-then-apply workflow is what makes Terraform safe to use for high-consequence operations: deleting a VPC or modifying a production database parameter group requires an explicit reviewed plan before execution.
Terraform excels at the things that are expensive to change: network topology, IAM policy structure, database instances, load balancers, managed Kubernetes cluster configuration. These are resources where idempotency through repeated API calls is not sufficient - you need an explicit change model that tells you what is going to happen before it happens, and that records what did happen in a format that future runs can reason about. Terraform's terraform plan output is the infrastructure equivalent of a database migration diff: a human-reviewable, machine-executable description of a state transition.
Docker: Application Packaging
Docker's role is packaging. It provides a standard format - the OCI image - for bundling an application and all its runtime dependencies (language runtimes, libraries, configuration) into a portable, reproducible, immutable artifact. The Dockerfile is the build recipe; the resulting image is the artifact. Images are versioned, tagged, and pushed to a container registry (Docker Hub, Amazon ECR, Google Artifact Registry, a self-hosted Harbor instance). Once an image is in the registry, it is the unit of deployment for every downstream system - Kubernetes, ECS, Nomad, and others consume images, not source code.
Docker's decisive contribution to the infrastructure stack is the elimination of environment parity problems. "Works on my machine" is a symptom of environmental inconsistency between development, staging, and production. A Docker image carries its environment with it. The same image SHA that passed CI runs in staging and is promoted to production. This immutability is a guarantee that the previous generation of application deployment - copying source code to servers, running npm install or pip install in place - could not provide. Docker also provides Docker Compose for local development multi-service environments, which is genuinely useful for running a service with its dependencies (a database, a cache, a queue) without a Kubernetes cluster.
Ansible: Configuration Management and Operational Automation
Ansible's role, as established in depth in the preceding article in this series, is configuration management and orchestration. In the four-tool stack, it operates at two distinct points. Before the Kubernetes layer exists - when machines are raw compute instances provisioned by Terraform - Ansible configures the operating system and installs the software required for those machines to serve their purpose. This might mean installing and hardening Docker on a standalone VM, bootstrapping a self-managed Kubernetes cluster via kubeadm (or delegating to a tool like Kubespray, which is itself an Ansible project), or configuring a database server with the right PostgreSQL parameters and pg_hba rules. After the Kubernetes layer exists, Ansible can also manage Kubernetes resources via the kubernetes.core collection, though this role is increasingly shared with or replaced by GitOps tooling.
Ansible's unique strength in this stack is its ability to glue other tools together. It can run terraform apply, capture outputs, and use them as variables in subsequent tasks. It can build Docker images, push them to registries, and trigger rolling deployments. It can generate kubeconfig files from Terraform outputs and apply Kubernetes manifests. This orchestration capability - wrapping other tools inside a coherent, sequenced, idempotent pipeline - is where Ansible's value in a multi-tool stack is highest, even if the individual steps are delegated to other tools.
Kubernetes: Container Orchestration
Kubernetes' role is running containers at scale with operational guarantees that no single machine can provide. It schedules containers across a cluster of nodes, ensures the declared number of replicas are running, reroutes traffic when a container fails, rolls out new image versions without downtime, manages configuration and secrets, enforces resource limits and requests, provides service discovery and internal DNS, and manages persistent storage attachment. Kubernetes is not a deployment tool in the sense of "put this software on a machine" - it is a distributed operating system for containerized workloads, and its reconciliation loop is the mechanism by which desired state and actual state are continuously aligned.
Kubernetes consumes Docker images as its deployment artifact. When you update a Deployment resource with a new image tag, Kubernetes pulls the updated image from the registry, starts new pods with the new image, waits for them to pass health checks, and then terminates the old pods - all without any external orchestration script. This self-healing, self-operating character is what distinguishes running containerized applications on Kubernetes from running them on raw VMs. The operational work shifts from "run this command to deploy" to "declare this desired state and let the control plane figure out how to achieve it." This is the right abstraction for high-availability, frequently-deployed applications, but it carries real complexity in the areas of networking, storage, and security configuration that simpler deployments do not.
The Integration Points: Where Tools Hand Off to Each Other
Terraform -> Ansible: Outputs as Inventory
The most consequential integration between Terraform and Ansible is the handoff of provisioned infrastructure identity to Ansible's inventory. Terraform knows the IP addresses, DNS names, and resource IDs of everything it creates. Ansible needs to know which hosts to manage. The connection is made through two mechanisms. The simpler one is Terraform's local_file or templatefile functions generating a static Ansible inventory file as part of the apply output. The more scalable one is Ansible's dynamic inventory support: the cloud.terraform Ansible collection provides a terraform_state dynamic inventory plugin that reads a Terraform state file directly and generates inventory from it, with resource metadata (region, tags, instance type) available as host variables.
This integration also flows through Terraform outputs. A Terraform run that creates an RDS database instance, an ElastiCache cluster, and an EKS cluster will expose their endpoints, ARNs, and connection details as outputs. Ansible playbooks consume these outputs either by reading the state file, by calling terraform output -json and parsing the result, or by using the cloud.terraform collection's terraform_output lookup plugin. The result is that Ansible's configuration tasks - deploying application .env files, generating database connection strings, writing kubeconfig files - are driven by actual infrastructure values rather than hardcoded configuration, making the entire pipeline reproducible from a clean state.
# terraform/outputs.tf
# Outputs that feed downstream Ansible and Kubernetes tooling
output "eks_cluster_endpoint" {
description = "EKS cluster API server endpoint"
value = module.eks.cluster_endpoint
sensitive = false
}
output "eks_cluster_name" {
description = "EKS cluster name for kubeconfig generation"
value = module.eks.cluster_name
}
output "eks_cluster_ca_certificate" {
description = "Base64-encoded cluster CA certificate"
value = module.eks.cluster_certificate_authority_data
sensitive = true
}
output "rds_endpoint" {
description = "RDS PostgreSQL endpoint"
value = module.rds.db_instance_endpoint
sensitive = true
}
output "ecr_repository_url" {
description = "ECR repository URL for Docker image push/pull"
value = aws_ecr_repository.app.repository_url
}
output "app_server_ips" {
description = "Public IPs of bastion/management hosts"
value = [for instance in aws_instance.management : instance.public_ip]
}
# ansible/inventory/terraform_state.yaml
# Dynamic inventory plugin reading Terraform state directly
# Requires: ansible-galaxy collection install cloud.terraform
plugin: cloud.terraform.terraform_state
project_path: "../terraform"
backend_type: s3 # or 'local', 'gcs', 'azurerm', etc.
backend_config:
bucket: "my-org-terraform-state"
key: "prod/terraform.tfstate"
region: "us-east-1"
# Map Terraform resource attributes to Ansible host variables
hostnames:
- "value.public_ip"
- "value.private_dns"
compose:
ansible_host: public_ip
ansible_user: "'ec2-user'"
environment: "value.tags.Environment"
# ansible/playbooks/post_provision.yaml
# Runs immediately after 'terraform apply' to configure provisioned infrastructure
- name: Bootstrap management hosts and write kubeconfig
hosts: all
gather_facts: false
become: true
tasks:
- name: Read Terraform outputs
ansible.builtin.command:
cmd: terraform output -json
chdir: "{{ playbook_dir }}/../../terraform"
delegate_to: localhost
run_once: true
register: tf_outputs
changed_when: false
- name: Parse Terraform output JSON
ansible.builtin.set_fact:
cluster_name: "{{ (tf_outputs.stdout | from_json).eks_cluster_name.value }}"
ecr_url: "{{ (tf_outputs.stdout | from_json).ecr_repository_url.value }}"
delegate_to: localhost
run_once: true
- name: Generate kubeconfig for EKS cluster
ansible.builtin.command:
cmd: >
aws eks update-kubeconfig
--region us-east-1
--name {{ cluster_name }}
--kubeconfig {{ playbook_dir }}/../kubeconfig/{{ cluster_name }}.yaml
delegate_to: localhost
run_once: true
changed_when: true
- name: Install and configure application prerequisites
ansible.builtin.include_role:
name: app_prerequisites
vars:
ecr_repository: "{{ ecr_url }}"
Docker -> Kubernetes: The Image as the Deployment Artifact
The integration between Docker and Kubernetes is the cleanest interface in the stack because it is deliberately designed as a standard: the OCI image format. Kubernetes does not know or care how an image was built. It pulls the image from the registry using the URL and tag specified in the manifest's spec.containers[].image field. The Kubernetes node's container runtime - containerd in modern clusters, CRI-O in RHEL-based distributions - handles the actual image pull and container execution.
The operational discipline at this interface is image tagging. Tagging images with latest is a known anti-pattern for production deployments because it is mutable - latest today may point to a different SHA than latest tomorrow, and Kubernetes's IfNotPresent pull policy means a node that already has the latest image cached will not pull the updated version. Production deployments should use immutable tags: the full git commit SHA, a semantic version, or a combination of both. The CI/CD pipeline builds the image, tags it with the commit SHA, pushes it to the registry, and updates the Kubernetes Deployment manifest with the new image reference. This produces a fully auditable deployment history where every running pod can be traced to an exact commit.
# Dockerfile - multi-stage build for a Node.js API service
# Stage 1: Build
FROM node:20-alpine AS builder
WORKDIR /app
# Copy dependency files first for better layer caching
COPY package.json package-lock.json ./
RUN npm ci --only=production
COPY tsconfig.json ./
COPY src/ ./src/
RUN npm run build
# Stage 2: Production image - minimal surface area
FROM node:20-alpine AS production
# Run as non-root user
RUN addgroup -g 1001 -S nodejs && \
adduser -S nodeapp -u 1001 -G nodejs
WORKDIR /app
# Copy only what the runtime needs
COPY --from=builder --chown=nodeapp:nodejs /app/node_modules ./node_modules
COPY --from=builder --chown=nodeapp:nodejs /app/dist ./dist
COPY --chown=nodeapp:nodejs package.json ./
USER nodeapp
EXPOSE 3000
# Use exec form to receive signals correctly
ENTRYPOINT ["node", "dist/server.js"]
# Health check baked into the image
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD node -e "require('http').get('http://localhost:3000/health', r => process.exit(r.statusCode === 200 ? 0 : 1))"
# kubernetes/deployments/api.yaml
# Kubernetes Deployment - image tag injected by CI pipeline
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: production
labels:
app: api
version: "{{ IMAGE_TAG }}" # Replaced by CI/CD via sed, kustomize, or Helm values
annotations:
kubernetes.io/change-cause: "Deploy {{ IMAGE_TAG }} from commit {{ GIT_COMMIT }}"
spec:
replicas: 3
selector:
matchLabels:
app: api
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0 # Zero-downtime: never reduce available replicas
template:
metadata:
labels:
app: api
version: "{{ IMAGE_TAG }}"
spec:
serviceAccountName: api-service-account
securityContext:
runAsNonRoot: true
runAsUser: 1001
fsGroup: 1001
containers:
- name: api
image: "123456789.dkr.ecr.us-east-1.amazonaws.com/myapp/api:{{ IMAGE_TAG }}"
imagePullPolicy: Always # Always pull to ensure the tag is current
ports:
- containerPort: 3000
name: http
resources:
requests:
cpu: "100m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
env:
- name: NODE_ENV
value: "production"
- name: PORT
value: "3000"
- name: DB_HOST
valueFrom:
secretKeyRef:
name: api-database-secret
key: host
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 10
Terraform -> Kubernetes: Direct Resource Management (with Caveats)
Terraform's Kubernetes and Helm providers allow it to manage Kubernetes resources directly - creating namespaces, applying RBAC policies, deploying Helm charts - using the same plan-apply workflow it uses for cloud resources. This is genuinely useful for a specific class of Kubernetes resource: the infrastructure-level resources that are provisioned once, change rarely, and need to exist before application workloads can run. Namespace creation, ClusterRoleBindings for AWS IAM integration, storage class definitions, and cert-manager or external-secrets-operator installation are good candidates for Terraform management because they have the same change cadence as infrastructure resources and benefit from Terraform's explicit state tracking and plan review.
Application workloads - Deployments, Services, HorizontalPodAutoscalers for application code - are poor candidates for Terraform management because their change cadence is driven by the application development cycle, not the infrastructure cycle. Updating an application Deployment via Terraform requires a terraform apply, which re-evaluates all infrastructure state, potentially triggering unnecessary refreshes and state lock contention. The idiomatic pattern is to manage infrastructure-tier Kubernetes resources with Terraform and application-tier resources with either direct kubectl apply (simple), Helm (parameterized), or a GitOps operator like ArgoCD or Flux (declarative, reconciled, auditable).
A Complete Infrastructure Pipeline: From Code to Running in Production
Provisioning the Foundation with Terraform
A realistic four-tool pipeline begins with Terraform provisioning the foundational cloud resources. In an AWS context, this means a VPC with public and private subnets across multiple availability zones, an EKS cluster in the private subnets, node groups sized for the expected workload, an RDS PostgreSQL instance in an isolated subnet group, an ElastiCache Redis cluster, an ECR repository for Docker images, IAM roles for the EKS node groups and for IRSA (IAM Roles for Service Accounts), and an Application Load Balancer. The EKS cluster itself, once created, exposes the Kubernetes API endpoint that all subsequent tooling communicates with.
# terraform/main.tf - foundational infrastructure
# (simplified for clarity; production would use separate module files)
terraform {
required_version = ">= 1.6"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.0"
}
helm = {
source = "hashicorp/helm"
version = "~> 2.0"
}
}
backend "s3" {
bucket = "my-org-terraform-state"
key = "prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-state-lock"
encrypt = true
}
}
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"
name = "prod-vpc"
cidr = "10.0.0.0/16"
azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
public_subnets = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
enable_nat_gateway = true
single_nat_gateway = false # HA: one NAT GW per AZ
enable_dns_hostnames = true
private_subnet_tags = {
"kubernetes.io/role/internal-elb" = "1"
"kubernetes.io/cluster/prod" = "shared"
}
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 20.0"
cluster_name = "prod"
cluster_version = "1.29"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
cluster_endpoint_public_access = true
cluster_endpoint_public_access_cidrs = ["203.0.113.0/24"] # Office IP
eks_managed_node_groups = {
general = {
instance_types = ["m6i.large"]
min_size = 2
max_size = 10
desired_size = 3
labels = { role = "general" }
}
}
}
# ECR repository for application images
resource "aws_ecr_repository" "app" {
name = "myapp/api"
image_tag_mutability = "IMMUTABLE" # Prevent tag overwriting
image_scanning_configuration {
scan_on_push = true
}
}
# Kubernetes provider configured from EKS outputs - used for infra-tier K8s resources
provider "kubernetes" {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
}
}
# Infrastructure-tier Kubernetes resources managed by Terraform
resource "kubernetes_namespace" "production" {
metadata {
name = "production"
labels = {
environment = "production"
managed-by = "terraform"
}
}
depends_on = [module.eks]
}
# Install AWS Load Balancer Controller via Helm - infrastructure dependency
provider "helm" {
kubernetes {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
}
}
}
resource "helm_release" "aws_load_balancer_controller" {
name = "aws-load-balancer-controller"
repository = "https://aws.github.io/eks-charts"
chart = "aws-load-balancer-controller"
namespace = "kube-system"
version = "1.7.2"
set {
name = "clusterName"
value = module.eks.cluster_name
}
set {
name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
value = aws_iam_role.aws_lbc.arn
}
depends_on = [module.eks]
}
Building and Pushing the Docker Image in CI
While Terraform manages cloud resources, the CI/CD pipeline - GitHub Actions, GitLab CI, Jenkins, or Buildkite - handles the Docker build and push on every merge to the main branch. The pipeline builds the image with the commit SHA as the immutable tag, runs any image-level tests (Trivy vulnerability scanning, container structure tests), and pushes the image to ECR. The commit SHA becomes the artifact identifier that flows from CI through to the Kubernetes deployment.
# .github/workflows/build-and-deploy.yaml
# CI/CD pipeline: build Docker image, push to ECR, deploy to Kubernetes
name: Build and Deploy
on:
push:
branches: [main]
env:
AWS_REGION: us-east-1
ECR_REGISTRY: 123456789.dkr.ecr.us-east-1.amazonaws.com
ECR_REPOSITORY: myapp/api
jobs:
build-and-push:
name: Build, Scan, and Push Docker Image
runs-on: ubuntu-latest
outputs:
image_tag: ${{ steps.meta.outputs.version }}
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC - no long-lived keys)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/github-actions-deploy
aws-region: ${{ env.AWS_REGION }}
- name: Log in to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
- name: Extract image metadata (tag = full commit SHA)
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.ECR_REGISTRY }}/${{ env.ECR_REPOSITORY }}
tags: |
type=sha,format=long,prefix=
type=semver,pattern={{version}}
- name: Build Docker image
uses: docker/build-push-action@v5
with:
context: .
push: false # Build first, scan before push
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
load: true
- name: Scan image for vulnerabilities with Trivy
uses: aquasecurity/trivy-action@master
with:
image-ref: ${{ env.ECR_REGISTRY }}/${{ env.ECR_REPOSITORY }}:${{ steps.meta.outputs.version }}
format: table
exit-code: 1 # Fail on HIGH or CRITICAL vulnerabilities
severity: HIGH,CRITICAL
- name: Push image to ECR
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
deploy-to-kubernetes:
name: Deploy to Kubernetes
runs-on: ubuntu-latest
needs: build-and-push
environment: production
steps:
- name: Checkout source
uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/github-actions-deploy
aws-region: ${{ env.AWS_REGION }}
- name: Update kubeconfig for EKS cluster
run: |
aws eks update-kubeconfig \
--region ${{ env.AWS_REGION }} \
--name prod
- name: Deploy to Kubernetes using Kustomize
run: |
cd kubernetes/overlays/production
kustomize edit set image \
api=${{ env.ECR_REGISTRY }}/${{ env.ECR_REPOSITORY }}:${{ needs.build-and-push.outputs.image_tag }}
kubectl apply -k .
- name: Wait for rollout to complete
run: |
kubectl rollout status deployment/api \
--namespace production \
--timeout=300s
Kubernetes Manifests with Kustomize Overlays
The deployment-facing Kubernetes manifests are managed with Kustomize, which provides environment-specific overlays without templating complexity. The base manifest defines the Deployment with a placeholder image tag; the production overlay patches in the real image reference, replica count, and resource limits. This approach makes the CI pipeline's kustomize edit set image command the single point of image tag injection, keeping manifest management declarative and the pipeline logic minimal.
# kubernetes/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- serviceaccount.yaml
- hpa.yaml
# kubernetes/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: production
resources:
- ../../base
patches:
- path: deployment-patch.yaml
# kubernetes/overlays/production/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
template:
spec:
containers:
- name: api
resources:
requests:
cpu: "200m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "1Gi"
Where the Boundaries Blur: Overlapping Capabilities
The State Management Divergence
The most significant overlap - and the one that causes the most architectural confusion - is that Terraform, Ansible, and Kubernetes can all apply Kubernetes manifests. Terraform has a kubernetes_manifest resource and a Helm provider. Ansible has the kubernetes.core.k8s module. And kubectl apply or a GitOps operator is the native Kubernetes approach. All three produce the same running workload. The question is which state model is appropriate for which resources.
The governing principle is: use the state model that matches the change cadence and ownership of the resource. Infrastructure resources that change on infrastructure timescales and are owned by the platform team belong in Terraform. Configuration resources that are managed as part of an application deployment and change with the application code belong in the application's Kubernetes manifests, applied via CI or GitOps. Resources that need to be bootstrapped once and then maintained idempotently across a fleet - particularly on nodes or in hybrid cloud environments - belong in Ansible. Violating this alignment by, for example, managing application Deployments in Terraform, creates a mismatch where the fastest-changing layer is constrained to the slowest-changing tool's workflow.
Docker Compose and Kubernetes: Local vs. Production Parity
Docker Compose is a common source of confusion in this stack because it occupies a role - multi-container application orchestration - that sounds like Kubernetes. The distinction is environment scope. Docker Compose is a development and testing tool. It defines a multi-service application (app server, database, cache, message queue) in a single YAML file and runs it on a single machine, which is exactly the right abstraction for a developer's laptop or a CI runner executing integration tests. It does not provide scheduling, horizontal scaling, health-based traffic routing, or distributed fault tolerance.
The practical corollary is that docker-compose.yaml and Kubernetes manifests can and should coexist in the same repository. One is for local development parity; the other is for production. Tools like Kompose can translate Compose files to Kubernetes manifests as a starting point, but the translation is imperfect - the abstractions are different enough that production manifests should be written with Kubernetes idioms directly rather than generated from Compose. The goal is not to make them identical; it is to make the container image itself the shared artifact that both use, so that the code running locally is demonstrably the same code running in production.
Trade-offs and Pitfalls
State Drift Across Tool Boundaries
Each tool in this stack maintains its own state model, and drift occurs when reality diverges from what any given tool believes to be true. Terraform drift - cloud resources modified outside of Terraform - is the most consequential because cloud resource changes are expensive and hard to reverse. It is also the most detectable: terraform plan on an unchanged configuration against a drifted environment will show unexpected changes. Kubernetes drift - resources in the cluster that are not represented in version-controlled manifests - is common in teams that apply ad hoc fixes via kubectl during incidents without subsequently committing those fixes to the manifest repository. A GitOps operator (ArgoCD, Flux) that continuously reconciles the cluster state against a git repository is the most effective mitigation for Kubernetes drift.
Ansible drift - managed hosts that have diverged from their playbook-defined state - is the hardest to detect without proactive tooling. Ansible does not maintain a state file the way Terraform does. Running a playbook against a host that has been modified manually will correct the drift, but you will not know the drift occurred until you run the playbook. Periodic automated playbook runs - daily or weekly, depending on the environment's change rate - are the standard mitigation. Some teams use --check mode runs scheduled regularly and alert on any run that would produce changes, treating unexpected changed results as drift detection signals.
Secret Management Across the Four Tools
Secrets traverse all four tools, and the failure mode at each boundary is different. Terraform manages secrets in its state file - database passwords, API keys generated during provisioning - and the state file must be encrypted at rest (S3 SSE with KMS, Terraform Cloud's encrypted state) and access-controlled (IAM policies on the state bucket). Ansible manages secrets via Vault, as covered in the preceding article. Docker images must never contain secrets - this is the most common Docker security mistake, and it is insidious because the secret may be baked into an intermediate layer invisible in docker inspect but recoverable by pulling the intermediate layer. Kubernetes manages secrets via the Secret resource, which is base64-encoded (not encrypted) by default; enabling Envelope Encryption with KMS is mandatory for production clusters.
The idiomatic solution to cross-tool secret management is a centralized secrets backend - AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault - that all four tools reference rather than own. Terraform reads secrets from AWS Secrets Manager using the aws_secretsmanager_secret_version data source. Ansible uses the community.aws.aws_secret lookup plugin. Kubernetes consumes secrets via the External Secrets Operator, which watches ExternalSecret resources and populates Kubernetes Secret objects from the external backend. Docker receives secrets at runtime as environment variables or mounted files, never baked into images. This architecture makes the secrets backend the single source of truth and eliminates the risk of secrets drifting between tool-specific stores.
Tool Proliferation and Cognitive Overhead
Using four tools where one might suffice is a genuine trade-off. Each tool has its own configuration language, its own CLI, its own debugging model, its own ecosystem of plugins and providers, and its own update cycle. A team of two engineers maintaining a simple web application does not need all four layers of this stack - a single docker-compose.yaml and a few Ansible playbooks, or a managed platform like Render or Railway, will be more productive. The four-tool stack earns its complexity at the scale where each tool's capabilities are genuinely necessary: when you need reproducible infrastructure provisioning across environments (Terraform), when you need immutable deployable artifacts (Docker), when you need configuration management at scale (Ansible), and when you need self-healing, auto-scaling workload orchestration (Kubernetes).
The most common anti-pattern is adopting all four tools based on industry reputation rather than demonstrated need, then accruing the cognitive overhead of four distinct systems without proportionate operational benefit. Kubernetes in particular has a well-documented tendency to be adopted before an organization has the operational maturity to run it effectively. The right adoption sequence for most teams is Docker first (packaging discipline), then Terraform (reproducible infrastructure), then Ansible (configuration management at scale), then Kubernetes (when running multiple containerized services with high availability requirements). Each step should be triggered by a real operational need, not by industry trend.
Best Practices for the Four-Tool Stack
Establish Clear Layer Ownership and Tooling Boundaries
The most important organizational decision in a four-tool stack is who owns what, encoded in which repositories. A common and effective structure is the three-repository model: an infrastructure repository containing all Terraform code and Ansible playbooks (owned by the platform team, change velocity: low), an application repository containing application code, Dockerfiles, and Kubernetes manifests (owned by product teams, change velocity: high), and a GitOps configuration repository containing environment-specific Kubernetes overlays and Helm values (owned jointly, change velocity: medium). This separation prevents the platform team's infrastructure changes and the product team's application deployments from coupling into the same pipeline, reduces blast radius of changes, and allows each layer to move at its natural cadence.
Enforcing the tooling boundary through CI pipeline design is more reliable than relying on team convention. Infrastructure changes should require a terraform plan review before apply. Application deployments should flow through Docker build, image scan, and manifest update - not through Terraform. Ansible playbooks affecting production hosts should require a --check --diff review artifact attached to the pull request. These pipeline gates make the boundaries tangible and reviewable rather than implicit agreements.
Treat the Pipeline as a Dependency Graph, Not a Sequence
The relationship between the four tools in production has dependency semantics: application Deployments depend on the Kubernetes cluster existing (Terraform), which in turn depends on the AWS VPC configuration (Terraform), and the running containers depend on secrets being available in the cluster (Kubernetes External Secrets Operator, backed by Terraform-provisioned Secrets Manager). Encoding these dependencies explicitly - in Terraform depends_on directives, in CI pipeline job dependencies, in Ansible when conditions checking for prerequisite resources - makes the system more robust to partial failures and easier to reason about during an outage.
A dependency graph that is implicit in the order of shell commands in a runbook is a dependency graph that will break in non-obvious ways when someone runs a step out of order or when a step fails mid-execution. Tools like terraform graph (visualizing the Terraform dependency graph), Ansible's --list-tasks (auditing task order), and Kubernetes's kubectl wait (blocking until resources reach a desired state) are all mechanisms for making the dependency graph visible and machine-enforceable rather than human-memorized.
Key Takeaways
Five concrete practices to apply when building a four-tool infrastructure stack:
-
Define layer ownership before writing any code. Decide explicitly: Terraform owns cloud resources and cluster-level Kubernetes infrastructure, Docker owns image building and artifact management, Ansible owns post-provisioning OS configuration, and Kubernetes manifests (managed via CI or GitOps) own application workloads. Write this down. Enforce it in repository structure and CI pipeline design.
-
Use Terraform outputs as the source of truth for downstream tooling. Every IP address, endpoint, ARN, and cluster name that Terraform creates should be exposed as an output and consumed by Ansible's dynamic inventory, the CI pipeline's environment variables, and Kubernetes external secrets references. Never hardcode infrastructure-generated values in playbooks or manifests.
-
Tag Docker images with immutable identifiers - never
latestin production. Use the full git commit SHA as the image tag. Configure ECR or your registry withIMMUTABLEtag mutability. Reference image SHAs (or pinned tags that map to SHAs) in production Kubernetes manifests. This makes every deployment fully traceable to a commit. -
Install a GitOps operator for Kubernetes application workloads. ArgoCD or Flux, reconciling against a git repository, eliminates Kubernetes configuration drift more reliably than any manual process. It also provides a visual audit trail of cluster state changes that
kubectl applyhistory cannot offer. -
Run
terraform planin CI on every infrastructure pull request and attach the plan output as a required review artifact. Never runterraform applywithout a reviewed plan. Store state remotely with locking (S3 + DynamoDB for AWS) and enable state encryption. These three practices together eliminate the most consequential Terraform failure modes.
Analogies and Mental Models
Think of the four tools as the disciplines required to build and operate a modern commercial kitchen. Terraform is the architect and general contractor - it designs and builds the physical space: the gas lines, the electrical capacity, the ventilation, the walk-in refrigerator. It is expensive to change, requires careful planning, and produces a physical reality that everything else depends on. Ansible is the kitchen setup crew - it installs the equipment, configures the ovens to the right temperatures, stocks the pantry with standard ingredients, and ensures every workstation is arranged according to specification before the kitchen opens. Docker is the standardized recipe and ingredient kit - each dish comes prepared and packed identically, no matter which chef prepared it or which kitchen it is cooked in. The kit is versioned; this week's beef Wellington is different from last week's, and both versions exist in the archive. Kubernetes is the head chef and expeditor combined - it schedules which dishes get cooked on which stations, ensures the line never stops even if one station fails, scales the operation up when the restaurant is fully booked, and routes each finished dish to the right table.
The key insight from this model is that none of these roles can substitute for another. The architect cannot manage the expeditor's scheduling complexity, and the expeditor cannot design the kitchen's plumbing. They each handle a layer of the operation that the others are not equipped for, and the whole system works precisely because each layer stays within its domain.
80/20 Insight
The 20% of the four-tool stack that produces 80% of the operational value is the clear definition of where one tool ends and the next begins, enforced by repository structure and CI pipeline gates. Teams that get this boundary definition right - Terraform for provisioning, Docker for packaging, Kubernetes for orchestration, and a clean handoff at each interface - find that the four-tool stack is less complex than it appears, because each tool is doing only what it is optimized for. Teams that get it wrong spend their engineering cycles fighting the impedance mismatch between a tool's design model and the work they are asking it to do.
The specific interface that pays the highest return on investment to get right is the Terraform -> Kubernetes boundary: using Terraform for cluster infrastructure and Helm/Kustomize/GitOps for application workloads, with Terraform outputs feeding the application pipeline through environment variables rather than through shared state. Every other integration in the stack - Ansible reading Terraform outputs, Docker images flowing into Kubernetes Deployments - follows naturally once this boundary is established cleanly.
Conclusion
Terraform, Docker, Ansible, and Kubernetes are not a monolith. They are four specialized tools that operate at four different layers of the infrastructure stack, and the most important skill in working with them together is knowing which layer each one belongs to and resisting the temptation to use one tool's capabilities to do another tool's job. Terraform is for provisioning. Docker is for packaging. Ansible is for configuration. Kubernetes is for orchestration. The integration points between them - Terraform outputs feeding Ansible inventory, Docker images flowing into Kubernetes Deployments, Ansible bootstrapping the cluster that Kubernetes runs on - are well-defined interfaces, not areas of ambiguity.
The stack earns its complexity in proportion to the operational requirements it serves. High availability, reproducible environments, immutable artifacts, and self-healing workloads are real engineering requirements at real scale - and this stack provides them. For smaller systems or teams earlier in their operational journey, adopting a subset of the stack, or adopting them incrementally as each tool's value becomes demonstrable, is the more pragmatic path. The goal is not to use all four tools - it is to use the right tool at the right layer, and to build the integration between them deliberately, with clear ownership and explicit dependency management.
References
- Terraform Documentation - https://developer.hashicorp.com/terraform/docs
- Terraform AWS Provider - https://registry.terraform.io/providers/hashicorp/aws/latest/docs
- Terraform Kubernetes Provider - https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs
- Terraform Helm Provider - https://registry.terraform.io/providers/hashicorp/helm/latest/docs
- Terraform AWS Modules: VPC - https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest
- Terraform AWS Modules: EKS - https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest
- Docker Multi-Stage Builds - https://docs.docker.com/build/building/multi-stage/
- OCI Image Format Specification - https://github.com/opencontainers/image-spec
- Kubernetes Deployments Documentation - https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
- Kustomize Documentation - https://kustomize.io/
- ArgoCD Documentation - https://argo-cd.readthedocs.io/
- Flux CD Documentation - https://fluxcd.io/flux/
- External Secrets Operator - https://external-secrets.io/
- AWS Load Balancer Controller - https://kubernetes-sigs.github.io/aws-load-balancer-controller/
- Ansible cloud.terraform Collection - https://docs.ansible.com/ansible/latest/collections/cloud/terraform/
- Ansible kubernetes.core Collection - https://docs.ansible.com/ansible/latest/collections/kubernetes/core/
- Kubespray - Production Kubernetes with Ansible - https://kubespray.io/
- AWS IAM Roles for Service Accounts (IRSA) - https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
- AWS ECR Image Tag Immutability - https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-tag-mutability.html
- Trivy Container Vulnerability Scanner - https://trivy.dev/
- GitHub Actions: aws-actions/configure-aws-credentials (OIDC) - https://github.com/aws-actions/configure-aws-credentials
- Kubernetes Envelope Encryption (KMS) - https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/