Terraform: Infrastructure as Code

Concept

Terraform is a tool that lets you write down your whole cloud as text files. That includes the network, servers, database and DNS (the system that turns names into addresses). Then Terraform makes the real cloud match the files. This is called Infrastructure as Code (IaC). You stop doing steps by hand. You start declaring the result you want. (Infrastructure means the basic parts that your app runs on: networks, servers and databases.)

This style is called declarative. You do not write “create a subnet, then attach a route”. That would be imperative, a list of steps. Instead you write “there is a subnet with this range and this route table”. Terraform works out the order, what already exists and what has changed.

Why clicking and shell history stop working

Count what you created by hand in Lessons 6 to 13. (A VPC is your private network inside AWS. A subnet is a smaller part of it. A Security Group is a firewall for a server. An ALB is a load balancer that shares web requests between servers. RDS is the AWS managed database service.) It was dozens of objects across ten services, with IDs copied from one command to the next. Now imagine these situations:

• “Recreate production as a staging environment.” (Staging is a practice copy of production.) Without IaC this means days of careful clicking and guessing. With IaC you run terraform apply -var environment=staging.
• “Who changed the security group last Tuesday?” Console changes leave no trail that you can review. Terraform changes are Git commits and pull requests. Each one has an author and a reviewer.
• Disaster recovery. Imagine a region outage, a hacked account or an accidental delete. You can rebuild everything from the repository in a short time. You do not need to rely on someone's memory.
• Drift. Drift means the real cloud no longer matches your code. For example, someone quickly opens port 22 to the whole world in the console and forgets. Terraform finds the difference on the next plan.
• Cost control. Delete the expensive ALB layer for the weekend with one command. Bring it back on Monday exactly as it was.
• Onboarding and audit. New people can learn from the repository, and auditors can check it. The repository is the documentation. It is always correct because it is what builds the system.

The nine words of Terraform

Idempotent means that running the same action again and again gives the same result. Terraform is idempotent: a second apply with the same code changes nothing. Terraform files are written in HCL (HashiCorp Configuration Language), which is a simple language for describing settings. It reads like a list of settings, not like a programming language:

resource "aws_s3_bucket" "uploads" {     # resource "<TYPE>" "<YOUR NAME FOR IT>"
  bucket = "myapp-uploads-abcd1234"       # setting = value
  tags   = { Name = "uploads" }
}

# Refer to it from anywhere with  <TYPE>.<NAME>.<ATTRIBUTE>
#   aws_s3_bucket.uploads.arn      (an ARN is the unique AWS name of a resource)
# Terraform sees that reference and knows: create the bucket BEFORE whatever uses its ARN.

Install and connect to AWS

Install the Terraform CLI (command-line program). Use Homebrew on macOS, winget on Windows or HashiCorp's apt repository on Ubuntu. Use version 1.11 or newer. Version 1.10 has the S3 locking used below only as an experimental feature. Terraform uses the same credentials as the AWS CLI from Lesson 5. Check that they still show your IAM user, never the root user. (An IAM user is a person's login inside your AWS account. The root user is the all-powerful first login, which you should not use for daily work.)

Terraform acts as you, so it can do everything you can do. Later, in CI, it gets its own role. A role is a set of AWS permissions that a person or program can take on. This role is broader than the app-deploy role of Lesson 14, and we keep them separate on purpose.

The daily loop: init, plan, apply

First, a safe home for state

A state file is a file in which Terraform remembers what it built. By default this file is on your laptop. If you lose the laptop, or if two teammates work at once, you have a disaster. So we keep state in an S3 bucket (an AWS storage folder) with three protections. Versioning keeps old copies, so you can undo. Encryption protects the contents. Locking stops two applies from running at once. There is a chicken-and-egg problem: the bucket must exist before Terraform can use it. So we create just this one bucket by hand.

In the S3 console, create a bucket named myapp-tfstate-<account-id> in the ap-south-1 region. Turn versioning on (every old state is kept). Keep default encryption on. Leave Block Public Access on.

In the past, locking needed a separate DynamoDB table (an AWS database service). Now the S3 backend can lock with a small lock file inside the bucket (use_lockfile = true). It was added as experimental in Terraform 1.10 and is stable from 1.11. This removes one whole service from the setup.

Build the project, file by file

Terraform reads every .tf file in a folder as one configuration. So the split into files is only to help humans. Here is a layout that keeps working as the project grows:

infra/
├── versions.tf        # Terraform and provider versions, and the state backend
├── variables.tf       # input values
├── network.tf         # VPC, subnets, gateway, routes            (Lesson 6)
├── security.tf        # security groups                          (Lesson 6)
├── database.tf        # RDS                                      (Lesson 8)
├── compute.tf         # ALB, launch template, Auto Scaling       (Lesson 12)
├── dns.tf             # Route 53 records                         (Lesson 13)
├── outputs.tf         # values worth printing
├── user-data.sh.tftpl # the boot script, with blanks to fill in  (Lesson 12)
├── prod.tfvars        # values for production (no secrets!)
└── .gitignore         # see below

versions.tf — pin every version

“Pin” means to fix a version so it cannot change by surprise.

terraform {
  required_version = ">= 1.11"

  required_providers {
    aws    = { source = "hashicorp/aws",    version = "~> 6.0" }
    random = { source = "hashicorp/random", version = "~> 3.6" }
  }

  backend "s3" {
    bucket       = "myapp-tfstate-123456789012"   # the bucket you just created
    key          = "prod/terraform.tfstate"       # one state file for each environment
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true                           # locking inside S3, no DynamoDB needed
  }
}

provider "aws" {
  region = var.region

  # Every resource that supports tags gets these automatically. Tags are labels, and they matter for cost reports (Lesson 19)
  default_tags {
    tags = {
      Project     = var.project
      Environment = var.environment
      ManagedBy   = "terraform"
    }
  }
}

~> 6.0 means “any 6.x version, but never 7”. You get fixes but not breaking changes. Terraform writes the exact versions it chose into .terraform.lock.hcl. You commit that file to Git (Git is the version-control tool that records every change to your files), so everyone, including CI, uses identical providers.

variables.tf

This file lists the input values, with defaults: project (“myapp”), environment, region, domain, the instance sizes, and the Auto Scaling min_size and max_size. Every file below uses them as var.project and so on, so you do not repeat values.

network.tf — Lesson 6, as code

You need one aws_vpc and four aws_subnet blocks: two public and two private. Make each pair with count = 2, one per Availability Zone (a separate data centre area). Add an aws_internet_gateway. Add the public route table with its 0.0.0.0/0 → gateway row (0.0.0.0/0 means “every address”, so all internet traffic goes through the gateway), plus two associations (links from the table to the public subnets). These are the same pieces you clicked through in Lesson 6. Each becomes one block, about 50 lines in all.

security.tf — Security Groups that reference each other

This is the best file to read closely, because it shows how blocks refer to each other. Here are the interesting parts:

resource "aws_security_group" "web" {
  name   = "${var.project}-web-sg"
  vpc_id = aws_vpc.main.id                 # a reference, so Terraform creates the VPC first
}

# ALB -> app servers: the traffic source is a SECURITY GROUP, not an IP address
resource "aws_vpc_security_group_ingress_rule" "web_from_alb" {
  security_group_id            = aws_security_group.web.id
  referenced_security_group_id = aws_security_group.alb.id
  from_port                    = 80
  to_port                      = 80
  ip_protocol                  = "tcp"
}

# IMPORTANT: unlike the console, Terraform removes the default "allow all outbound" rule.
# Without an egress (outbound) rule the servers cannot reach ECR, SSM, S3 or the internet.
resource "aws_vpc_security_group_egress_rule" "web_out" {
  security_group_id = aws_security_group.web.id
  cidr_ipv4         = "0.0.0.0/0"
  ip_protocol       = "-1"
}

The alb and db groups and their rules follow the same pattern. The internet reaches the ALB on ports 80 and 443. The ALB reaches the web servers on port 80. The web servers reach the database on port 5432.

database.tf — Lesson 8, as code

These are the same settings as in Lesson 8's table, written as code. They are a subnet group made of the private subnets, publicly_accessible = false, the db security group, encryption, 7-day backups and deletion_protection = true. The admin password comes from a random_password resource. One extra setting makes even terraform destroy refuse to delete the database:

lifecycle {
  prevent_destroy = true
}

compute.tf — Lesson 12, as code

This is the largest file, because Lesson 12 had the most pieces. It holds these parts:

• The ALB and its target group, with the /api/health check.
• The HTTPS listener (the part of the ALB that waits for requests on one port), which uses the ACM certificate (ACM, AWS Certificate Manager, issues free TLS certificates for HTTPS) that a data block looks up. Also the listener that redirects HTTP to HTTPS.
• The launch template. It sets IMDSv2 (the safer way for a server to read its own details), a hop limit of 2, an encrypted disk, and the user-data script (the script that runs on first boot), filled in with templatefile().
• The Auto Scaling group with ELB health checks and a rolling instance_refresh. A rolling refresh replaces servers a few at a time.

Nothing here is new. These are the same fields you set in the console. Now a pull request can review them.

dns.tf and outputs.tf

dns.tf holds the alias records from Lesson 13. They point at aws_lb.app.dns_name, so if the ALB is replaced, the record follows it automatically. outputs.tf prints useful values after an apply, such as the ALB name and the database endpoint. prod.tfvars holds the values for this environment. Add .terraform/, *.tfstate and any .tfvars file that holds secrets to .gitignore.

Now run the loop, from the infra/ folder:

Then run the plan once more. If you see “No changes. Your infrastructure matches the configuration.”, your setup is healthy.

How to read a plan

  # aws_security_group.web will be updated in-place
  ~ resource "aws_security_group" "web" {
      ~ description = "App servers" -> "App servers behind the ALB"   # (~ changed in place)
    }

  # aws_subnet.private[1] must be replaced
-/+ resource "aws_subnet" "private" {                                  # (-/+ destroy, then create again!)
      ~ availability_zone = "ap-south-1b" -> "ap-south-1c" # forces replacement
    }

  # aws_route53_record.old will be destroyed
  - resource "aws_route53_record" "old" { ... }                        # (- will be deleted)

  # aws_s3_bucket.assets will be created
  + resource "aws_s3_bucket" "assets" { ... }                          # (+ will be created)

Plan: 2 to add, 1 to change, 2 to destroy.

State: the part everyone gets wrong

The state file is a JSON file (a text format for structured data). It links “this block in my code” to “that real resource ID in AWS”. Terraform needs it to know that aws_vpc.main is vpc-0abc…. If you lose it, Terraform forgets that it owns anything. The next apply then tries to create everything again and clashes with what already exists.

• Remote and locked: the S3 backend gives the whole team one shared copy. Locking makes a second apply that starts at the same time wait, instead of corrupting the state.
• Never edit it by hand. Use the terraform state commands instead. list and show look at the state. mv is for when you rename a resource. rm makes Terraform forget something without destroying it.
• It contains secrets. The generated database password is saved in it as plain text. Encrypt the bucket. Allow only the Terraform role to read it. Never put the state, or a .tfvars file that holds secrets, in Git. It is even better to keep secrets out of the state. For RDS you can set manage_master_user_password = true. Then AWS creates the password in Secrets Manager (an AWS service for storing secrets), and the password does not pass through Terraform.
• One state for each environment. Use a separate key (or folder) for dev, staging and prod. Then a mistake in dev cannot touch prod, and each plan stays fast.

You will also use a few more commands. One lists everything in the state. One shows the saved settings of one resource. A refresh-only plan (plan -refresh-only) reports drift without proposing changes. The -replace option forces one resource to be created again.

Six common pitfalls on day one

Three lifecycle settings to know

A lifecycle block changes how Terraform treats one resource. Here are the three settings you will use most.

lifecycle {
  prevent_destroy       = true          # plan, apply and destroy refuse to delete this. Use it on databases and state buckets
  ignore_changes        = [desired_capacity]   # something else changes this on purpose, so do not fight it
  create_before_destroy = true          # build the replacement first, then remove the old one (no downtime)
}

Adopting the things you already built by hand

You already have a state bucket, an IAM role from Lesson 5 and maybe a domain zone. Terraform can adopt resources that already exist, instead of creating them again. Write the resource block. Then add an import block that says which real thing it is:

import {
  to = aws_s3_bucket.uploads
  id = "myapp-uploads-abcd1234"          # the bucket's real name
}

resource "aws_s3_bucket" "uploads" {
  bucket = "myapp-uploads-abcd1234"
}

Terraform can even write the resource block for you from the real settings. Use -generate-config-out for this. Review the generated file and apply it. After that, the plan must say “no changes”. If it shows changes, your code differs from reality. Decide which one is right.

Move in small slices, not all at once. Import one area, for example the network. Get a clean “no changes” plan. Commit. Then do the next area. Importing everything at once is where mistakes happen.

Modules and environments

Sometimes you need the same setup twice, for example a staging copy and a prod copy. A module packages the setup so you can reuse it. A module is a folder of .tf files. Variables go in and outputs come out, so using a module is like calling a function. Staging becomes a few lines that call the same module with smaller sizes:

infra/
├── modules/
│   └── web-stack/          # network + ALB + ASG + RDS: everything above, with inputs for the differences
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
└── envs/
    ├── staging/
    │   ├── main.tf         # calls the module with small sizes
    │   └── backend.tf      # its own state key: staging/terraform.tfstate
    └── prod/
        ├── main.tf         # calls the same module with real sizes and Multi-AZ
        └── backend.tf      # prod/terraform.tfstate

Terraform in a pipeline

In a mature team, nobody runs apply from a laptop. A pull request runs plan and posts the result. Merging the pull request runs apply after approval. Every infrastructure change is reviewed like code, with a plan attached.

• On a pull request that changes infra/: run init, fmt -check and validate. Then run plan and post the plan as a comment on the pull request.
• On merge to main: run the same plan, then apply. The production environment in GitHub must approve it first.
• The job signs in to AWS with OIDC (Lesson 14), as a role called github-terraform.

The github-terraform role needs wide permissions, because it creates VPCs and IAM roles. That is why it is a separate role from the narrow github-deploy role of Lesson 14. It is trusted only for the infra path and the production environment. You can also add tflint (a checker for Terraform mistakes) and a policy scanner such as checkov or trivy (the tool that replaced tfsec). They catch open Security Groups and unencrypted disks before those exist.

What this costs

Troubleshooting table

Interview corner

Practice task before Lesson 16

1. Create the state bucket, then write the infra/ files described above. Run init and plan. Read the whole plan from top to bottom before you apply.
2. Apply, then run plan again and confirm “No changes”.
3. Cause drift on purpose. Add an extra inbound rule to a Security Group in the console. Run terraform plan and read how it reports the change. Then run apply to undo it.
4. Change max_size to 8 and check that the plan is an in-place update. Change the database identifier and check that it becomes -/+. Do not apply that one!
5. Import your existing uploads bucket with an import block. Get a clean plan.
6. Run terraform destroy on a throwaway copy. Remove the database's prevent_destroy first. Time it. Then run apply again and time the rebuild. Write down how long it would take to rebuild production from nothing.

Conclusion

You built this architecture by hand over thirteen lessons. Now it is a few hundred lines of text that people can review. Anyone can rebuild it, compare it and roll it back.

• Declare the result and let Terraform work out the order and the differences. The references between resources set the dependencies.
• init, then plan, then apply. Read the plan and look for -/+ and destroys.
• State is precious. Keep it remote, versioned, encrypted and locked. Never edit it by hand. Never put it in Git. Use one state for each environment.
• Protect things that hold data with prevent_destroy. Use ignore_changes where something else changes a value on purpose. Prefer for_each over count.
• Infrastructure changes go through pull requests with the plan output. They use a separate role, with tight control, that is different from the app pipeline.

Next we make the app faster with a cache. Lesson 16 covers Redis for sessions, rate limits and data that is read often.

End of Lesson 15. Next: Lesson 16 — Redis: Caching and Sessions.