Security Hardening and Backups

Concept

Security is not a product that you switch on. It is the habit of making every part of the system hard to misuse and cheap to recover. This lesson has two halves, and each half is a question:

• Hardening — “how do I make it hard for the wrong person to do harm?” Hardening means closing weak points so the system is harder to attack.
• Backups — “when something bad happens anyway, how do I get back?”

Why this matters

You now hold your customers' data, their logins and their payments. The results of a mistake are real. Laws such as DPDP in India and GDPR in Europe require you to report a data breach. You can lose your customers' trust, and a small company often does not survive. Attackers do not pick you personally. Their programs scan the whole internet automatically. A server with a weak point is found in minutes, not months.

There is good news. Most real incidents come from a short list of ordinary mistakes. If you fix that list, you are safer than most companies.

Shared responsibility: what AWS secures, what you secure

Most cloud breaches come from the customer's own settings and stolen credentials, not from failures inside AWS. The right-hand column is your job.

The five ways small teams actually get hurt

Notice how much you have already done. You use roles instead of keys (Lesson 5). Your databases are private (Lessons 6 and 8). Your buckets are private (Lesson 11). Your deploy role has narrow permissions (Lesson 14). This lesson closes the remaining gaps. It also adds detection and recovery.

Layer 1 — identity: keys, MFA, roles

Identity means who or what is allowed to log in. It is the front door, and attackers use it most often. First check it with AWS's own tools. Then apply the rules:

• In IAM, open the Credential report. It is a CSV file (a simple table) with one row for every user. It shows password age, whether MFA is on, and the age and last use of each access key. MFA (multi-factor authentication) means a second proof of identity, such as a code from a phone app. Look for users with no MFA, keys older than 90 days, and keys that were never used.
• Check the user list itself. Does each person still work here? Does each identity still need to exist?
• IAM Access Analyzer (the external access check is free) finds buckets, roles and keys that are shared outside your account. It can also write least-privilege policies (policies that give only the access that is really used) from real usage.

• Root user: the root user is the all-powerful first account. Turn MFA on (Lesson 0), create no access keys, and never use it for daily work. Check this in the credential report. The <root_account> row must show access_key_1_active = false.
• MFA for every person, preferably with a hardware key or a passkey. A hardware key is a small USB device that proves you are you. A passkey is a login method that replaces the password. An AWS login with only a password is one stolen password away from total loss.
• People use IAM Identity Center (single sign-on, SSO). Machines use roles. SSO lets people sign in once with a central account. If any long-lived access key still exists, rotate it, give it the minimum permissions, and name an owner for it.
• Remove people on the day they leave. This is why groups matter (Lesson 5). One action removes one person, and no access is left over.
• Review permissions using real data. IAM shows when each service was last used (“last accessed”). If a policy allows 40 services but only 3 are used, make the policy smaller.

Stop secrets reaching Git at all

• Turn on GitHub secret scanning and push protection. Push protection blocks a push that contains an AWS key. It is free for public repositories. For private repositories it is a paid GitHub feature (GitHub Secret Protection).
• Also add a pre-commit scanner such as gitleaks on your own computer. It checks your code for secrets before each commit.
• AWS also scans public GitHub. When it finds your key, it restricts the key and emails you. But attack bots are faster than email. If a key ever reaches a public repository, treat it as stolen, however fast you delete the commit. Rotate it.

Layer 2 — network: shrink the attack surface

The safest port is a closed port. (A port is a numbered door on a server that a service listens on.) Check what is open to the world. The list should be very short:

In EC2, go to Security Groups. Look at every inbound rule whose source is 0.0.0.0/0. In our design the only such rule should be alb-sg on ports 80 and 443. If web-sg, db-sg or cache-sg appears, or any group has port 22, 5432 or 6379 open to the world, fix it now.

• Keep databases and caches in private subnets with no public IP. Only web-sg may reach them (Lessons 6, 8, 16).
• App servers accept traffic only from the ALB (Lesson 12). If they have public IPs, those IPs are for outbound traffic, not inbound.
• Add AWS WAF (Web Application Firewall) on the ALB or CloudFront. It blocks common web attacks, such as SQL injection (an attack that sneaks database commands into a form field), bad bots and too many requests from one IP. It costs about $5 a month plus $1 per rule and $0.60 per million requests. Start with the AWS-managed core rule set.
• Turn on VPC Flow Logs. They record the network connections in your VPC, and storing them in S3 is cheap. Later you can answer “who talked to this server?”

Layer 3 — no SSH: Session Manager

In Lesson 7 you opened port 22 to your IP address. That is fine for learning. It is a poor choice for production. Keys are files, and files get copied. Your IP address changes. And every scanner on earth probes an open port 22. AWS Systems Manager Session Manager gives you a shell on a server with no open port, no key pair and no bastion host. (A bastion host is a server that you log in to first, in order to reach other servers.) The SSM agent on the server connects out to AWS, and you connect using your IAM identity.

1. Give the instance role permission to talk to SSM

   Attach the AWS-managed policy AmazonSSMManagedInstanceCore to myapp-ec2-role. The migrations in Lesson 14 already need it. The agent is preinstalled on Ubuntu 24.04 images. It needs only this permission and outbound HTTPS. After a few minutes the instance shows as Online in Systems Manager, under Fleet Manager.

2. Install the Session Manager plugin and connect

   Install the Session Manager plugin for the AWS CLI on your laptop. Then run:

   On your laptopCopy all

   • aws ssm start-session --target i-0abc123def456Copy

     Opens a shell on the server with no port 22 and no key pair. The Connect button, then Session Manager, in the console does the same in the browser

   The same tool can also forward a port. This replaces Lesson 8's SSH tunnel to the database, and needs no SSH at all.

3. Close port 22 and drop the key pair

   Once start-session works, delete the SSH rule from web-sg. The new launch templates already have no key pair (Lesson 12).

Layer 4 — the server: patching and IMDSv2

In our design the server is disposable, and that helps security. Think of servers as cattle, not pets. You do not nurse each one. The best way to patch them is to replace, not repair. On a schedule, rebuild from the latest AMI (the saved server disk image) and the latest app image. Then servers never collect drift (small unplanned changes) or old security holes.

• Automatic OS security updates as a safety net for servers that live a long time.

Install the unattended-upgrades package in the user-data script (the script that runs when the server first boots) and enable it. Then Ubuntu applies security updates by itself every day.

• Replace servers regularly. Run an instance refresh (Lesson 12) at least once a month, even when the code has not changed. Then servers get the latest AMI and OS patches. A scheduled GitHub Actions workflow can do this for you.
• IMDSv2 required (set since Lesson 7). IMDS is the instance metadata service. A server can ask it for its own details, including temporary credentials. Version 2 needs a session token, so a bug in your app cannot easily be turned into credential theft. Check this on every instance.

In the EC2 instance list, add the IMDSv2 column. Every instance should say Required. Any instance that says Optional still accepts the old, weaker version. Fix it in place: choose Actions, then Instance settings, then Modify instance metadata options.

• Encrypted disks by default. In the EC2 settings, turn on “EBS encryption by default” for the region (EBS is the disk service for EC2). Then no disk is ever created without encryption. The command is aws ec2 enable-ebs-encryption-by-default. It applies to one region at a time.
• An instance role with least privilege. Review myapp-ec2-role. It should read its own parameters, pull its own image and write its own logs. It should do nothing else.

Layer 5 — secrets management

A secret is anything that gives access. Examples are database passwords, API keys, signing keys and tokens. Here is a ladder of practices, from worst to best:

Our setup already uses Parameter Store (Lesson 12). The upgrade worth making is for the database password. Let RDS manage it in Secrets Manager with rotation. Then it never appears in the Terraform state, in chat messages or on anyone's clipboard.

In RDS, choose Modify and tick Manage master credentials in AWS Secrets Manager. RDS creates the secret and rotates it. The app, or its boot script, reads the secret with permission from its role. The password never has to be copied anywhere.

Layer 6 — data: encryption and access

Encryption at rest means data is encrypted while stored on disk. It protects you from a stolen disk or snapshot. It does not protect you from a compromised application, because the app can read the decrypted data. That layer is access control, which decides who and what may read which rows, files and keys. Here are two points for a multi-tenant product (one app that serves many customers):

• Tenant isolation is a security control, not an extra feature. Every query filters by tenant. Consider PostgreSQL row-level security as a second layer. It is a database rule that limits which rows a user can see. It still protects you if a developer forgets a WHERE clause. S3 keys and cache keys also include the tenant (Lessons 11, 16).
• Force TLS on the bucket. Add a deny statement for requests where aws:SecureTransport is false.

Layer 7 — the application and its dependencies

Infrastructure controls cannot fix a vulnerable app. These habits give the most protection for the least effort:

• Keep dependencies up to date. A dependency is a package that your code uses. Turn on Dependabot (.github/dependabot.yml). It opens update requests for npm packages, GitHub Actions and Docker base images. Most breaches through code use flaws that were fixed months earlier.
• Scan in the pipeline. Add pnpm audit --prod and an image scan as CI steps. For the image scan use trivy image or ECR scan-on-push from Lesson 9. Make the build fail on serious findings.
• Follow the basics of the OWASP Top 10. OWASP publishes a list of the ten most common web security risks. Validate input. Use parameterised queries, where the values are sent separately from the SQL text, and never build SQL from strings. Escape output. Check authorisation on every request. Authorisation means checking that the user may access this record, not only that the user is logged in. Set secure cookies. Add security headers (Lesson 10).
• Do not trust the client. The browser can send anything. Checks on the server are the only real checks. A hidden button is not access control.
• Pin and review code from other people. Pin means to fix an exact version. This includes GitHub Actions (Lesson 14) and the curl | bash installers that you copy from the internet.

Layer 8 — detection: CloudTrail, GuardDuty, Config

You cannot respond to what you cannot see. Here are three AWS services, in order of importance:

1. CloudTrail: create a trail for all regions. Deliver it to a new S3 bucket with tight access rules. Turn on log file validation, so you can tell if anyone changes the log files.
2. GuardDuty: choose Get started, then Enable. Then send its findings to the same SNS topic as your alarms (Lesson 17).

Send GuardDuty findings to the alerts topic with an EventBridge rule (EventBridge is the AWS service that reacts to events). Use source: aws.guardduty and a severity of 7 or higher. Then a serious finding reaches a person in the same way as a 5xx alarm. Also set an alarm on root user login. Use a CloudTrail metric filter on userIdentity.type = Root. The root user should almost never sign in.

Backups: the part that saves the company

Hardening lowers the chance of a disaster. Backups lower its cost. Think of ransomware (malware that locks your data and demands money), a bad migration, a deleted table, a hacked admin account or a region outage. The answer is the same for all of them: a recent, separate, tested copy.

Put it on autopilot with AWS Backup

AWS Backup is one service where you manage backup schedules, retention times and copies for RDS, EBS, S3, DynamoDB and more. It stores backups in a vault. You can lock a vault so that even an administrator cannot delete the recovery points. This is a real defence against ransomware and against bad insiders. Here is a small but useful plan for this architecture:

• Daily backups of the RDS instance, kept for 35 days.
• Weekly backups kept for 12 months. Add a monthly backup kept for 7 years if the law or your rules require it.
• A copy of each backup in a vault in a second region (for example ap-southeast-1).
• Backup Vault Lock in compliance mode. Turn it on once you are sure about your retention times.

• Copy an important RDS snapshot to a second region (choose Actions, then Copy snapshot).
• Keep versioning on for every bucket that holds user data (Lesson 11).

Test the restore — the drill

A backup that you have never restored is a hope, not a backup. Backups fail quietly, in boring ways. The wrong database was included. The encryption key was deleted. The restore takes nine hours instead of one. The instructions are in the head of someone who has left. The only way to find out is to practise on a calm day.

1. Restore the production database to a new instance called myapp-db-drill, at the latest restorable time (Lesson 8). Production is not touched. Note how long it takes. That is your real recovery time.
2. Connect to the drill copy from an app server. Check that the data is really there. Count the rows and look at the newest created_at value.
3. Delete the drill instance so it stops billing.

Do this every quarter and after any major change. The time it took is your real RTO. It is usually longer than the one in your head. For the object store (S3), also practise “restore a deleted file”. Delete a test object, then bring it back from the previous version in the console.

When a key leaks: the first hour

Sooner or later this will happen to someone on your team. If the steps are written down, a stressful moment becomes a checklist. Follow these steps in order:

1. Contain — deactivate, do not delete yet

   Kill the credentialCopy all

   • aws iam update-access-key --user-name <USER> --access-key-id <AKIA…> --status InactiveCopy

     Deactivating stops the key at once and keeps the evidence. You can delete it later

   Then attach an explicit Deny everything policy to that identity. An explicit deny always wins (Lesson 5). Also check that the attacker did not create a second key for themselves.

2. Find out what the key did

   In CloudTrail, open Event history and filter by the access key ID (or the user name). Look for actions that you did not do. Examples are new users or keys, new instances (often used for crypto-mining), changed Security Groups, S3 downloads, snapshot copies to unknown accounts, and changes in other regions. Attackers like regions that you never look at.

3. Remove what they left behind

   Terminate instances that you do not know. Delete users, roles and keys that you do not know. Review Security Groups and bucket policies. Check for new IAM roles that trust an outside account. If you are in doubt, treat the whole account as unsafe. Rotate everything that the leaked identity could reach, including the database password and the application secrets.

4. Tell people, then write it up

   If customer data may have been accessed, the law may require you to report it by a deadline. Open a case with AWS Support. They can help, and they sometimes waive the charges from fraudulent use if you act quickly. Then hold a blameless post-mortem, a review that looks for causes and not for someone to blame. Ask how the key leaked, and which control would have stopped it.

The hardening checklist

Copy this into your repository. Every line is something that you have already met in the series:

## Identity
[ ] Root user: MFA on, no access keys, not used (credential report clean)
[ ] MFA required (by policy) for every human; humans use SSO / IAM Identity Center
[ ] No long-lived access keys on servers or in CI (roles + OIDC)
[ ] Any remaining keys are less than 90 days old, each with a named owner
[ ] Leavers removed the same day; access reviewed every quarter
[ ] GitHub secret scanning and push protection on

## Network
[ ] Only the ALB accepts traffic from 0.0.0.0/0 (ports 80 and 443)
[ ] Port 22 closed; access through SSM Session Manager
[ ] DB and cache in private subnets, no public IP, reachable only from web-sg
[ ] WAF (managed core rule set) on the ALB / CloudFront

## Servers
[ ] IMDSv2 required on every instance
[ ] EBS encryption by default; unattended security upgrades on
[ ] Servers rebuilt from a fresh AMI and image at least every month
[ ] Instance role is least-privilege (reviewed with Access Analyzer)

## Secrets and data
[ ] No secrets in Git, images or build args; all in SSM / Secrets Manager
[ ] DB password managed and rotated by Secrets Manager
[ ] RDS, S3, EBS, Redis encrypted at rest; TLS everywhere in transit
[ ] S3 Block Public Access on at the account level
[ ] Every query, cache key and S3 key is tenant-scoped

## App
[ ] Dependabot, pnpm audit and image scan in CI (fail on high severity)
[ ] Authorisation checked on every request, not only the login

## Detection
[ ] CloudTrail multi-region trail with log validation
[ ] GuardDuty on, findings routed to the alerts topic
[ ] Alarm on root login and on unusual spend

## Backups
[ ] RDS backups kept 7 to 35 days, plus a manual snapshot before risky changes
[ ] Cross-region copy of RDS snapshots; S3 versioning and replication
[ ] Backups protected from deletion (locked vault / separate account)
[ ] Restore drill done this quarter; RTO measured and written down
[ ] Key-leak runbook written and its steps rehearsed

What this costs

The most valuable items are free: MFA, no keys, no port 22, private data stores, Access Analyzer and CloudTrail. Everything that costs money here costs less per month than an hour of an engineer's time. That is far cheaper than one incident.

Interview corner

Practice task before Lesson 19

1. Run the credential report. Write down every finding, such as missing MFA, old keys and unused users. Fix each one.
2. Check your Security Groups as described above. List everything that is open to the internet. Explain why each item is needed, or close it.
3. Attach the SSM policy to the instance role and connect with start-session. Then remove the port 22 rule and delete the key pair from the launch template.
4. Turn on CloudTrail, GuardDuty and IAM Access Analyzer. Read the first findings.
5. Move the database password to RDS-managed Secrets Manager. Update the boot script so that it reads the password from there.
6. Run the restore drill. Restore to a point in time into a new instance, check the data, record the time it took and delete the copy. Then delete a test S3 object and restore it from its earlier version.
7. Write the key-leak runbook for your team in ten lines. Then ask a teammate to read it and say what is unclear.
8. Copy the checklist into your repository. Tick every box that you can honestly tick. The boxes you cannot tick are next quarter's work.

Conclusion

A secure system is not one that cannot be attacked. It is one where each layer limits the damage that gets past the layer before it. Problems are visible quickly, and recovery is a practised routine.

• Identity first. Use MFA for people and roles for machines. Have no long-lived keys. Give least privilege. Remove leavers on the same day.
• Shrink the attack surface. Only the ALB is public. Data stores are private. Close port 22 and use Session Manager.
• Treat servers as cattle. Rebuild them regularly. Use IMDSv2, encrypted disks and automatic security updates.
• Keep secrets in a secret store and rotate them. Use encryption at rest and in transit. Scope everything by tenant.
• Detect problems with CloudTrail, GuardDuty and alerts. Recover with layered, protected and tested backups and a written incident routine.

The system is secure, observable and recoverable. One question remains. What does all of it cost, and how do you keep the bill under control? That is the final lesson.

End of Lesson 18. Next: Lesson 19 — Cost Review and Final Production Architecture.