Monitoring and Logging with CloudWatch

Concept

Monitoring is how you find out what your system is doing right now. Alerting is how the system tells you when something is wrong. On AWS the built-in tool is CloudWatch. It collects numbers (called metrics) and text (called logs). It draws them on dashboards, which are screens of graphs. It also sends a message when a number crosses a line you set. These rules are called alarms.

Why this matters

• Without monitoring, your customers are your alarm system. The first sign of trouble is an angry email or a public post. You learn about an outage from the person who pays you.
• “It is slow” has no answer without data. Slow where? Since when? After which deploy? For whom? Metrics and logs answer these questions in seconds. Guessing takes hours.
• Everything you built can fail silently. Examples are a full disk, a database that reached its connection limit, an expired certificate, a health check that keeps switching between pass and fail (flapping), or an Auto Scaling group stuck at its maximum size. All of them look like “the site works for me” until it does not.
• You need proof that it is fixed. After a change, the graph either went back to normal or it did not.
• Capacity and cost planning need history. You must know how much traffic grows each month and how high the peak is.

Metrics, logs and traces

Telemetry means the data a system sends out about itself. It comes in three kinds, and each kind answers different questions. You want all three, but start in this order:

Metrics tell you that something is wrong. Logs tell you why. A typical investigation starts with a graph. For example, 5xx errors (server errors) rose at 14:02. Then you open the logs for that minute and find a database timeout on one endpoint. You use traces only for complex systems with many services. Our app is a monolith, which means one single application. For a monolith, metrics plus good logs solve most problems.

The four golden signals

Google's site-reliability book answers “what should I watch?” with four things. If you watch only these, you already see almost every problem that users can notice:

You may hear two memory aids. RED is for services: Rate, Errors, Duration. USE is for resources: Utilisation, Saturation, Errors. They describe the same ideas from two sides.

CloudWatch vocabulary

What you get for free — and what is missing

As soon as you create resources, AWS starts publishing metrics for them at no charge. A few names you will see: EC2 is the AWS service that rents virtual servers. An Auto Scaling group is a group of servers that AWS adds to and removes automatically. An ALB (Application Load Balancer) spreads web traffic over the servers. RDS is the managed database. ElastiCache is the managed Redis cache.

• EC2: CPU, network in and out, disk operations and status checks, every 5 minutes. With “detailed monitoring” you get them every minute, for about $2 per instance per month.
• ALB: request count, response codes, latency and the number of healthy and unhealthy hosts. RDS: CPU, connections, free storage, IOPS (disk operations per second) and latency. ElastiCache: memory, hit rate and evictions.

Nothing collects your application logs by default either. They sit in a Docker file on a server. Auto Scaling may delete that server at any moment, and the evidence is deleted with it.

Step 1 — a channel that reaches you

An alarm needs a place to send its message. SNS (Simple Notification Service) is the AWS broadcast channel for this. Start with email. Later you can switch to Slack or a pager without changing any alarm.

In SNS, go to Topics and create a Standard topic called myapp-alerts. Then create a subscription with the protocol Email and your address. AWS sends you a confirmation email. Until you click “Confirm subscription”, you receive nothing.

Step 2 — the alarms every production app needs

You do not need fifty alarms. You need the right dozen. Start with these. Each one is tied to a symptom that users feel, or to a resource that is about to run out:

Build the first one carefully. The others are small variations.

In CloudWatch, go to Alarms and choose Create alarm. Select the metric: ApplicationELB, then Per-AppELB Metrics, then HTTPCode_Target_5XX_Count for myapp-alb. Set the statistic to Sum and the threshold to ≥ 5. Send the notification to myapp-alerts. Under “Additional configuration” are the settings that make an alarm trustworthy:

The other rows of the table use the same screen with a different metric, statistic and threshold. Watch for two details. Latency uses the p95 statistic, not the average. The database disk threshold is in bytes. 5 GB is 5368709120 bytes.

Then prove that an alarm works. Do not trust it until it has fired once. You can force it into the alarm state without breaking anything:

Step 3 — get the logs off the server

A container writes its log lines to stdout (the standard output stream of a program), and Docker captures them. By default Docker saves them in a JSON file on that server's disk. The file is gone when Auto Scaling replaces the instance, which is exactly when you want to read it. Docker's awslogs driver sends each line to CloudWatch Logs instead, almost at once.

Give myapp-ec2-role (the IAM role of the server) permission to create the log group and log streams and to put log events in log groups under /myapp/*. The permissions are logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents. Then change one thing in the boot script: the log driver.

docker run -d --name myapp --restart unless-stopped \
  --env-file /etc/myapp.env \
  --log-driver awslogs \
  --log-opt awslogs-region=ap-south-1 \
  --log-opt awslogs-group=/myapp/web \
  --log-opt awslogs-create-group=true \
  -p 80:3000 $REGISTRY/myapp:$TAG

Each container becomes a log stream inside the /myapp/web group. So you can read one server or all servers together. Now here is the most important log setting of all.

Log groups keep data forever by default, and you pay for storage every month. On the day you create the group, set its retention (Actions, then Edit retention setting) to 14, 30 or 90 days. Then you can watch every server at once from your terminal:

Step 4 — write logs you can actually search

Compare these two ways of logging the same event:

# Hard: you can only search it as plain text (grep), and every developer words it differently
Error saving order for user 42 after 1204ms: connection timeout

# Easy: one JSON object per line, and every value is a field you can search
{"level":"error","time":"2026-01-15T09:30:12.041Z","msg":"order save failed","requestId":"Root=1-65a4-1c9d","tenantId":"acme","userId":"42","route":"/api/orders","status":500,"durationMs":1204,"err":"connection timeout"}

With the second form, “show me all errors for tenant acme on /api/orders slower than one second” is a one-line query. With the first form you would need a complex text pattern (a regex).

import pino from "pino";

export const log = pino({
  level: process.env.LOG_LEVEL ?? "info",              // use "debug" only while you are investigating a problem
  base: { service: "myapp", version: process.env.APP_VERSION },
  timestamp: pino.stdTimeFunctions.isoTime,
  // Anything at these paths is replaced with "[Redacted]". Do this from day one
  redact: ["*.password", "*.token", "*.secret", "req.headers.authorization", "req.headers.cookie"],
});

// per request:
//   const requestId = req.headers.get("x-amzn-trace-id") ?? crypto.randomUUID();  // the ALB adds this header
//   const l = log.child({ requestId, tenantId, route });
//   l.info({ status: 200, durationMs }, "request completed");
//   l.error({ err }, "order save failed");

The rules of good logging

• One correlation ID for each request, on every log line. A correlation ID is a unique code that ties together all the lines of one request. The ALB already adds X-Amzn-Trace-Id. Log it and also return it to the client. Then, when a user sends you an error ID, you can find the matching log lines in one second.
• Include the tenant. A multi-tenant product serves many customers from one app. In such a product, “is it only one customer?” is the first question in every incident.
• Use log levels honestly. error means a person should probably look. warn means something unexpected happened but the app handled it. info means normal business events. debug means very detailed messages, which stay off in production.
• Never log secrets or personal data. This includes passwords, tokens, session IDs, full card numbers and national ID numbers. Many people copy, keep and read logs. Leaking such data is a legal and security problem, not a style problem.
• Do not log every health check. The load balancer checks every server every few seconds. These lines hide the useful ones and make the bill larger.
• One line for each event, and no multi-line stack traces. Put the stack trace (the list of function calls at the error) inside a field. Otherwise the lines are split and Insights cannot read them.

Step 5 — search with Logs Insights

Logs Insights is the CloudWatch tool for searching logs with queries. In the console, go to CloudWatch, then Logs, then Logs Insights. Select /myapp/web, pick a time range and paste a query. Insights finds the JSON fields by itself.

# Recent errors
fields @timestamp, msg, route, tenantId, err
| filter level = "error"
| sort @timestamp desc
| limit 50

# Which endpoints are slow?
filter ispresent(durationMs)
| stats count() as requests, pct(durationMs, 95) as p95_ms by route
| sort p95_ms desc

Save a few more queries. One shows everything for one requestId. One shows status codes over time in 5-minute groups. One shows errors grouped by tenantId.

Insights charges for the data it scans (about $0.005 per GB). So make the time range and the log group small before you run a query over weeks of data.

Step 6 — turn logs and business events into metrics

A metric filter watches a log group. Each time a line matches a pattern, it adds to a metric. So an error in your code can trigger an alarm even when AWS never sees a 5xx. For example, you may have handled the error and returned a friendly 200.

In the log group, choose Actions, then Create metric filter. Use the pattern { $.level = "error" }, the metric MyApp/AppErrors and the value 1. Then create an alarm on that metric like any other alarm. For example, alarm on 10 or more in five minutes.

The same trick can monitor the business, which is often what matters most. Everything can look healthy on a technical level while signups have silently stopped. So send a metric for the events you care about.

You can send one data point for each event from your code, using the AWS SDK's PutMetricData call. A cheaper way is to write the event in Embedded Metric Format. This is a special log format that CloudWatch turns into a metric, so you need no extra API call.

Good events to measure are signups, logins, successful and failed payments, emails sent, finished background jobs and queue depth (how many jobs are waiting). An alarm on “zero signups for an hour during business hours” has caught more real outages than most technical alarms.

Step 7 — memory and disk with the CloudWatch agent

The agent is a small program that runs on each server. It reads numbers from the operating system and publishes them as custom metrics. Put it in the boot recipe (the user-data script of Lesson 12), so every new server gets it automatically. First attach the AWS-managed policy CloudWatchAgentServerPolicy to the instance role.

The agent's config is a small JSON file. It says to collect mem_used_percent and the root disk's used_percent every 60 seconds. The metrics go into the namespace MyApp/EC2, grouped by Auto Scaling group. (CloudWatch shows the disk metric as disk_used_percent.) Keep the file in SSM Parameter Store (an AWS service that stores settings) as /myapp/cloudwatch-agent, so every server reads the same file.

The user-data script needs three extra steps. Install the agent package. Fetch the config from SSM. Start the agent with that config.

Grouping by Auto Scaling group gives you one graph for “memory across all servers”. Without it, you would get one line for each short-lived instance ID. Then add the memory and disk alarms from the table (namespace MyApp/EC2, metrics mem_used_percent and disk_used_percent).

Step 8 — one dashboard

A dashboard is for the moment when someone asks “is everything OK?” It should answer in five seconds without scrolling. In CloudWatch, go to Dashboards and choose Create. Here is a layout that works for this architecture. It goes from top to bottom in the order that a request travels:

Your first three dashboards are free. After that, each costs about $3 a month. Keep one dashboard that the whole team knows by name. When you write Terraform for your setup, define the dashboard there too. Then it is rebuilt together with everything else.

How to alert without burning out your team

Alarms that warn about nothing get muted. A muted alarm system is worse than no alarm system. These habits separate experienced teams from the rest:

• Alert on symptoms, not causes. A symptom is what users feel. “Users are getting errors” and “the site is slow” always deserve attention. “CPU is 75%” may be perfectly healthy. Put cause metrics on the dashboard to help you diagnose. Do not send them to your phone.
• Every alert must lead to an action. If your reaction to an alert is “ignore it, that always happens”, delete it or change its settings that day.
• Two levels. A page wakes a person now. Use it for a total outage, errors that continue, or data at risk. A ticket can wait until tomorrow. Use it for a disk at 70%, a certificate that expires in 20 days, or one failed job. Send the two levels to different channels.
• Put a runbook link in every alarm description. A runbook is a short guide that says what the alarm means, how to confirm it, the first three things to try and who to call next. The person at 3 a.m. is tired and may not be the person who built the system.
• Alert when good things stop. No traffic, no signups, no finished jobs. These silent failures produce no errors.
• Review your alerts every month. Which ones fired? Which ones mattered? Which never fired? Alerts, like tests, go out of date.

When the alarm fires: a debugging routine

Have a routine. Then a tired mind can follow steps instead of guessing:

1. Is it real, and how big?

   Open the dashboard. Are errors up? Is it for everyone, or for one part (one tenant, one endpoint, one Availability Zone)? Since when? Is there a deploy marker just before?

2. Stop the bleeding first

   If a deploy caused it, roll back now (Lesson 14) and investigate afterwards. Getting the service back matters more than understanding the cause. You can also add servers, switch to a standby, or turn off a feature flag (a switch that turns a feature on or off). Use whatever restores service to users fastest.

3. Walk the golden signals down the stack

   Start with the ALB errors or latency. Then check the servers behind it: healthy hosts, CPU and memory. Then check what they depend on: RDS CPU, connections and storage, and Redis memory and hit rate. The first red graph in this chain is usually the cause.

4. Read the logs for the exact minute

   Run the “recent errors” and “errors by tenant” Insights queries for the time of the incident. Take one requestId and read its whole story.

5. Write down what you learn, while it is fresh

   Write the timeline, the cause and the fix. Then write the most valuable part: what would have caught it sooner? Add that alarm or log field. A blameless post-mortem is a review that looks for what to improve, not for who to blame. It turns each outage into a lasting improvement.

What this costs, and the log bill trap

Troubleshooting table

Interview corner

Practice task before Lesson 18

1. Create the SNS topic, subscribe your email and confirm it. Publish a test message and receive it.
2. Create the six alarms. Force each one into ALARM with set-alarm-state. Check that the email arrives and that its description would help a stranger.
3. Send logs to CloudWatch with the awslogs driver. Set a 30-day retention. Run aws logs tail while you click around the site.
4. Switch the app to JSON logging with request IDs and tenant IDs. Save three Logs Insights queries: recent errors, slowest routes and the story of one request.
5. Break something on purpose. Stop the app on one server, or make a route throw an error. Then watch three things: an alarm fires, you find the cause in the logs, and the recovery email arrives.
6. Install the CloudWatch agent on one server. Find mem_used_percent in the console. Add a memory alarm.
7. Build the dashboard. Sort one alarm as page-worthy and one as ticket-worthy. Write a three-line runbook for each.

Conclusion

You can now see the system, and it can call you. This changes how you run it. You no longer guess nervously. You follow a routine.

• Watch the four golden signals: latency (percentiles), traffic, errors and saturation. Remember that EC2 hides memory and disk until you install the agent.
• Alarms send messages to SNS. Confirm the subscription and test the path. Alert on symptoms, put a runbook in the description, and also alert when traffic disappears.
• Send logs off the server. Use structured JSON with request IDs and tenant IDs, and never log secrets. Search with Logs Insights. Turn log patterns and business events into metrics.
• One dashboard, ordered like the path of a request, with deploy markers.
• Log ingestion is the cost trap. Use the info level, set a retention time, and do not log health checks.

You can now observe the system. Next, make it hard to break into and impossible to lose. That means security hardening (making a system harder to attack) and backups.

End of Lesson 17. Next: Lesson 18 — Security Hardening and Backups.