ALB and Auto Scaling: Scaling to 1000+ Concurrent Users

Concept

Two AWS services, one job: make your app survive success.

• An Application Load Balancer (ALB) is an AWS service that gives users one stable address. It shares their requests across many identical servers. It sends traffic only to servers that are healthy.
• Auto Scaling is an AWS service that decides how many servers exist. It starts more servers when traffic rises. It replaces servers that fail. It removes extra servers when the busy time is over.

Why one server is not enough

• A limit on capacity. A t3.small has 2 vCPUs (virtual CPU cores). After a certain point, extra requests only wait in line, slow down, and then time out.
• A single point of failure. This means one part that stops everything when it breaks. With one server, one crash, one bad deploy or one reboot for updates takes the whole site down.
• Deploys cause downtime. With one server, users have nowhere to go while you restart it.
• You pay for the peak all month. A server big enough for the busiest hour sits idle for the other 23 hours. Elastic capacity (capacity that grows and shrinks) charges you only for what you use.

Now the comparison between AWS and Vercel from Lesson 0 becomes real. Vercel scaled your app for you, out of sight. You are now building the same behaviour yourself. You can see every part and change every part.

The prerequisite: stateless servers

Load balancing has one strict rule: any request may go to any server. A stateless server keeps no data of its own between requests. If a server remembers something that the other servers do not, users get random bugs that are hard to find. Lessons 8 and 11 removed the two biggest causes (the database and the files). Check the rest before you scale:

What does “1000 concurrent users” actually mean

The lesson title promises 1000+ concurrent users. Before you buy anything, learn what that number means. “Concurrent users” are people who use the site at the same time. A request is one call from a browser to your server. Concurrent users are not requests per second. A person spends most of the time reading, not clicking.

Next, ask how many requests one server can handle:

These ranges are wide on purpose. You must measure your own app (see the load-test section). Say your app does about 60 requests per second per vCPU. Then a t3.small (2 vCPUs) handles about 120 requests per second. 1,000 casual users send about 100 requests per second. That is close to the limit of one server. You also need extra room for spikes and for the loss of one server, so you need 2–3 servers. That is why the group has a minimum of 2 (one in each Availability Zone, or AZ: a separate group of data centres inside a region) and a maximum of 6.

One more warning about t3 servers. They are burstable. They can use more than their normal CPU share only while they have CPU credits. Under steady heavy load the credits run out. Then the server slows down, or (in the default “unlimited” mode) AWS charges extra. For steady heavy load, test a non-burstable type such as m7i.

How an ALB works: listener, rule, target group, health check

Rules let one ALB serve many apps. A host-based rule looks at the domain name: api.yourapp.com goes to the API target group, and yourapp.com goes to the web target group. A path-based rule looks at the URL path: /api/* goes to one group and everything else goes to another. This way, one load balancer (about $20 a month) does the work of several.

Which load balancer?

AWS has four types. The “layer” is a level in the network model. A layer 7 balancer can read web requests. A layer 4 balancer only sees connections (TCP or UDP).

How Auto Scaling works: template, group, policy

There are three parts. Each one builds on the one before:

1. Launch template is a saved recipe for one server. It holds the AMI (a saved copy of a server disk with the operating system), the instance type (the size), the key, the Security Group, the IAM profile (the permissions the server gets), and a user-data script that runs on the first boot. It has versions, so you can change it and go back to an older one.
2. Auto Scaling Group (ASG) is a group that follows this rule: “keep between min and max servers from this template, spread across these subnets, and registered in this target group.” It replaces failed servers automatically.
3. Scaling policy is the rule that moves the wanted number of servers between min and max. For example: “keep the average CPU near 50%”.

Build it, step by step

You need the network from Lesson 6, an image in ECR (the AWS store for Docker images, Lesson 9) and a domain (Lesson 3). Each step is one console screen. We only mention the fields that matter.

1. Request the HTTPS certificate

   Open ACM (AWS Certificate Manager) in ap-south-1, the ALB's region. Request a public certificate for yourapp.com and *.yourapp.com. Choose DNS validation. This means you prove that you own the domain by adding a DNS record. Click “Create records in Route 53”, and the certificate is ready in a few minutes. The wildcard covers every tenant subdomain. The bare domain is listed on its own, because *.yourapp.com does not match yourapp.com itself.

2. Create the target group (the pool)

   Go to EC2 → Target groups → Create. Choose type Instances, HTTP port 80, and the myapp VPC. Set the health check path to /api/health, the interval to 15 s, healthy after 2 passes, and unhealthy after 3 failures. Do not register any targets. Auto Scaling will do that. Then, under Attributes, lower the deregistration delay from 300 to 30 seconds. This is how long the ALB waits for requests that are still running on a server before it removes the server.

3. Create the load balancer and its listeners

   Go to EC2 → Load balancers → Application Load Balancer. Choose internet-facing, the two public subnets, and the security group alb-sg. Add two listeners:

   • HTTPS 443: forward to myapp-tg, with the ACM certificate. The encryption ends here.
   • HTTP 80: redirect to HTTPS 443 (a 301 “moved permanently” answer).

   alb-sg (Lesson 6) already accepts ports 80 and 443 from everyone. web-sg accepts port 80 only from alb-sg. So the public can no longer reach a server directly. Remove the temporary 0.0.0.0/0 rules for ports 80 and 443 on web-sg that Lesson 10 added.

4. Store the deploy settings where new servers can read them

   A server that Auto Scaling starts at 3 a.m. has no human to tell it which image to run. It reads this from SSM Parameter Store. This is an AWS service that stores settings and secrets as named values. It is free for standard parameters. Create two:

   • /myapp/image-tag is a plain String. It holds the commit SHA (the short ID of a Git commit) to run, for example a1b2c3d.
   • /myapp/env is a SecureString (it is stored encrypted). It holds the whole production env file, including DATABASE_URL, plus KEEP_ALIVE_TIMEOUT=65000 (see the common pitfalls below).

   Then give myapp-ec2-role permission to read the /myapp/* parameters and to pull from the myapp ECR repository. Use the AWS-managed AmazonEC2ContainerRegistryReadOnly policy, plus a small inline policy that allows ssm:GetParameter on that path. Because the parameter is encrypted, the role also needs permission to decrypt it with its KMS key (KMS is the AWS Key Management Service, which holds encryption keys; the default AWS-managed key needs no extra rule).

5. Write the user-data script — the boot recipe

   User data is a script that runs once, as the root user, the first time a server boots. With it, a brand-new machine turns itself into one of your app servers while nobody is logged in:

   user-data.sh — the shape of itCopy

   #!/bin/bash
   set -euxo pipefail
   # 1. install Docker and the AWS CLI
   # 2. read which version to run, and its settings (never hard-code them)
   TAG=$(aws ssm get-parameter --name /myapp/image-tag --query Parameter.Value --output text)
   aws ssm get-parameter --name /myapp/env --with-decryption \
     --query Parameter.Value --output text > /etc/myapp.env
   # 3. log in to ECR and start the container on port 80
   docker run -d --name myapp --restart unless-stopped \
     --env-file /etc/myapp.env -p 80:3000 $REGISTRY/myapp:$TAG

   The container is published on port 80, because the target group sends traffic to port 80 and web-sg allows port 80 from the ALB. If the boot fails, read the log at /var/log/cloud-init-output.log.

6. Create the launch template

   Go to EC2 → Launch templates → Create. Fill it in like the server in Lesson 7 (IMDSv2 is the safer version of the metadata service explained below; gp3 is a type of SSD disk): Ubuntu 24.04, t3.small, web-sg, the myapp-ec2-role profile, a 20 GB encrypted gp3 disk, and IMDSv2 required. Paste the script above into User data. There are two differences from Lesson 7:

   • Metadata hop limit: 2. The instance metadata service (IMDS) is a local address where a server gets its role credentials. A container is one network hop further away from it than the host. With the default limit of 1, your app inside Docker cannot get the role's credentials, and every AWS call fails with “could not load credentials”.
   • No key pair, on purpose. Production servers should not accept SSH logins (SSH is the tool for logging in to a server from far away). In Lesson 18 you will get a shell through SSM Session Manager instead.

7. Create the Auto Scaling group

   Go to EC2 → Auto Scaling groups → Create, and use the launch template. These settings matter most:

   • Two subnets in two AZs, min 2, desired 2, max 6. If one whole data centre is lost, one server is still running.
   • Attach it to myapp-tg, and turn on ELB health checks. Then the group replaces servers that the load balancer reports as unhealthy. Without this, the group only checks the EC2 machine. It would miss a server whose machine is fine but whose app is dead.
   • Health check grace period 180 s. Do not judge a new server for 3 minutes while it boots and pulls the image. Make this longer than your real boot time.

8. Add the scaling policy

   On the same screen, choose Target tracking, the metric Average CPU utilisation, and the target 50. This means: “keep the average CPU across the group near 50%.” When the CPU is above 50%, the group adds servers. When it is well below, the group removes servers. AWS creates and manages the CloudWatch alarms for you (CloudWatch is the AWS monitoring service).

9. Watch it come alive

   The group's Activity tab shows “Launching a new EC2 instance…”. If something fails, it also shows why. After 2–4 minutes, the target group's Targets tab shows both servers as healthy. Then open the ALB's DNS name in a browser to reach the app.

Health checks done properly

The health endpoint decides which servers stay and which are replaced. Keep it honest and cheap:

export const dynamic = "force-dynamic";   // never cache a health check

export function GET() {
  return Response.json({ ok: true, version: process.env.APP_VERSION ?? "dev" });
}

Remember this rule: a health check that can fail because of another service (a dependency) can cause a chain of failures. A health check answers “is this server broken?”. An alarm answers “is the whole system broken?”.

Choosing a scaling policy

Why scaling is never instant

A new server needs a few minutes. It must launch, boot, pull the image, and pass two health checks. So scaling reacts a few minutes after the load rises. There are three ways to be ready for sudden spikes: a higher minimum, a scheduled action before an event that you know about, and a lower CPU target (40% leaves more room than 70%).

A good rule is to scale out fast and in slowly. Imagine you remove a server during a short quiet moment, and five minutes later you need it again. You waste time and money. This up-and-down switching is called flapping. Target tracking already avoids it by waiting longer before it scales in.

Deploying without downtime

With many servers, a deploy is no longer “log in with SSH and restart”. The clean pattern is replace the servers, do not change them:

1. CI (a robot that builds and tests your code on every change) pushes the new image to ECR, tagged with the commit SHA.
2. Update /myapp/image-tag to that SHA.
3. Start an instance refresh on the Auto Scaling group (min healthy 100%, max healthy 110%). An instance refresh is an ASG feature that replaces every server with a new one, a few at a time. AWS waits for each new server to pass its health check.

A rollback is the same move. Put the previous SHA back in the parameter and refresh again.

Min healthy 100% with max healthy 110% means AWS starts a new server before it retires an old one, so capacity never drops. With a very small group, a larger maximum such as 150% or 200% makes sure there is room for an extra server. The ALB only sends traffic to healthy targets, so users never reach a server that is still starting. The deregistration delay of 30 seconds lets requests that are still running on a retiring server finish. Lesson 14 starts the refresh from GitHub Actions.

Prove it: load test and kill a server

Do not trust a design until you have tried to break it. Two experiments turn diagrams into real confidence.

Experiment 1 — does it scale?

Look at three numbers in the hey summary, and at one graph:

• Requests per second. This is your measured speed. Use it to improve the capacity estimate above.
• Latency percentiles (p50, p95, p99). p95 means that 95% of requests were faster than this number. Averages hide problems. The slowest 1% of users are the ones who complain.
• Status code counts. Any 5xx (server error) means something broke under load.
• In CloudWatch, look at the ALB's TargetResponseTime and the database's DatabaseConnections. The database usually fails first.

Experiment 2 — does it heal?

While the load test runs, terminate one of the group's instances in the console, as if a data centre lost a machine. The ALB needs a few failed health checks, so it takes about a minute to mark the server unhealthy. A few requests may fail during that time. After that, the other server keeps serving the users. The group sees that it has fewer servers than wanted, and it starts a replacement by itself.

If the site shows errors for much longer than that, you have found a real weakness. It is usually a keep-alive timeout, a health check that is too slow, or a min-size of 1. It is cheap to fix it now.

The common pitfalls that cause real 502s

Keep-alive timeout: the intermittent 502

A keep-alive connection is a connection that stays open so that many requests can reuse it. The ALB reuses connections to your servers. It closes a connection after 60 seconds without traffic. Node closes idle connections after only 5 seconds by default. So sometimes the ALB sends a request on a connection that Node has just closed. The result is an instant 502. It happens for about one request in thousands, and you cannot reproduce it on your laptop. The rule is: the server's keep-alive timeout must be longer than the load balancer's idle timeout.

With the Next.js standalone server, you set this with the KEEP_ALIVE_TIMEOUT=65000 line that is already in /myapp/env (the value is in milliseconds). A custom Node server sets server.keepAliveTimeout = 65_000 and a slightly higher headersTimeout. Any value above the ALB's 60 seconds works.

Sticky sessions

The ALB can keep one user on the same server with a cookie. This is called “stickiness”. It is tempting when sessions live in server memory. But it spoils the even sharing of load, and the user loses the session when that server is replaced. Use it only as a short-term bridge while you move sessions to Redis (Lesson 16). It is not a real solution.

WebSockets

A WebSocket is a long-lived two-way connection (used for chat and live updates). ALBs support WebSockets with no extra setup. But such a connection ties a user to one server for as long as it lasts. Idle connections are closed after 60 seconds unless the app sends small “ping” messages. For such apps, scale on the number of connections, not on CPU.

Real client IP

Behind an ALB, every connection comes from the ALB. The user's address is in the X-Forwarded-For header. Set your framework to trust that header only from inside the VPC (see the real-IP note in Lesson 10). If you do not, every rate limit and log shows the ALB's IP.

What this costs

A realistic total for this tier is about $60–75 a month. That is much more than one server. What you buy is: no single point of failure, deploys without downtime, and room to grow. If you do not need this yet, one server with Nginx is a good choice. Do not add an ALB only because the course has a lesson about it.

Troubleshooting table

Interview corner

Practice task before Lesson 13

1. Check your app with the statelessness table. Write down anything that would break with two servers, and fix it or make a note of it.
2. Build the tier: certificate, target group, ALB, parameters, launch template, ASG, policy. Wait until both targets are healthy and load your site via the ALB DNS name.
3. Break the health check on purpose (change the path in the target group to /nope). Watch the targets turn unhealthy. Notice that the ALB still sends traffic to them. When all targets are unhealthy, the ALB tries all of them (this is called fail-open). Restore the path quickly. With ELB health checks on, the ASG will soon start to replace the “unhealthy” servers.
4. Run the load test and the terminate-a-server experiment. Record requests/sec, p95 latency, and how long the replacement took.
5. Publish a new image tag, update /myapp/image-tag and run an instance refresh. Confirm the site stayed up the whole time.
6. Work out the monthly cost of your setup at min-size 2, and at max-size 6 for one hour a day. Which line is the biggest?

Conclusion

You now run the same kind of setup that large sites use, only smaller: one address, many servers that are easy to replace, automatic repair, and automatic growth.

• ALB = listener + rules + target group + health checks. It ends HTTPS with a free ACM certificate.
• Auto Scaling = launch template + group + policy. Servers are cattle, made from a recipe with zero manual steps.
• Stateless first. Sessions, files, data and cron jobs must live outside the web servers.
• Health checks test the server, not its dependencies. Otherwise a short problem becomes an outage.
• Do the maths (requests per second, capacity per server, and the number of database connections at maximum size), and test by breaking things.

The ALB gives you a long, ugly DNS name. Next, we connect your real domain to it, and to CloudFront, with Route 53.

End of Lesson 12. Next: Lesson 13 — Route 53: Production Domain.