Autoscaling: signals, cooldowns and cost
Adding and removing servers automatically as traffic changes.
Delivery traffic is not flat: it is low at 3pm and enormous at 7pm. If you size for the peak you pay for idle servers most of the day; if you size for the average you fall over at dinner. Autoscaling adds servers when demand rises and removes them when it falls. It relies on everything earlier in this module: stateless servers and a load balancer that can add or drop targets.
Autoscaling reacts to a signal. Common ones are average CPU, requests per server, queue length (for workers) or latency. A simple and popular rule is target tracking: pick a target such as 60 percent CPU and compute the desired size as ceil(current × observed / target). With 10 servers at 90 percent CPU, desired = ceil(10 × 90 / 60) = 15 servers. Scale-in works the same way in reverse when load is low.
Why naive autoscaling goes wrong
Step 1 of 4
Boot time
A new server may take 2 minutes to start, load code and warm caches. A sudden spike can overwhelm you before help arrives, so keep headroom, scale on leading signals, or pre-scale for known events.
Autoscaling is mainly a cost tool. Suppose a server costs about $0.10 per hour (an illustrative figure). Running 15 servers all day is 15 × 24 × 0.10 = $36 per day. If you need 5 servers all day and 15 for the 4-hour dinner peak, autoscaling costs 5 × 24 × 0.10 + 10 × 4 × 0.10 = 12 + 4 = $16 per day. That saving only exists because the servers are interchangeable and start quickly.
Your turn
Your queue-processing workers run at one per 1,000 messages in the backlog, and the backlog jumps to 20,000. Choose a scaling signal, a max, and what else could go wrong.
Stuck? Think about the downstream database as well as the worker count.
What to remember
Autoscaling resizes the server group automatically based on a signal such as CPU, requests or queue length.
Target tracking: desired = ceil(current × observed / target).
Boot time, flapping and shared bottlenecks are the usual traps: use cooldowns, headroom and limits.
It saves money only when servers are stateless and quick to start.