Medium · 13 steps · ~60 min

Design an API gateway and load balancer

One front door that authenticates, limits and routes without becoming the weak point.

Your company runs dozens of services and every one is exposed to the internet differently. Design the front door: a load balancer and an API gateway that spreads traffic, authenticates requests, enforces rate limits, routes by path and protects backends when they struggle, while staying highly available itself.

Same problem as KongAmazon API GatewayEnvoy

How it works

• Balance: spread connections or requests across healthy instances.
• Gate: verify identity, apply limits, pick the route.
• Protect: shed load and fail fast when upstreams misbehave.

What you'll build

1. The gateway API and its configuration schema
2. A load balancer and a cache for hot reads
3. A gateway with CDN and token offload
4. A rate limiter against an abusive client and a breaker for a failing upstream
5. A design that survives a lost balancer, a lost server and a cold cache

Steps

1. 1Design the API
2. 2Design the data model
3. 3Handle 2,000 requests/sec
4. 4A load balancer at 15,000 requests/sec for $900
5. 5L4 versus L7
6. 6A gateway as the front door at 40,000 requests/sec for $1,100
7. 7Where to verify the token
8. 8An abusive client sends 100,000 requests/sec for $1,300
9. 9Rate limit design
10. 10Shed load from a failing upstream for $1,000
11. 11Lose the load balancer
12. 12Lose an API server
13. 13A cold cache stampede