Scaling WebSocket connections

Connection limits, sticky routing and pub/sub fan-out.

The chat app grows to 2 million people online at once. A single server cannot hold them: each open connection uses a file descriptor, memory for buffers (tens of kilobytes is a fair planning number) and some CPU for pings. A well-tuned gateway might hold 50,000 to 100,000 idle connections, so you need dozens of gateway servers, not one. Having many gateways creates two new problems: how clients land on them, and how a message from a user on gateway A reaches a recipient on gateway B.

For routing, a load balancer spreads new connections across gateways. Once a WebSocket is open, it stays on that gateway for its whole life, which acts as built-in stickiness. A reconnect may land on a different gateway, so nothing about a user's identity should depend on which one it is. For capacity, balance on connection count rather than request rate, and drain gracefully during deploys: stop accepting new connections, then ask clients to reconnect gradually.

The fix for cross-gateway delivery is pub/sub fan-out. Each gateway subscribes to the rooms its connected users belong to. When Alice sends a message, her gateway publishes it to the room's channel. Every gateway with a member of that room receives it and pushes to its local sockets. Gateways never need to know about each other. The cost is fan-out arithmetic: a message in a 500-person room becomes 500 deliveries.

Here the delivery workers can push 8,000 deliveries per second. A burst of 12,000 per second for three seconds, perhaps a popular room reacting to a goal, builds a backlog that peaks at 12,000 and drains once traffic falls back. Delivery during the burst is delayed by up to about a second and a half (12,000 / 8,000). Whether that is acceptable depends on the product; you size consumers for the delay you can tolerate.