Connection pooling and running out of connections
Why databases refuse new connections.
The food-delivery app has grown to 40 API servers. At peak, each server handles 50 requests at once and opens a database connection per request. That is 2,000 connections hitting a database configured for 500. The surprise is not that it slows down: new connections are refused outright, and every server logs 'too many connections' at the same moment. Why are connections so limited?
A database connection is not free. Each one is a session with memory, an authentication handshake and often a dedicated process or thread on the server. Many databases are tuned for a few hundred connections, not thousands. Worse, a database can only do so much real work at once, roughly bounded by its CPU cores and disk. A thousand connections all running queries just means a thousand queries fighting over the same 16 cores.
A connection pool keeps a small set of connections open and lends them out. A request borrows one, runs its queries and returns it. If every connection is busy, the request waits in line for a short time instead of opening another. The pool turns an unbounded rush into a bounded, orderly stream.
Sizing a pool, step by step
Step 1 of 4
Start from what the database can do
Suppose the database server has 8 cores. A common rule of thumb is a pool of a few times the core count, so roughly 20 to 30 active connections gives the best throughput. More does not mean faster; it means more contention.
Your turn
A single slow report endpoint is draining your connection pool and causing timeouts on checkout. Give two different fixes.
Stuck? Think about isolation and about shortening the hold time.
What to remember
Connections cost memory and processes; databases allow only hundreds.
A pool lends a bounded set of connections and queues the rest briefly.
Size pools from the database limit divided across all app servers.
Use a pooler proxy for big fleets and always set timeouts.