Medium · 13 steps · ~60 min

Design a metrics and monitoring system

Ingest millions of samples, keep dashboards snappy and wake humans only when it matters.

Design the system that watches everything else: thousands of hosts push numeric samples every few seconds, engineers query dashboards, and alert rules page on-call when something is wrong. It must absorb spikes, keep alerts firing while dashboards are slow, control cost as data ages, and not be brought down by one bad label.

Same problem as PrometheusDatadogGrafana

How it works

• Ingest: agents push batches of timestamped samples (write-heavy).
• Query: dashboards read ranges, repeatedly and in bursts.
• Alert: rules are evaluated continuously on the same data.

What you'll build

1. The API and the series schema
2. A buffered, batched write path
3. A cache for dashboards and a separate consumer group for alerts
4. A quota against cardinality bombs
5. A design that survives losing a store node, a consumer and a cold cache

Steps

1. 1Design the API
2. 2Design the data model
3. 3Handle 2,000 samples/sec
4. 4Handle 50,000 samples/sec for $3,200
5. 5Cardinality
6. 6Dashboards at 60,000 requests/sec for $1,100
7. 7Alert evaluation beside storage for $3,500
8. 8Alert evaluation
9. 9A label bomb sends 90,000 samples/sec for $900
10. 10Retention and downsampling
11. 11Lose a storage node
12. 12Lose a stream consumer
13. 13Dashboard cache restart