← All posts · Engineering · February 27, 2024 · 6 min read
Taming metric cardinality
Cardinality is the number of unique time series a metric produces. A counter with no labels is one series. Add a label with a thousand values and you have a thousand series. Add a second and you may have a million.
The label that gets everyone
It is almost always user_id, or something shaped like it: a request ID, a full URL with query string, an email address. Each unique value is a new series that has to be indexed, stored and scanned at query time. One well-meaning label can quietly multiply your storage by four.
Rules that keep it in check
- Put unbounded identifiers in logs and traces, where you look them up by trace ID — not in metric labels.
- Bucket before you label:
status_class="5xx"instead of the exact status text. - Set a per-metric series budget and alert on the metrics approaching it, not just on the total.
Finding the offenders
Our cardinality explorer ranks metrics by how many series each label contributes, so you can see that http_requests_total is fine but http_requests_total{path=…} is 80% of your bill. Usually a single dashboard tells the whole story.