How to Debug Performance

The scariest thing about a slow production system is the feeling that you’d need to already know the answer — that debugging is a talent some people have. It isn’t. Debugging is a loop, and the loop is what finds the answer. You do not need to know the landmine’s name before you step near it; the loop hands you the name.

This page is the loop. Every other article in this group — cardinality, indexes and read amplification, finding slow queries in ClickHouse, debugging with SigNoz — is just a smell this loop will eventually walk you into.

The mindset shift#

You are not trying to recall the cause. You are trying to corner it.

A senior engineer isn’t someone who memorised every failure mode. They’re someone who runs this loop fast and has seen enough smells to recognise a few on sight. Both of those are learnable. The loop you can learn today; the smells you pick up one incident at a time — and each article here is one incident, pre-packaged so you don’t have to earn it the hard way.

The loop#

1. Measure — replace the feeling with a number#

“It feels slow” is not debuggable. “The p99 latency panel takes 19 seconds to load over a 7-day range” is. The first move is always to turn the vibe into a number you can watch go up and down. If you can’t measure it, you can’t know whether you fixed it — you’ll just move on when it feels better, which is how bugs come back.

2. Locate — find where the time actually goes#

Do not guess which part is slow. Ask the system. Almost everything has a flight recorder that will tell you:

• A database → its query log. In ClickHouse that’s system.query_log — see finding slow queries. It lists every query with its duration.
• The app → traces. Open the slow request in SigNoz and look at the waterfall: which span ate the time?
• A page → the browser’s Network and Performance tabs.

Locating is not solving. It’s narrowing “the system is slow” down to “this one query / span / call is slow.” You cannot fix what you cannot point at.

3. Ask “why is it doing that much work?”#

This is the heart of it, and it has a single golden signal:

Work done vs. result produced. A query that reads 15.5 million rows to return 20 is screaming that it’s doing 750,000× too much work — before you know why.

That ratio is the bug, even when you don’t yet have a name for the cause. High ratio → you ask the follow-up questions, and each one is just another query:

• Reading the wrong (too-fine) data? → check which table it actually hit (rollups).
• Not using the index? → EXPLAIN indexes = 1 (indexes).
• Too many unique things to chew through? → count them (cardinality).

Notice: the cause (cardinality) was discovered by the loop. Nobody started the incident knowing the word. The loop walked us from “reads 15.5M rows” → “why?” → “there are 12,000 unique series” → “what makes them unique?” → “an unnecessary label.” Each arrow was one query.

4. Change exactly one thing#

Form one hypothesis and test only it. If you change three things and it gets faster, you’ve learned nothing — you can’t tell which one worked, and two of them might be quietly making it worse. One variable at a time is slower per step and far faster overall, because you actually learn.

5. Re-measure — did the number move?#

Go back to the number from step 1 and check. Not “does it feel better” — did 19s become 3s? If yes, you’ve confirmed both the cause and the fix. If not, your hypothesis was wrong (great — that’s information), revert it, and loop again. “CI is green” or “it loaded once” is not acceptance; the original number moving is.

A worked example (a real incident)#

This is the loop, verbatim, from a real “the dashboards are slow” report:

1. Measure: the metrics dashboard took ~11s over 24h; traces and logs were fine. → metrics-specific.
2. Locate: system.query_log showed the slow queries read ~158M rows from the raw samples table.
3. Why so much? 158M rows for a chart with ~300 points → huge amplification. Follow-ups: which service emits all this? One service was exporting metrics every 1 second instead of every 60s — 60× too much data.
4. Change one thing: fix that one service’s export interval.
5. Re-measure: metric volume dropped ~98%; the 24h query fell from ~11s to ~2.5s.

No prior knowledge required at step 1. The loop produced the concepts (export cadence, then later cardinality and rollups) as it went.

The one-line version#

Measure → Locate → “why so much work?” → change one thing → re-measure. Everything else in this group is a smell you’ll recognise faster next time. You don’t need the map of every landmine. You need the metal detector — and this is it.