How to Debug Performance

The scariest thing about a slow production system is the feeling that you’d need to already know the answer, that debugging is a talent some people have. It isn’t. Debugging is a loop, and the loop is what finds the answer. You do not need to know the landmine’s name before you step near it; the loop hands you the name.

This page is the loop. Every other article in this group, cardinality, indexes and read amplification, finding slow queries in ClickHouse, debugging with SigNoz, is just a smell this loop will eventually walk you into.

The mindset shift#

You are not trying to recall the cause. You are trying to corner it.

A senior engineer isn’t someone who memorised every failure mode. They’re someone who runs this loop fast and has seen enough smells to recognise a few on sight. Both of those are learnable. The loop you can learn today; the smells you pick up one incident at a time, and each article here is one incident, pre-packaged so you don’t have to earn it the hard way.

The loop#

1. Measure: replace the feeling with a number#

“It feels slow” is not debuggable. “The p99 latency panel takes 19 seconds to load over a 7-day range” is. The first move is always to turn the vibe into a number you can watch go up and down. If you can’t measure it, you can’t know whether you fixed it, you’ll just move on when it feels better, which is how bugs come back.

2. Locate: find where the time actually goes#

Do not guess which part is slow. Ask the system. Almost everything has a flight recorder that will tell you:

• A database → its query log. In ClickHouse that’s system.query_log, see finding slow queries. It lists every query with its duration.
• The app → traces. Open the slow request in SigNoz and look at the waterfall: which span ate the time?
• A page → the browser’s Network and Performance tabs.

Locating is not solving. It’s narrowing “the system is slow” down to “this one query / span / call is slow.” You cannot fix what you cannot point at.

3. Ask “why is it doing that much work?”#

This is the heart of it, and it has a single golden signal:

Work done vs. result produced. A query that reads 15.5 million rows to return 20 is screaming that it’s doing 750,000× too much work, before you know why.

That ratio is the bug, even when you don’t yet have a name for the cause. High ratio → you ask the follow-up questions, and each one is just another query:

• Reading the wrong (too-fine) data? → check which table it actually hit (rollups).
• Not using the index? → EXPLAIN indexes = 1 (indexes).
• Too many unique things to chew through? → count them (cardinality).

Notice: the cause (cardinality) was discovered by the loop. Nobody started the incident knowing the word. The loop walked us from “reads 15.5M rows” → “why?” → “there are 12,000 unique series” → “what makes them unique?” → “an unnecessary label.” Each arrow was one query.

4. Change exactly one thing#

Form one hypothesis and test only it. If you change three things and it gets faster, you’ve learned nothing, you can’t tell which one worked, and two of them might be quietly making it worse. One variable at a time is slower per step and far faster overall, because you actually learn.

5. Re-measure: did the number move?#

Go back to the number from step 1 and check. Not “does it feel better”, did 19s become 3s? If yes, you’ve confirmed both the cause and the fix. If not, your hypothesis was wrong (great, that’s information), revert it, and loop again. “CI is green” or “it loaded once” is not acceptance; the original number moving is.

A worked example (a real incident)#

This is the loop, verbatim, from a real “the dashboards are slow” report:

1. Measure: the metrics dashboard took ~11s over 24h; traces and logs were fine. → metrics-specific.
2. Locate: system.query_log showed the slow queries read ~158M rows from the raw samples table.
3. Why so much? 158M rows for a chart with ~300 points → huge amplification. Follow-ups: which service emits all this? One service was exporting metrics every 1 second instead of every 60s, 60× too much data.
4. Change one thing: fix that one service’s export interval.
5. Re-measure: metric volume dropped ~98%; the 24h query fell from ~11s to ~2.5s.

No prior knowledge required at step 1. The loop produced the concepts (export cadence, then later cardinality and rollups) as it went.

The one-line version#

Measure → Locate → “why so much work?” → change one thing → re-measure. Everything else in this group is a smell you’ll recognise faster next time. You don’t need the map of every landmine. You need the metal detector, and this is it.