Debugging with SigNoz

SigNoz is the front door to production behaviour: traces, metrics, and logs, all correlated. This is the Locate step of the debugging loop when the question is “which part of my app is slow or broken” — before you drop down into ClickHouse for the store itself.

Start from the symptom#

Match where you look to what you’re chasing:

The trace waterfall — your most powerful view#

When one request is slow, open its trace. The waterfall shows every span (each unit of work — an HTTP handler, a DB call, an outbound request) as a bar on a timeline. The longest bar is where the time went. This is the app-level version of “why is it doing so much work”: you can see that 900ms of a 1s request was a single Mongo query, or fan-out to five sequential calls that should’ve been parallel.

Reading it:

• Width = duration. Find the widest bar that isn’t just “the whole request.”
• Nesting = causality. A child span is work done inside its parent.
• Gaps = time not covered by any span — often network, queueing, or un-instrumented work.
• Red = an error span. Click it for the exception and stack.

From a span you can jump straight to that service’s logs for the same trace — trace and logs are correlated by trace id, so you go from “this span was slow/errored” to “here’s exactly what it logged” in one click.

Metrics: dashboards, and the RED method#

For “is this service healthy overall,” use a dashboard built on the RED method — Rate (requests/sec), Errors (error rate), Duration (latency percentiles). Our Application Performance dashboard is built entirely on spanmetrics (signoz_calls_total, signoz_latency), grouped by service.

One rule that saves you from slow dashboards: build panels from metrics, not from traces. Computing “request rate” by scanning raw trace spans is enormously more expensive than reading a pre-aggregated metric — the same metric is metric, trace is trace principle from cardinality. If a dashboard is slow, check whether its panels are secretly trace queries.

When SigNoz itself feels slow#

Sometimes the tool is slow because the system under it is strained — too much telemetry coming in, or expensive queries competing for CPU. Two moves:

1. The strain view. A dashboard that surfaces the offenders directly: top metrics by ingestion volume, top metrics by cardinality, and an export-frequency panel that flags any metric emitting far more often than once a minute (a service exporting every second instead of every 60s is a classic). This turns “it feels heavy” into a named culprit.
2. Drop into the store. SigNoz runs on ClickHouse; the real query costs live in system.query_log. See finding slow queries in ClickHouse — that’s where you confirm which query is heavy and why.

Reducing what you collect#

Not every span earns its place. Auto-instrumentation emits a span for every middleware, router layer, and DB-driver call — and those can be the majority of your span volume while adding little debugging value (the outer request span and the DB-call span already tell the story). Dropping that noise at the collector cut our trace volume by ~76% and shrank metric cardinality at the same time. Collect what you’d actually open a trace to see; drop the rest.

The loop, in SigNoz terms#

Measure (the latency number on the panel) → Locate (Services → the slow operation → open a trace) → why so much work (the widest span, or a metric query reading too much) → change one thing → re-measure. Same loop, tool-shaped.