Debugging with SigNoz

SigNoz is the front door to production behaviour: traces, metrics, and logs, all correlated. This is the Locate step of the debugging loop when the question is “which part of my app is slow or broken”, before you drop down into ClickHouse for the store itself.

Start from the symptom#

Match where you look to what you’re chasing:

The trace waterfall: your most powerful view#

When one request is slow, open its trace. The waterfall shows every span (each unit of work, an HTTP handler, a DB call, an outbound request) as a bar on a timeline. The longest bar is where the time went. This is the app-level version of “why is it doing so much work”: you can see that 900ms of a 1s request was a single Mongo query, or fan-out to five sequential calls that should’ve been parallel.

Reading it:

• Width = duration. Find the widest bar that isn’t just “the whole request.”
• Nesting = causality. A child span is work done inside its parent.
• Gaps = time not covered by any span, often network, queueing, or un-instrumented work.
• Red = an error span. Click it for the exception and stack.

From a span you can jump straight to that service’s logs for the same trace, trace and logs are correlated by trace id, so you go from “this span was slow/errored” to “here’s exactly what it logged” in one click.

Metrics: dashboards, and the RED method#

For “is this service healthy overall,” use a dashboard built on the RED method, Rate (requests/sec), Errors (error rate), Duration (latency percentiles). Our Application Performance dashboard is built entirely on spanmetrics (signoz_calls_total, signoz_latency), grouped by service.

One rule that saves you from slow dashboards: build panels from metrics, not from traces. Computing “request rate” by scanning raw trace spans is enormously more expensive than reading a pre-aggregated metric, the same metric is metric, trace is trace principle from cardinality. If a dashboard is slow, check whether its panels are secretly trace queries.

When SigNoz itself feels slow#

Sometimes the tool is slow because the system under it is strained, too much telemetry coming in, or expensive queries competing for CPU. Two moves:

1. The strain view. A dashboard that surfaces the offenders directly: top metrics by ingestion volume, top metrics by cardinality, and an export-frequency panel that flags any metric emitting far more often than once a minute (a service exporting every second instead of every 60s is a classic). This turns “it feels heavy” into a named culprit.
2. Drop into the store. SigNoz runs on ClickHouse; the real query costs live in system.query_log. See finding slow queries in ClickHouse: that’s where you confirm which query is heavy and why.

Reducing what you collect#

Not every span earns its place. Auto-instrumentation emits a span for every middleware, router layer, and DB-driver call, and those can be the majority of your span volume while adding little debugging value (the outer request span and the DB-call span already tell the story). Dropping that noise at the collector cut our trace volume by ~76% and shrank metric cardinality at the same time. Collect what you’d actually open a trace to see; drop the rest.

The loop, in SigNoz terms#

Measure (the latency number on the panel) → Locate (Services → the slow operation → open a trace) → why so much work (the widest span, or a metric query reading too much) → change one thing → re-measure. Same loop, tool-shaped.