Dashboards

For anyone building or reviewing a dashboard, in SigNoz, Metabase or anywhere else. Structured as a kernel: what is going wrong, what we do about it, and the habits that follow.

Nobody was checking the numbers#

Every dashboard we own was built one at a time, by whoever needed it that week, with no shared standard and no check that what it displayed was true. The result was not ugly, it was wrong, and the wrongness was invisible because a panel that renders looks finished.

What that cost, all of it found in a single audit in August 2026:

• A replication outage ran for nine days with every panel green. The MongoDB analytics replica stopped syncing on 14 August. Twenty five panels watched the destination database, which stayed perfectly healthy the whole time, so metric freshness read zero minutes stale while Metabase served nine-day-old data to anyone building a report. We were measuring the process, not the replication.
• A panel errored on every single load and nobody noticed, because a broken panel and an empty one look the same at a glance.
• Requests / min overcounted by roughly 3x for months. It counted every span in a trace, so one HTTP request that produced a server span plus two internal spans counted as three requests.
• 122 panels were doing 442x more work than they needed to. The worst took 66.6 seconds; eight had degraded until they failed outright on memory.
• A 20-panel dashboard took down log ingestion on 28 July 2026. Opening it fired enough concurrent queries to saturate ClickHouse, which then could not accept writes. Two database boards had since grown to 27 panels, larger than the one that caused the outage.

None of this was carelessness. Each panel was reasonable when it was written. What was missing was a bar for calling one finished.

Done means acted on, not rendered#

A dashboard is finished when someone can act on it and its numbers have been checked against real data.

That rules out four things we used to do routinely:

• Shipping a panel you have not executed. Not previewed, executed, against production data, with the result read.
• A percentage without the count beside it. 50% of two requests is noise, 0.4% of two hundred thousand is an incident, and the percentage alone cannot tell you which.
• A title that names the widget instead of the subject. Two of our boards were called “(selectable instance)”, which describes a dropdown.
• Copying a board to another instance without re-checking what it assumes. One board spent months advertising a different instance’s services in its own description.

Everything below follows from that one sentence.

Name it for the subject, not the widget#

Owner · Subject. The owner prefix earns its place only when the instance hosts more than one owner.

Bad: Payments DB (selectable instance), and Application Performance (RED) sitting beside APM Endpoints (RED by route).

Good: PostgreSQL, APM · Services, APM · Endpoints, Billing · Invoicing Funnel.

The first set fails three ways. “(selectable instance)” describes a dropdown, and every board with a variable has one. The second pair are near synonyms for two different scopes, so the names cannot tell you which to open. And the separator changes between boards, which is how you end up with five conventions across ten dashboards.

Three rules:

1. The subject is what is being observed, not how the page works. If the title needs a parenthetical to explain its own UI, rename it.
2. Add the owner prefix only to disambiguate. On a single-product instance, prefixing everything with that product’s name carries no information. On a shared one it carries a lot: Billing · Invoicing, Support · Queue, SigNoz · Ingestion Strain.
3. Shared infrastructure takes no owner. Server Health and VPN Gateway serve everyone.

How do I tell? Read the titles as a list with no other context. If you cannot say which one to open for a given question, the names are doing no work.

Run every query before you ship it#

A panel you have not executed is a guess with a chart on it.

Executing it is the cheap part. The rule that catches real defects is asserting something about the output, because a query can run cleanly and still be wrong:

• Row count above zero, except where empty is a meaningful answer. A “most errors” table filtered to rows with errors returns nothing when nothing is failing, and that is correct.
• Series count matches expectation. Selecting three endpoints must produce three lines. This one is worth automating: an early version of our per-endpoint panels grouped by time only, so three selections produced one combined line and the selector appeared broken.
• Values are plausible. An average latency of 0.0000162 ms is not a fast service, it is a bug. Assert a range and refuse to publish outside it.

When you change a query, run the old and the new side by side and compare the output rather than assuming they are equivalent. Two traps: a live counter moves between the two runs, so compare over a window ending in the past; and rows tied on the sort column come back in no fixed order, so sort by every grouping column before diffing, or you will chase a mismatch that is not there.

Counts beside percentages, limits beside numbers#

A number with nothing to compare it to cannot drive a decision.

Bad: a tile reading 47 connections. Healthy, or one away from the cap?

Good: 47 / 100 (47%), with the ceiling plotted on the same chart.

Two habits carry this:

• Pair every metric with its limit, and prefer the utilisation percentage as the headline. Where the ratio is the signal, show the ratio: cache hit rate, rollback rate, error rate.
• Show the count next to every percentage. Both, always. This is the same rule as the ceiling, applied to the other direction.

Counters climb forever, so plot the per-second rate rather than the cumulative total. Rate needs at least two samples per bucket, so keep the bucket interval at roughly twice the scrape interval or wider.

Read the aggregate, not the raw events#

Query pre-aggregated metrics, not the raw traces or samples underneath them.

Bad: computing p95 by running quantile(0.95) over every span in a week of raw trace rows.

Good: reading the same p95 from spanmetrics, letting the tool pick the pre-aggregated rollup for the time range.

This is the mistake with the widest blast radius, because the raw version returns correct numbers. It just costs twenty times more to get them, on a box that is also trying to accept writes. Three module dashboards built this way took roughly 2.5 seconds per panel; rebuilt on spanmetrics the same panels came back in 0.2 to 1.7 seconds over a seven-day window.

The corollary is worth stating plainly: if the tool’s query builder can express your panel, use it instead of hand-written SQL. It selects the right table for the time range automatically, which is a decision you would otherwise have to revisit every time someone changes the time picker.

The other query rules, span-kind filtering, per-series deltas on counters, deduplicating an append-only series index, and the ClickHouse limits that stop a heavy board starving ingestion, are mechanics rather than standards. They live in building dashboards.

Keep the board small enough not to melt the box#

A dashboard is a query generator, and N panels means N concurrent queries on every refresh.

If a panel takes longer than the refresh interval, queries arrive faster than they finish and pile up without limit. ClickHouse saturates, stops accepting writes, the collector’s inserts time out, and ingestion stops. That is not hypothetical: it is the 28 July outage, where a 20-panel board reached 59 concurrent queries and load went from 35 to 63.

Budget the panels. Fewer, denser panels beat thirty small ones, and our boards now sit at 15. Bare inventory counts, the number of tables or collections, belong in one collapsed row rather than six separate tiles.

Before you publish#

• Every query executed against real data, and the output read.
• Series count asserted where a selector drives the panel.
• Values within a plausible range.
• Counts shown beside percentages, caps beside numbers.
• Title names the subject; owner prefix only if the instance needs it.
• Panel count budgeted, and the board’s query cost sanity-checked.
• Description says which panels respond to which selector, and names what actually reports here.

Everyone on the team has shipped a panel that failed one of these, including whoever wrote this page. The point of the list is that it is cheaper to run than to discover the failure nine days later.