Reading Data

Data is evidence, not truth. A number is the answer to one specific question, measured one specific way, over one specific slice of the world. Reading it well means knowing exactly what question it answers, and, just as important, what it doesn’t.

This page is about reading data honestly. Its companions are How Data Misleads (the traps) and Trust the Report, Doubt the Data (what to do when the numbers and reality disagree).

First of three. This page is how to read a number correctly. How Data Misleads is the catalogue of named traps, and Trust the Report is what to do when the number and reality disagree.

The kinds of data#

Before you read a number, know what kind of number it is. The distinctions change what it can and can’t tell you.

Distinction	What it means	Why it matters
Quantitative vs qualitative	A count or a rate, vs an observation or account	Both are data. The qualitative “report” from someone on the ground is often the truer signal; see Trust the Report.
Leading vs lagging	Predicts an outcome, vs records one that already happened	Leading indicators let you act; lagging ones only let you grade.
Stock vs flow	A level at a moment (users, backlog), vs a rate over time (signups/day)	Confusing the two is how a growing flow hides a shrinking stock.
Absolute vs rate	A raw count, vs a count over a denominator	A count with no denominator is almost always misleading. Always ask “out of what?”
Sample vs population	A slice, vs everyone	A slice only generalises if it’s representative and large enough.
Event vs state	Something that happened, vs how something is right now	State is reconstructed from events; if the events are wrong, the state is wrong.

The disciplines#

Five habits that catch most misreadings before they start.

Find the denominator. “300 failures” means nothing until you know out of how many. A rate beats a count almost every time. If someone shows you an absolute number, your first question is “out of what, over what period?”
Look at the distribution, not the average. An average is a single point standing in for a whole shape. If the shape is bimodal or has a long tail, the average describes no one: “average session length” can be a number that not one real session is near. Look at the median, the spread, and the histogram.
Segment before you conclude. Aggregates blur. A metric that’s flat overall can be rising in every segment (or the reverse; see Simpson’s paradox in How Data Misleads). Cut by the dimension that matters (device, cohort, region) before you believe the headline.
Trend over snapshot. One number is a dot; you can’t tell a spike from a level or a recovery from a decline. Look at the line, with a baseline to compare against.
Know the base rate. A result only means something against how often it happens anyway. A “40% lift” on something that occurs 0.1% of the time is a different claim than the same lift on something that occurs half the time.

Frameworks that turn numbers into decisions#

Reading is not the goal; deciding is. A few reusable shapes:

Funnel. Break a flow into ordered steps and measure conversion at each. The biggest drop is usually the highest-leverage place to work, and often the most surprising.
Cohort. Group by when someone started (or which version they got) and follow each group over time. Cohorts separate “did behaviour change” from “did the mix of people change”: the single most common way trends fool you.
North star + inputs. One outcome metric that captures real value, and the two or three input metrics you can actually move that drive it. If you can’t name the inputs, you can’t act on the outcome.
Segmentation. The same funnel or cohort, cut by a dimension, to find where the effect really lives. “It works, but only on desktop” is a different plan than “it works.”

The last question#

For any number, end on: so what, and now what? If a metric can go up or down and no decision changes, it’s telling you nothing you can use; it’s decoration, not evidence. A number you’d act on is worth reading carefully. A number you wouldn’t is worth cutting.