How Data Misleads

Data rarely lies outright. It misleads, and it does so almost always through what isn’t on the screen: the rows that were filtered out, the dimension you didn’t cut by, the events that never got logged. Knowing the common traps by name is how you catch them before they cost a decision.

If a number surprises you, treat that surprise as the signal. The first suspect is not reality; it’s the data. See Trust the Report, Doubt the Data.

Second of three. Reading Data covers the habits that catch most misreadings; this page names the traps that survive them; Trust the Report is what to do when a number and reality disagree.

The traps#

Survivorship / selection bias. You’re only looking at what’s left. “Our active users love feature X” ignores everyone who bounced before they ever saw it. Ask who is missing from this dataset, and whether they’d change the story.
Simpson’s paradox. A trend in the aggregate reverses inside every segment (or vice versa). Overall conversion drops while it rises for both new and returning users, because the mix shifted. Segment before you trust a headline direction.
Correlation is not causation. Two lines moving together may share a cause, run backwards from what you assume, or coincide. “Users who do X retain better” rarely means “make them do X.” Look for the mechanism, or test it.
The average hides the distribution. A mean flattens a bimodal or long-tailed shape into a number that describes no one. “Average time to first value: 8 minutes” can hide that half do it in 30 seconds and half never do. Read the median and the spread.
Vanity metrics. A big, always-up number that no decision depends on: total signups ever, page views. Impressive, useless. If it can’t go down and nothing changes when it moves, it’s decoration.
Small samples and noise. With few data points, randomness looks like signal. A “60% improvement” on 10 events is a coin flip. Check the sample size and whether the difference could just be noise.
Goodhart’s law. “When a measure becomes a target, it stops being a good measure.” Put pressure on a metric and people optimise the metric, not the thing it stood for: response time drops because tickets get closed faster, not solved. This is why metrics need counterbalances (see Trust the Report).
Denominator neglect. A rate presented as a count, or a count presented as if it were a rate. “Failures doubled” while traffic tripled is a rate that improved. Always separate the count from what it’s over.
Instrument and collection error. Sometimes the data is simply wrong: an event that stopped firing, double-counting, a timezone bug bucketing a day into the wrong slot, a default value polluting a field. The dashboard looks confident either way. A number that can’t be reconciled with reality is often this.
Silent gaps. The most dangerous data is the data that isn’t there. If failures aren’t logged, the failure rate looks perfect. Absence of evidence gets read as evidence of absence. Ask what wouldn’t show up in this data even if it were happening.
Confirmation bias. With enough cuts, you can find a slice that agrees with what you already believed. Decide the question and the cut before you look, not after.

The habit#

For any surprising or convenient number, run three checks: who’s missing (selection), what’s it over (denominator), and is it even real (instrument). Most misreadings die at one of those three. What survives all three is worth building on.