Data-Driven Decision Making

“Let the data decide” sounds bulletproof, but data is only as good as the questions you ask of it. A confident chart can send a whole company in the wrong direction just as easily as a hunch can — sometimes more easily, because numbers feel objective. Being genuinely data-driven isn’t about worshipping the spreadsheet; it’s about knowing which numbers to trust, what they can and can’t prove, and when to set them down and use your head.

Correlation is not causation

The most expensive mistake in the whole field is confusing two things that move together with one thing causing another. Correlation means two numbers rise and fall in step. Causation means one actually makes the other happen. They look identical on a chart, and they are wildly different.

Here’s the classic trap. Ice-cream sales and drowning deaths rise together every summer. Ice cream doesn’t cause drowning — hot weather drives both. That hidden third factor is a confounder, and confounders are everywhere. Your premium customers churn less and use a certain feature more — so does the feature reduce churn, or do loyal people simply happen to use it? If you can’t rule out a confounder, you have a correlation and nothing more. Acting on a correlation as if it were causation is how teams pour money into things that change nothing.

Vanity metrics vs actionable metrics

The second trap is measuring things that feel good rather than things that help. A vanity metric is a number that reliably goes up and makes you look successful but never tells you what to do differently. Total registered users only ever rises — even if everyone left months ago. Page views, app downloads, social followers, cumulative anything: big, flattering, and almost useless for a decision.

An actionable metric ties to a specific cause and points to a specific action. “Week-one activation dropped from 60% to 45% after Tuesday’s release” tells you what happened, when, and where to look. The test is simple: if a metric moved, would you know what to change? If yes, it’s actionable. If it would only ever make you feel good or bad, it’s vanity. The cruelest part is that vanity metrics are usually the easiest to grow, so a team can look like it’s thriving while the numbers that matter quietly rot.

Baselines and A/B tests

You can’t tell whether a number is good without a baseline — the normal level before you changed anything. “Sales hit $90k” means nothing until you know last month was $70k or $110k. Always capture the before, or you’ll have no honest way to judge the after.

The cleanest way to prove cause is an A/B test: split your audience at random, show group A the old version and group B the new one, and compare. Because the only systematic difference between the groups is the change you made, a gap in results can fairly be credited to that change — this is one of the few times you can claim real causation. Two cautions, though. A tiny test can swing on pure luck, so the sample has to be big enough to trust. And a difference can be real but trivially small — “statistically significant” is not the same as “worth doing.”

Rule of thumb: before you trust a number, ask “compared to what?” A result without a baseline or a control group is a story, not evidence.

Cherry-picking and the limits of data

Even good data can be abused. Cherry-picking is quietly choosing the time window, segment, or chart that flatters your case and hiding the rest. “Revenue is up 40%!” — from the worst week to the best. Watch for oddly specific date ranges, a y-axis that doesn’t start at zero, and conclusions that only survive if you ignore most of the data.

And sometimes the data simply runs out. New markets have no history. Rare, high-stakes calls can’t be A/B tested. Ethics, brand, and long-term trust often don’t show up in this quarter’s numbers at all. That’s when seasoned judgment earns its keep. Being data-driven doesn’t mean surrendering to the spreadsheet; it means using data where it’s strong and being honest about where it’s silent.

Spot it: real or misleading?

Read each claim and decide what statistical issue it has, then tap a card to flip it.

Sort the red flags

Drag each item into the bucket it belongs to — or tap an item, then tap a bucket. Hit Check placement when you’re done.

How to use it

When someone presents a chart, ask the three questions that cut through almost everything: “Compared to what baseline?”, “Could a confounder explain this?”, and “If this number moved, what would we actually do?” When you share your own numbers, bring the baseline and the full window, not just the flattering slice. Lead with actionable metrics and treat vanity numbers as decoration, not evidence. And when the data genuinely runs thin, say so plainly: “The numbers don’t cover this, so here’s the judgment call and why.” That honesty builds far more trust than a confident chart that can’t survive a second question.