Test by segment, not by average

You run a test. Overall result: 0% lift. No significant difference. You move on to the next test.

But if you split the data by new vs. returning visitors, a different picture emerges. New visitors converted 20% better with Version B. Returning visitors converted 15% worse. The average washed it out to zero.

This happens more often than most teams realize. Different segments respond differently to the same change. Mobile vs. desktop. Paid vs. organic. First-time vs. repeat buyer. A change that helps one group can hurt another, and the overall average tells you nothing happened.

This doesn't mean you should mine every test for segment-level wins. That leads to false positives. It means that when a test comes back flat and you expected a result, check segments before calling it dead. The insight might be that the change works for a specific audience, not everyone.