I was analyzing retention for a new feature and my initial model showed a strong positive effect. I presented it to the PM and we moved forward with a broader rollout plan. A week later I realized I hadn't properly segmented new versus returning users — the effect was real for returning users but near zero for new users. I corrected the segmentation, updated the numbers, and reshared the analysis. After that I made sure to always check user segmentation earlier in my process.
I built a model showing a feature drove a twelve percent lift in seven-day retention. The PM used it to justify a full rollout. Two weeks in, a peer DS flagged that my randomization unit was at the user level but the feature had social sharing mechanics — classic network interference. The lift was partially a social spillover, not a product effect. I killed the original estimate, reframed the analysis using a clustered design, and the true lift was closer to four percent. More importantly, I realized I had been defaulting to user-level randomization without asking whether the feature had cross-user dynamics. That question is now the first thing I ask before I design any experiment. The PM delayed the rollout based on the corrected number — which was the right call.