We were building a new notifications feature and the user research wasn't complete yet. I had some early qualitative feedback from about fifteen users, and the eng team was ready to go. I made the call to ship to five percent of users rather than wait another three weeks for the full study. We launched, monitored the metrics, and engagement went up. It validated that the direction was right. I think the key lesson was that you can learn faster by shipping than by researching.
We were launching a re-engagement notifications feature on a low-connectivity surface. I had 7-day retention signal from a comparable cohort but was missing a full notification fatigue study — that would have taken four more weeks. I mapped what the missing data could change: if fatigue was high, we'd see unsubscribe rate spike above two percent in the first week. That was a measurable, fast signal. I set that as our kill threshold, shipped to three percent of users, and watched daily. Unsubscribes held at 0.8 percent. We rolled out fully. Day-30 retention improved 11 percent. The decision to ship wasn't about speed — it was about knowing exactly which signal would tell me I was wrong fast enough to reverse course.