How to avoid trading short-term wins for long-term losses 👇
In Prod Dev, accurately estimating long-term impact of actions is crucial. Many teams typically focus on short-term metrics due to the simplicity and immediacy of data, often relying on traditional A/B tests. However, these tests often fall short in capturing extended impacts.
Future Incremental Value (FIV) helps measure those long-term effects of today's actions. Instead of just looking at immediate results, FIV provides insight into how actions impact key metrics over time, ensuring decisions are sustainable.
When I worked at Nextdoor, I faced this challenge many times. One particular case was the effect of push notification disables on weekly active users (WAU). Push notifications boosted short-term WAU but led to more disables, harming long-term WAU.
I used propensity score matching (PSM) to estimate FIV:
1) Calculate Propensity Scores: Model the likelihood of disabling notifications.
2) Match Users: Pair users who disabled notifications with similar users who didn't.
3) Estimate FIV: Compare long-term WAU between matched pairs.
4) Leverage FIV: Evaluate short-term results, balancing gains with long-term impacts.
This is fairly standard approach for inferring FIV, but what I find really cool is how Airbnb significantly expanded on this methodology by automating and scaling the FIV process:
1) Input: Teams submit configurations.
2) Data: Constructs groups and stores data.
3) Modeling: Processes samples, applies PSM, computes FIVs.
4) Dashboard: FIV results presented via a dashboard 🎉
The end product is a comprehensive, scalable, and automated framework that allows for standardized, data-informed decisions across various teams!
💡 Which got me thinking: could there be a business around providing such a tool? A specialized tool that automates PSM and calculates FIV? Is there already?
If you're interested in discussing further or exploring how these concepts can be applied to your work, let's connect! 👉 Follow Shane Butler or schedule a 1:1 chat: https://lnkd.in/g2zV_xFQ
🔗 Check out the full article on AirBnb's Tech Blog to learn more about scaling FIV: https://lnkd.in/gJvjMy55
#DataScience #CausalInference #LongTermImpact #DataDrivenDecisionMaking
I will transform your brand with powerful stories that sell | Helping Busy founders write copy that generates sales | Copywriter
6moI've heard people saying it to me, "it's impossible". Now I am working to make that possible. It's just a matter of time. 🤞