Pod review: Experimentation in Product Management
Estimated Reading Time: 8 minutes - time saved 41 minutes 🔥
In this episode of “The product experience”, titled "Experimentation," the hosts discuss the importance of experimentation in product development and share insights from their guests, Faith Forster, VP of Product at Dext, and Iqbal Ali, an experimentation coach and consultant. The episode covers various aspects of experimentation, including its definition, common mistakes, setting up a culture of experimentation, and its role throughout the product lifecycle.
🔍 Theme 1: Defining Experimentation
[00:06:16 - Introduction and definition of experimentation]
The hosts begin by asking their guests to define experimentation. Faith Forster describes it as a way to find data to make decisions when there is limited information available. Iqbal Ali agrees, defining it as a systematic data-driven approach to decision-making by mitigating risks and optimizing user experiences. They emphasize that experimentation goes beyond simple A/B testing and should be seen as a process rather than a one-time activity.
🔬 Theme 2: Common Mistakes in Experimentation
[00:08:54 - Common mistakes in experimentation]
The guests discuss common mistakes they have observed in experimentation. Iqbal Ali highlights the tendency to treat experimentation as a bolt-on process, only validating decisions after the product is developed. Faith Forster adds that not being clear on what you're trying to learn, improper measurement, and using experimentation to validate the product without understanding the problem are also common pitfalls. They stress the importance of focusing on insights and learnings rather than just statistical significance.
🧠 Theme 3: Setting Up a Culture of Experimentation
[00:11:15 - Setting up a culture of experimentation]
The hosts and guests delve into how to establish a culture of experimentation within a business. Faith Forster shares her experience at Dext, where they used experimentation to demonstrate progress and secure funding. They set up a system to prioritize experiments, focusing on the most impactful ideas and running quick tests to gather data. Iqbal Ali emphasizes the need to train teams in the experimentation process and align their roles with the overall goal. They also discuss the importance of reframing conversations and metrics to foster a culture of curiosity and learning.
🚧 Theme 4: Overcoming Resistance to Experimentation
[00:16:37 - Overcoming resistance to experimentation]
The hosts and guests address the challenge of getting people to embrace experimentation, especially when experiments yield negative results or challenge existing beliefs. They suggest reframing the conversation from failure to learning and focusing on insights gained rather than conversion rates. Faith Forster shares her experience of using experimentation to de-risk a major pivot at Dext, emphasizing the need to communicate the purpose and metrics effectively. They also discuss the importance of involving data experts or statisticians to guide decision-making.
🧭 Theme 5: Experimentation Throughout the Product Lifecycle
[00:19:41 - Experimentation throughout the product lifecycle]
The guests discuss how experimentation evolves throughout the product lifecycle. Faith Forster introduces a two-by-two matrix that maps the clarity of opportunity against the risk of getting it wrong. This matrix helps teams determine the right approach for each initiative, whether it's design-led, tech-led, or requires experimentation. Iqbal Ali adds that experimentation can be used to de-risk major changes during the discovery phase or to validate assumptions during the mature stage. They emphasize the importance of breaking down problems and using experiments to mine for insights.
⏰ Theme 6: Duration of Experiments
[00:35:24 - Duration of experiments]
The hosts and guests address the question of how long experiments should run. Iqbal Ali explains that the duration depends on variables such as traffic volume, conversion rates, seasonality, and the desired level of certainty. He mentions the concept of minimum detectable effect and the need to balance the time invested with the results obtained. Faith Forster suggests that the decision to continue or end an experiment should be based on the results obtained and the potential for further learnings.
⛗ Theme 7: Experimentation with Limited Traffic
[00:43:14 - Experimentation with limited traffic]
A listener raises the question of how to experiment with limited traffic. Iqbal Ali advises focusing on secondary metrics and using A/B testing to gather insights. He suggests looking beyond conversion rates and analyzing user behavior to understand the impact of changes. Faith Forster adds that user testing can also be valuable in understanding usability and perception issues. They emphasize the importance of de-risking solutions and adapting the experimentation approach based on the available traffic.
Overall, this podcast episode provides valuable insights into the world of experimentation in product development. The guests' experiences and practical tips offer guidance on setting up a culture of experimentation, overcoming resistance, and leveraging experimentation throughout the product lifecycle.
Podcast Link: [Apple Podcasts]
Timestamp Details:
[00:00:44] Experimentation at Mind the Product conference.
[00:03:34] Product experimentation and insights.
[00:08:54] Common experimentation mistakes.
[00:11:15] Experimentation for business success.
[00:15:09] Going to markets through small businesses.
[00:19:41] Fixing the language around experiments.
[00:24:04] Assessment and prioritization process.
[00:27:13] When to stop experimenting.
[00:33:14] Other forms of experimentation.
[00:35:11] How long should experiments run?
[00:41:05] Minimum customers for experimentation.
[00:44:24] A-B testing and metrics.
[00:46:59] Navigating change as a product leader.
If you liked (or didn’t like) this summary, be sure to let us know in the comments. Please also sign-up to our newsletter to be the 1st to receive our latest reviews
