Zoom Out: Precision, Accuracy, and Mitigating Risk

---

Decision-makers are increasingly looking to big data for decision-making. Though great for facilitating tactical decisions, an over-reliance on the same KPIs can lead to strategic blindspots. Use these 4 simple tips to mitigate this risk.

---

Increasingly, decision-makers at major companies are leveraging data to inform decision-making. It’s easy to see why: loyalty data, sell-through sales data, clickstream, social media campaign response and other quantitative data provide the ability to leverage  fact-based information to reduce decision risk. It seems that every day new innovative data sources become available to analyze customer attitudes and behavior. For market researchers and data scientists, this is great news because it means our skills in analyzing and interpreting complex data will remain in demand. 

Increasingly, decision-makers at major companies are leveraging data to inform decision-making. It’s easy to see why: loyalty data, sell-through sales data, clickstream, social media campaign response and other quantitative data provide the ability to leverage  fact-based information to reduce decision risk. It seems that every day new innovative data sources become available to analyze customer attitudes and behavior. For market researchers and data scientists, this is great news because it means our skills in analyzing and interpreting complex data will remain in demand. 

When budget, timing, and other circumstances allow, we may seek out larger datasets and increase sample sizes to ensure we have enough responses to measure statistically significant differences in the data. This allows us to drive greater precision in the data and can be useful for a variety of strategic and tactical decisions. But with this focus on data precision, we must also ask ourselves if the findings are also accurate. Does it tell a complete story? Equally importantly, are these findings actionable? 

Too much of a good thing 

It’s common for companies to calculate metrics such as market share to one tenth of a percentage point. Over time, such benchmarking can potentially show inroads by competitors. This can prompt deeper investigations in terms of what’s working and what’s not (categories, pricing/discounting, specific products/features, etc) and can help identify whitespace opportunities and more.

However, common issues that often accompany such data include false precision and blindspots.

Ascribing too much meaning to small changes may be an example of  false precision. Are monthly changes in market share meaningful or a reflection of statistical noise or one-off external events such as a competitor’s product launch, uncharacteristically bad weather, unusual promotional activity, etc.? Importantly, do my clients believe the results, or will they explain it away?

Despite a high degree of precision, internal beliefs and politics often undermine what the data is saying. If key stakeholders do not trust the findings, then the value of the data and analysis is limited. If stakeholders frequently challenge results by explaining extenuating circumstances, it may be an indication that the underlying metrics need to be reexamined.

Big data can lead to big blindspots

While large quantitative datasets certainly have their strengths, over-reliance can also be a source of blindspots. When focusing too heavily on KPIs, it is not uncommon to overlook emerging trends that may impact the business. For example, an overly granular focus on daily or weekly performance could mask larger issues related to competitive threats, cultural shifts, and changing consumer needs. There are countless examples where industry leaders focused too narrowly while ignoring changes in the bigger picture. Some high-profile examples include: 

• Blockbuster paying too much attention to metrics like revenue per title and late fees while ignoring the threats posed by new business models.
• Both Kodak and Polaroid focusing (no pun intended) on film camera market share and gross margin while missing the digital revolution, despite significant research in digital technology.
• The Borders Group bookstore chain measuring success based on retail footprint and total sales while underestimating the threat of e-commerce. 
• More recently, the spectacular failure of Quibi underscores the need to take a broad assessment of changing market dynamics rather than relying on the instincts of a few accomplished industry veterans.

Mitigating measurement risk

While these infamous blindspots were revealed after the damage was done, there are ways to proactively mitigate the risk of such oversight. Below are a few tips data analysts and market researchers can employ to help ensure they are measuring what matters.

1. Engage in meaningful dialog with decision-makers. Rather than simply delivering KPIs, be sure to ask what is surprising. When results aren’t in line with expectations, probe for hypotheses that can be followed up with further investigation.

2. Vary sources. Although a single source of truth is a desirable goal, do other sources lead to similar conclusions? If not, explore the differences before dismissing the conflicting results.

3. Imagine the extremes. After clarifying hypotheses, consider a future in which the results are 5X what was anticipated. What would you tell the CEO in terms of why things are so much better than expected? Now consider results 5X in the opposite direction. How would you explain these results to the CEO? The answers can often reveal blindspots in the research itself.

4. Present findings that pass the squint test. Results shown in an overly complex manner may be be indicative of false precision. Truly actionable findings should be simple to understand.

While these guidelines cannot prevent all blindspots, they help to reduce risk and lead to more insightful and actionable analysis.

How do you mitigate the risk of blindspots and ensure you’re focusing on the right things? Leave your suggestions in the comments.