Where Data-Driven Decision-Making Can Go Wrong

In “Where Data-Driven Decision-Making Can Go Wrong,” Michael Luca and Amy C. Edmondson argue that the biggest risk in data-driven leadership is not using too much data or too little data — it is using data uncritically. Leaders often treat evidence as either definitive proof or irrelevant noise, when the better approach is to pressure-test the evidence, ask whether it applies to the current business context, and invite rigorous discussion before making high-stakes decisions. The article appeared in the September–October 2024 issue of Harvard Business Review.

Executive Summary for Business Leaders

The article’s central message is that data should inform judgment, not replace it. Luca and Edmondson encourage leaders to examine both internal validity — whether the analysis accurately answers the question — and external validity — whether findings from one context can be generalized to another. Their framework is especially useful for executive teams evaluating research, experiments, dashboards, AI recommendations, workforce analytics, pricing studies, marketing attribution, or operational performance data.

Major Takeaways

1. Correlation is not causation. A metric may move alongside another without proving that one caused the other. Leaders should ask how the study was designed and whether confounding factors were controlled.
2. Sample size and confidence matter. A small or noisy sample can make a weak effect look persuasive. Executives should ask about confidence intervals, effect size, and statistical power before scaling a decision.
3. Measure what matters, not just what is easy. The article warns that analytics often overemphasize convenient metrics while missing long-term outcomes, unintended consequences, or stakeholder impact.
4. Context determines usefulness. Evidence from another company, geography, workforce, product category, or time period may not transfer cleanly to your organization.
5. Better data decisions require better conversations. Edmondson’s leadership lens is clear: teams need psychological safety, dissent, and curiosity to challenge assumptions instead of rubber-stamping the loudest or most senior voice.

Leadership Talking Points

Use these prompts in an executive meeting:

• “What decision are we actually trying to make?”
• “Does this data prove causation, or only show correlation?”
• “What important outcomes are missing from the analysis?”
• “How similar is the study context to our own business?”
• “What would change our mind?”
• “Who sees the evidence differently, and why?”

Reflection Questions

• Where are we currently treating dashboard metrics as facts without questioning how they were produced?
• Are we rewarding leaders for being data-driven, or for being data-literate?
• Which business decisions rely too heavily on short-term metrics?
• Do people feel safe challenging the interpretation of data in leadership meetings?
• What evidence would we need before scaling a pilot, experiment, or AI recommendation?

Potential Action Items

• Create a decision-evidence checklist for major investments, pricing moves, workforce changes, product launches, and AI-enabled recommendations.
• Require teams to distinguish between correlation, causation, and prediction in every executive data review.
• Add a “what are we not measuring?” section to dashboards and performance reviews.
• Ask analytics teams to include confidence levels, assumptions, limitations, and applicability notes in reports.
• Assign a rotating “data skeptic” role in leadership meetings to challenge interpretation constructively.

Similar Articles to Recommend

• “Leaders: Stop Confusing Correlation with Causation” by Michael Luca — a practical companion piece on why causal reasoning matters for executives.
• “Data-Driven Decisions Start with These 4 Questions” by Eric Haller and Greg Satell — useful for leaders who want a simple questioning framework before trusting analytics.
• “How to Make Sure You’re Not Using Data Just to Justify Decisions You’ve Already Made” by Kevin Troyanos — a strong follow-up on confirmation bias and retrofitted analytics.
• “When Analytics Should Drive Sales Decisions — and When They Shouldn’t” by Prabhakant Sinha, Arun Shastri, and Sally Lorimer — helpful for applying judgment to AI and analytics recommendations in sales.
• HBR IdeaCast: “Is Your Company Reading Data the Wrong Way?” — a related conversation with Edmondson and Luca on interpreting data in leadership teams.