"Data-Driven" Usually Means "I Found Data That Agreed With Me"
The phrase "data-driven" has done more damage to organizational decision-making than gut instinct ever did.
At least when someone was running on gut instinct, you felt the spidey sense to push back. When they say data-driven, everyone nods and the bad decision gets carved into stone because data.
Confirmation Bias With a Spreadsheet Attached
Here's what data-driven actually looks like in practice.
Let’s say a program director wants to expand a service. She pulls utilization numbers, finds the one month where demand spiked, and builds her entire case around that month. The finance team wants to cut overhead. They pull the overhead ratio, stack it against the average data and present it as hard evidence. The board wants to stay the course. They find donor retention numbers that look stable and do… if you don't ask what happened to the donors who left.
Everyone is data-driven. Everyone already knew what they wanted to do.
It is confirmation bias with a spreadsheet attached. This isn't analysis. It's prosecution.
The problem isn't that the data is fake. The data is often completely real. The problem is the selective focus on what supports a preferred conclusion while quietly burying anything that doesn't. [1] You can't audit your way out of this. The numbers check out. The methodology looks clean. The bias is in what never made it into the deck.
The Tell Is in the Sequence
Real analysis starts with a question you don't know the answer to.
Most "data-driven" decisions start with a conclusion and work backward to find support. One of the premier global consulting firms puts it plainly: confirmation bias is the tendency to look for evidence that supports your hypothesis, or to interpret ambiguous data in a way that achieves the same result. It looks like this: "I have a hunch that investing in X would create value. Let's find some facts that back that up." [2]
The entire foundation of the scientific method is the opposite. You look for contradicting evidence. Almost no one in organizational life does this voluntarily.
If your analyst brings you findings that genuinely surprised them, you might be doing it right. If they bring you findings that confirm exactly what leadership already suspected, be skeptical. Not because the data is wrong. Because the question was probably too targeted to find anything else.
Nonprofits Are Especially Good at This
In group settings, people conform to the dominant viewpoint. Friction gets smashed into submission to maintain harmony. That's true everywhere, but in the nonprofit sector, the pressure is structural. [3]
Funders want certainty. Boards want reassurance. Staff want to protect the programs they built. Everyone has a reason to make the data behave. So organizations learn to present numbers as more conclusive than they are. They pick the metric that looks best. They report on the population that responded well. They build dashboards that make the board feel informed without surfacing the tensions that actually matter.
Over time, that habit bleeds inward. You stop being honest with yourselves because you got so good at being selective with funders.
The Overhead Ratio Is the Perfect Crime
Nothing illustrates this better than the overhead ratio, a genuinely terrible metric that tells you almost nothing about organizational effectiveness, and a sector that has known this for years and kept using it anyway.
Nonprofits have been so obsessed with keeping overhead low that they've been slowly gutting their own ability to function. We're not talking about trimming fat. We're talking about organizations that stopped replacing computers, stopped training staff, and let their offices deteriorate so badly that, in at least one documented case, the movers refused to move the furniture because it was too far gone. [5]
That's not efficiency. That's an organization eating itself alive to hit a number that doesn't even measure what anyone actually cares about.
That is what "keeping overhead low" looks like on the ground.
And here's the part that stings, organizations already know the metric is garbage. Research on four youth-serving nonprofits found that their actual overhead ran between 17 and 35 percent. What they reported to funders: 13 to 22 percent. [6]They weren't cooking the books out of corruption. They were lying because the metric was designed to punish anyone who told the truth.
In 2013, the CEOs of GuideStar, Charity Navigator, and BBB Wise Giving Alliance got fed up and wrote an open letter to American donors calling out the overhead ratio for exactly what it is. [7] Their argument was simple: the people served by charities don't need low overhead. They need results. That letter is now over a decade old.
Boards still pull up the ratio. CFOs still apologize for it.
The debunking didn't stick because the incentives never changed. Funders still reward the number. So organizations still chase it. And CFOs who try to make an honest case for investing in their own infrastructure, because the team is underwater and the systems are held together with exported CSVs and sheer willpower, get told the data doesn't support it.
The data was never designed to support it. That's not an evidence problem. That's a power problem dressed up as an evidence problem.
More Data Isn't the Fix
The thing that actually improves decisions isn't better data. It's debate. Research on large organizational decisions found that high-quality debate made outcomes 2.3 times more likely to succeed. Not more sophisticated modeling. Not cleaner reporting. People in a room, allowed to disagree, allowed to be wrong, and not punished for either.
That requires something most organizational cultures don't have and almost none actively build: a genuine tolerance for being wrong in public.
Being wrong can feel like it costs something real, because sometimes it does. A program that doesn't work is funding you argued for. A metric that doesn't hold up is a funder relationship you now have to renegotiate.
The stakes make people defensive. Defensiveness makes people selective. And selectivity produces organizations that have optimized for looking right rather than actually being effective.
Run the Test
Find the last three major decisions your organization made. For each one, identify the data that was presented in support. Then ask: was there any data that pointed in a different direction and if so, where did it go? Was it in the appendix? Discussed and dismissed? Or did it never make it into the room?
If you can't answer that question, you don't have a data practice. You have a formatting convention.
The difference matters a lot less to your board than it does to the people your programs are either helping or failing.
The problem with analyzing your own data is that everyone in the room has skin in the game. The executive needs the program to work. The board member needs the position to hold. The CFO needs the numbers to land. Nobody is actually neutral and neutral is the only place truth comes from.
Big Left has no stake in what the data says. That's not a tagline, that's the entire structural advantage. A licensed Professional Engineer and MPA with concentrations in public finance and infrastructure policy, who has spent years inside the decisions that break organizations, reads your data differently than anyone sitting inside your building ever could. What we bring isn't a better-looking report. It's the first honest read your organization may have ever had. If you're ready for that, the Strategic Working Session is where it starts.
Citations
[1] Breaking the Cycle of Confirmation Bias in the Workplace SHRM. Notes that the problem with confirmation bias is not whether information is true, but how organizations selectively focus on what supports a preferred conclusion while dismissing anything that challenges it.
[2] Biases in Decision-Making: A Guide for CFOs McKinsey & Company, March 2025. Defines confirmation bias and cites research finding that for big-bet decisions, high-quality debate led to outcomes 2.3 times more likely to be successful.
[3] Impact of Confirmation Bias on Decision-Making with a Moderating Role of Tolerance for Disagreement ResearchGate / Sage Journals, April 2024. Drawing on Janis (1982) and Sunstein (2009), describes how group settings amplify confirmation bias through conformity and suppression of dissenting opinions.
[4] Guidestar, Charity Navigator, and Wise Giving Alliance Call for End to Overhead Obsession Nonprofit Quarterly, June 2013. Finds that use of overhead as a primary proxy has always been intensely flawed, with organizations starving themselves of necessary infrastructure.
[5] The Nonprofit Starvation Cycle Gregory, A.G. and Howard, D. Stanford Social Innovation Review, Fall 2009. Based on analysis of more than 220,000 IRS Form 990s and 1,500 in-depth surveys, documenting severe consequences of overhead underfunding.
[6] Strategies for Navigating the Nonprofit Starvation Cycle Walden University / ProQuest Dissertations, 2019. Citing Gregory and Howard (2009): four youth-serving nonprofits had actual overhead rates of 17-35% but reported 13-22% to satisfy funder expectations.
[7] BBB Wise Giving Alliance, Charity Navigator, and GuideStar Join Forces to Dispel the Charity Overhead Myth GuideStar / Candid, June 17, 2013. Historic joint letter from all three leading nonprofit information providers denouncing the overhead ratio as the sole measure of nonprofit performance.
