🔊 Caring Loudly 🔊
🔊 Caring Loudly 🔊
📊 Loving Data and Hating Surveys 🗳️
“What I hear when I'm being yelled at is people caring really loudly at me.”
-Leslie Knope, Parks and Recreation
I love data, but I hate surveys.
I know, ironic…
If a person asks me to fill out an evaluation of a program, seminar, webinar, etc., I rarely complete it (with one notable exception.) And if they ask me to fill out a long survey, I won’t complete it, on principle.
Do I leave surveys incomplete because I think surveys don’t matter? Not necessarily.
I could argue that I’m lazy, but that would be a lie.
Honestly, I don’t have a reason. But I suspect I’m not alone.
The sectors that gave birth to the Moneyball revolution were always rich in data; baseball assiduously collected statistics since its founding, and Moneyball as a strategy largely comes from the financial sector, where data is not only abundant but legally required to collect.
And unlike Moshe in Parashat Bemidbar, unless God commands you to produce a census, there is little we can do to make people provide us with data. Once, I had a job where I had to fill out a 10-page survey just to get my final paycheck. I was pissed; I probably gave more negative answers, just out of spite.
But I won’t blame you if this is a frustrating reality. How can we think in an evidence-based way if we cannot acquire the essential resource necessary to produce evidence?
This week, we are going to take a look at why (1) you need to commit yourself to finding the data, no matter how hard it may be, and (2) why sometimes the simplest data is the best.
Non-Response Bias
Picture a “town hall meeting.” Describe for me the type of person who is in attendance, and the type of person who actively participates.
If your initial response is, “People who want to complain,” you may be right. To be honest, I’m not sure. Because an assumption in your initial answer is that most attempts at group feedback do not result in a strong sample of the group whose feedback one desires. As such, the thinking goes:
People who are extremely happy with something tend not to show up to these meetings. Why waste time just telling someone that everything is awesome?
What about people who want things to change but are skeptical that things will change? Doesn’t seem worth it. Someone once told me, “Everyone talks about apathy, but nobody does anything about it…”
But who loves to show up? People who want to extrovert their feelings. Like the citizens of Pawnee, Indiana.
On the flip side, while most of us are wary of those people who say “Everyone tells me things are going great,” it is possible that the leader has surrounded themselves with sycophants to the point where everyone is telling them that things are awesome (i.e. groupthink, the Abilene Paradox, etc.). But that doesn’t mean things are actually great.
Each of these phenomena relates to a heuristic known as “participation bias” or “non-response bias,” where the people who do not participate in a survey or study provide systematically different responses from those who do, thereby skewing or tainting the salience of the data. While no single person can be given credit for coining either term, Science Direct provides a fantastic set of definitions concerning how this bias can manifest itself in psychiatry, politics, web-based surveys, etc.
Non-Response bias is a fantastic example of why bad data can, at times, be worse than no data. I do not need to remind people about the merits of third-party evaluation, quantitative surveys, house meetings, etc. But if we are not paying attention to who will or will not show up to provide data, we may end up spending a lot of time and money on something that is, at best, unreliable.
The One Number You Need to Grow
Of course, the non-response bias is a fairly obvious problem that only reinforces an issue facing many organizations, namely how to gather enough data such that one can draw evidence-based conclusions. If an organization with limited time and finite resources cannot create a useful data set, then all the preaching in the world about the importance of data is meaningless.
And this is why I love the net promoter score (NPS).
(Love may be too gentle a term. Former colleagues might charitably say I’m an evangelist for the NPS, or less charitably say I’m blindly devoted to it.)
The NPS is a formula one calculates based on how a customer answers the question, “How likely are you to recommend X to a friend or a colleague,” with a scale from 0-10. In case you are not familiar with the NPS, you can read a quick summary here. You probably interact with the NPS every day without even realizing it, as some variation of the NPS is used when you ride an Uber, finish a call with customer service, or rate the cleanliness of a bathroom at a rest stop.
For our purposes, the NPS is a great reminder to read The Ultimate Question by Frederick Reichheld, the foundational text of the NPS. However, if you already feel overwhelmed by my weekly book recommendations, the truth is that you can gain a decent understanding of Reichheld’s book simply by reading a piece he wrote in The Harvard Business Review entitled “The One Number You Need to Grow.” No offense, Fred.
Reichheld is an expert on customer loyalty and ultimately advocates for the NPS because of the relevant conclusions that can be generated from a single number. The NPS produces “timely data that is easy to act on,” whereas many surveys produce “complex information that’s months out of date” by the time anyone sees it.
Do Jewish organizations exist that are capable of the highest quality analytics using the world’s most sophisticated CRM or dashboard? Possibly. But I have yet to meet that organization. Most organizations need something powerful, trustworthy, and simple (i.e. cheap). The NPS meets all the criteria.
And for hypocrites like me, who love data but hate surveys, knowing that I can take a survey quickly will guarantee that even I will fill your survey out.
As a Reminder…
You can evaluate Moneyball Judaism anytime you want by inputting your NPS score. Hint hint…
Winning On Purpose
Fred Reichheld on his latest book and the legacy of the NPS.
Weekly Links
ChatGPT and Higher Fundraising? Yes, please: Read more. But we still need big action on AI.
How Helping Others Can Make You Feel Less Rushed: Funny how things work…turns out you DO have the time to help others, no matter how busy you claim to be. Here’s why.
We Need to Talk about Suicide: In truth, I had no idea that there was any conversation about nonprofit professionals, social justice activists, and suicide. Nonprofit AF changed my mind.
Creating A Culture of Listening: Serious subjects like the previous article is all the more reason we need to remember that listening is a core competence of effective leadership. Read more.
Gardening as a Communal Pastime: Dedicated to my former boss and favorite gardener, Kathy Elias.
Frederick R. Reichheld, “The One Number You Need to Grow,” The Harvard Business Review (December 2003).