Measuring "Impact": New Charity Ratings Group

In the year 2019, “impact” became one of the hottest buzzwords in the nonprofit sector. But nonprofits have struggled to measure “impact.”

Just ahead of last month’s Giving Tuesday event, a new ratings organization – ImpactMatters – appeared on the philanthropy scene. Whether or not this new watchdog group meets an unmet need – and, if so, whether it does the job well – is an open question.

Struggling to Measure Impact
”It’s no longer good enough,” we wrote in Struggling to Measure Charitable Impact (October 22, 2019), “to have programs and activities that are popular and apparently well-received or to have services that beneficiaries often use.” Now, funders want their grantees to demonstrate “impact.”

As explained by Michael Thatcher, the CEO of rating organization Charity Navigator, “previously – the discussion in the nonprofit sector has been about ‘… where the money is going.’ The focus is changing; ‘now it’s more of what the money is doing.’”

In Measuring for Impact: It’s Not a ‘One-Size-Fits-All’, the editor-in-chief of Nonprofit PRO, Nhu Te, explains in more detail how the nonprofit sector “has been struggling with figuring out the right way to articulate impact and across all of the different areas of the sector.” Part of the challenge is developing useful “taxonomy or language for measuring” it, given the breadth and scope of nonprofit organizations. But she emphasizes that while “quantifying impact is not easy,” that doesn’t mean that philanthropy experts and leaders shouldn’t try.

So how can outsiders – funders, perhaps – measure or evaluate the impact of a prospective grantee? It’s not easy.

Impact Matters: A New Approach
ImpactMatters is a startup created in 2015 by CEO and Executive Director Elijah Goldberg along with Dean Karlen, professor of economics and finance.

Why create a new ratings organizations? “There’s a lack of information available about the impact of nonprofits,” according to Mr. Goldberg. Telling prospective funders and donors about a nonprofit by “stories and anecdotes” is “a totally valid way” to ask for money, but that approach omits information about “cost-effectiveness.” ImpactMatters seeks to fill what it sees as a critical gap in the ratings process, notwithstanding what ED Goldberg acknowledges are “difficulties of measurement.” According to the website: “We rate nonprofits on their impact – how much good a nonprofit achieves per dollar of cost — so you can give confidently.”

ImpactMatters selects the organizations it will rate; so far it as evaluated some 1,000 groups around the United States. The rating scale ranges from three to five stars, using “publicly available information to estimate the actual impact the nonprofit’s program has” on the lives of the people served. It then “compares those impact estimates to benchmarks to determine if the nonprofit is cost-effective.”

Mr. Goldberg adds: “We always account for what would have happened in the absence of the program. For example, if a jobs program claims that 98 percent of graduates get jobs after they complete the program,” ImpactMatters asks “how many would have secured jobs had they not gone through the program?”

The new ratings service offers an “Impact Calculator that allows donors to see how their donation amount will impact the beneficiary and giving guides that showcase the most effective nonprofits by geographic region and cause.” There is also a free app for nonprofits called Guided Impact Reporting (GIR), that “uses information on the organization’s activities to calculate an estimate of impact.”

The Bill and Melinda Gates Foundation, along with other foundations and private donors, fund ImpactMatters.

Reaction
According to Mark Hrywna, writing for The Nonprofit Times in Another Nonprofit Ratings Organization Joins The Party (November 26, 2019), “if Twitter is any indication so far, the nonprofit sector is unimpressed with the concept.” After a profile in the New York Times, there was a “flurry of discussion among nonprofits, academics, and others.”

Mr. Hrywna notes that one such Twitter thread was started by University of Michigan Professor Megan Tompkins-Stange: “‘Rating nonprofits, especially small charities, based on their ‘bang for the buck,’ or cost-effectiveness per dollar spent vs. their ‘competitors,’ is very shortsighted. But it’s more than that. It’s destructive.”

Another example of skepticism is from CEO Paul Hamann of Chicago’s Night Ministry. He acknowledges the importance of “measuring and sharing impact,” but worries that “a rating that is reduced to a dollar-per-output needs to be taken with a grain of salt.” Mr. Hamann adds: There are complexities to measuring the impact of our services that can be lost in the translation.”

Conclusion
Elijah Goldberg acknowledges that his group is in “… uncharted territory on how a system like this would work for the sector.” In the coming months, the organization intends to gather feedback and make improvements to the product as well as “build out the number of groups rated and the audience of donors.”