posted on: Monday, December 15, 2008

By Niki Jagpal and Lisa Ranghelli

Ken Berger, the CEO of Charity Navigator, recently blogged about his goal of identifying new measures of nonprofit effectiveness that go beyond simply rating an organization’s financial health. He rightly points out that financial efficiency is only ‘part of the puzzle’ and is not necessarily always associated with positive outcomes. He indicates that Charity Navigator will be developing an expanded rating system to more comprehensively evaluate nonprofits. This also aligns with the IRS’s efforts to include efficiency indicators when it revised the form 990 earlier this year (the indicators didn’t get included but the new 990 does include governance information).

This move toward using better metrics to help individual donors decide which charitable organizations to donate to has striking parallels in institutional philanthropy. For example, there continues to be a lot of debate about what types of information foundation leaders should collect from their grantees and how grantees’ work should be evaluated best.

The bottom line, of course, is impact. Is the nonprofit organization having an impact on the issues and communities it seeks to address? From the perspective of individual or institutional donors, is their check or grant contributing effectively to achieve that impact? As Buzz Schmidt recently noted, foundations lack accountability mechanisms to make them show that their grantmaking is even having impact.

Although Charity Navigator’s leadership is to be commended for seeking to broaden donors’ conception of effectiveness beyond financial stability, there are myriad problems with relying on outcome metrics. And anyone, individuals or grantmaking institutions, attempting to do so should proceed with caution. One challenge is that measures need to be tailored to the roles and missions of different types of nonprofits. An organization that engages ordinary people in the democratic process and helps them advocate and organize to improve their communities will have very different benchmarks and measures of success than a food pantry or an arts organization. And with any of these types of organizations, it is much easier to find out whether the nonprofit did something, but much harder to determine if that achievement made a difference.

Michael Edwards wrote about what some call the ‘tyranny’ of business metrics in Just Another Emperor? The Myths and Realities of Philanthrocapitalism. Because the bottom line for evaluation in business is profit, measures aren’t easily transferable to the civil society sector where the bottom line is impact. Yet, many in institutional philanthropy appear to have embraced wholesale the idea that predetermined metrics built in from the inception of a funded piece of work is the ultimate way for grantmakers to evaluate the impact of their grants.

Take a look at the hard and uncertain work of advocacy, organizing, and civic engagement: it can take years to achieve a desired impact. How can a uniform measurement tool account for the interim years, when important work is being done toward a larger goal that has yet to be attained but very well might be? How will metrics capture the strengthened social fabric in a community as neighbors unite to rally around a common cause?

In recent years, there have been tremendous strides made by advocates, philanthropists, and researchers to develop benchmarks and measures for these strategies. NCRP drew on these efforts to document the impact of advocacy, organizing, and civic engagement in New Mexico (see Strengthening Democracy, Increasing Opportunities). But the key distinction is that we measured impacts across a sample of 14 groups over a five-year period. We didn’t evaluate individual organizations on a yearly basis. The aggregate accomplishments were quite impressive, but in any one year for a single group they may not have been.

As Charity Navigator moves forward, we’d offer the same advice that should inform a foundation’s approach to evaluating its grantees’ impact:
1. Seek input from many quarters of the nonprofit community to ensure that any new measures are relevant to the diversity of goals within the sector;
2. Ensure that data collection tools are not overly burdensome for either grantees or donors; and most importantly,
3. Make the outcome measurement process valuable to the grantees, helping them use the information to improve their own effectiveness.

Donors, individuals or institutional, can expect an evaluation that uses these three guidelines to provide them with value, too: it can help them figure out whether their contributions are actually having broad social impact today and in the future. By making evaluation more reasonable and following these guidelines, we’ll avoid the counterproductive scenario of evaluation for the sake of evaluation, with a sizeable share of outcome measurement reports never being read or used for future grant allocation or nonprofit program decisions.

Perhaps the best thing Charity Navigator could do is educate donors about what to look for in an organization to determine its effectiveness. Empower donors to do their own due diligence by teaching them to read between the lines of annual reports, analyze media coverage of an organization, and find other clues to assess its impact. Teach donors in plain terms about the concept of effectiveness and share what research has shown is evidence of that effectiveness. Robert Herman and David Renz found that nonprofits that can engage with and respond to stakeholders are better positioned to achieve their mission than those that do not. Encouraging donors to learn more about the organizations they want to give to will ultimately strengthen donors’ commitment to those groups and give donors the sufficient skills to assess any potential gift recipient in the future.

Niki Jagpal is research director and Lisa Ranghelli is senior research associate at the National Committee for Responsive Philanthropy (NCRP).