Evaluating Impact: Beware the Vanity Metric

Jul 27, 2023

Not all impact is created equal, and if you’re not well-versed in nonprofit measurement & evaluation (M&E) practices, it can be hard to tell the difference. Without a strong background, or internal resources to invest in quality M&E, many organizations will present statistics in their marketing materials and annual reports that have little-to-no connection with the impact their programs are actually having.

Does this sound familiar?

“2022 was a banner year at our organization. With 95% of our annual budget going directly to programs, we’ve never operated more efficiently! For the first time ever, our programs reached more than 750 children, and all but 4 of them reported complete satisfaction with our programs. Not only that, more than 10,000 people viewed the movie we made about the cause, and we grew our social media following by 150%.

Sounds pretty good, right? Not so fast. All of these statistics are known as “vanity metrics,” and as nice as they sound, they give us little indication of how effective an organization actually is. Let’s probe a bit deeper…

Vanity Metric 1: Overhead

Overhead can be one of the most misleading statistics any organization will report. Efficiency can certainly be a virtue, but so too is hiring key positions, and paying competitive salaries and benefits. So too purchasing and maintaining crucial equipment.

What about investment in innovation? With overhead so low, how are we actually implementing programs? Are volunteers staffing roles that should really be done by full-time employees? Even if we did have answers to all of these questions, it still doesn’t tell us anything about the quality of the work being done or how durable the outcomes are being created.

When considering overhead, it’s important to ask questions like these. As funders, we should be looking at how effective the programs are at achieving their missions, and how deep and lasting the results are.

All things being equal, reporting only 5% overhead is probably more of a red flag than a virtue. As long as they’re a solvent and following strong business practices, they probably should be spending a bit more on infrastructure.

Vanity Metric 2: Reach

It’s impressive that so many people watched the movie, and so many kids were reached, isn’t it? Of course, but things like “reach,” “attendance,” “graduates,” and “viewers” don’t tell us anything about what really happened.

It’s nice that so many children were reached, but reached how?

Were they given flyers or pamphlets, or did they participate in services over an extended period of time? Did they complete course work of any kind, and if so, what difference did those services make in their lives? Are they safer, smarter, or more confident? Did they learn any new skills, or are they reading at a higher grade level than they would have otherwise?

For people who viewed the movie, what followup actions were they asked to complete, and how many of them did so? As a result of their viewing, did they learn anything new? Did they build a stronger bond with the organization, or was the movie itself a disincentive for continued engagement with the program?

Vanity Metric 3: Client Satisfaction

We always want people to be satisfied with our services, don’t we? Of course we want positive interactions, and it’s always important to get feedback from stakeholders to drive innovation. However, satisfaction in the social sector can be misleading.

Sometimes organizations need to do things to serve clients as part of their drive to create positive social impact that those clients might not like. What about food pantries that serve only healthy options in an effort to combat obesity? Their clients might prefer less healthy options and therefore be less satisfied with the service.

Counselors may be ethically or legally bound to tell clients things they don’t want to hear. What about residential treatment or rehab programs? What if clients love the residence so much that they never want to leave? There are many programs out there that create a cycle of codependency that inhibits the very results they seek to achieve.

So, as philanthropic funders, what should we be looking for…

Impact is what happens after. Impact is what people do with the information or services they’ve received. It’s the extent to which their lives were improved, and for how long.

It’s important to examine how an organization is performing relative to its size, revenue, and stage of growth - both against its own potential, and against how other, similar programs perform. To the extent possible, it’s important to consider causation - would the outcomes an organization reports have happened anyway, without their intervention? Did their programs accelerate, inhibit, strengthen, or weaken the results their clients would’ve seen without their support?

Quality impact evaluation is hard. It’s complex, and it requires deep domain expertise and cause (and region) specific understanding. It’s because it’s hard that so many organizations understandably revert back to vanity metrics, and why so many funders accept them at face value.

For philanthropists, foundations, companies and other organizations looking to have real and lasting impact, it’s important to start by asking questions, and to keep asking them until the true picture of organizational effectiveness begins to emerge. For more on how to do this well, check out our blog on key metrics to look for when evaluating programmatic impact.

The Altruous Team

Staff

Keep Reading...

The $18 Trillion Opportunity You're Probably Ignoring (And Why Your Clients Are Going Elsewhere for Philanthropy Advice)

While wealth managers focus on optimizing portfolios, their clients are making crucial philanthropic decisions—involving $18 trillion over the next 25 years—without professional guidance. Despite 91% of high-net-worth households making charitable donations, only 5% of wealth managers feel confident discussing philanthropy with clients. This disconnect creates the largest missed opportunity in wealth management today. This article outlines five strategic approaches for wealth managers to integrate philanthropic planning into their practice, from starting with client values rather than vehicles to mastering planned giving conversations. Advisors who offer charitable planning report 6x the median assets and 3x the organic growth of those who don't, while also creating deeper client relationships that transcend investment performance.

The Altruous Team

Beyond Greenwashing: How Corporate CSR Can Actually Move the Needle

Many corporate social responsibility (CSR) programs are essentially "greenwashing," offering minimal impact despite significant funding. This article, part of a series on philanthropic impact, argues for transforming CSR from mere feel-good activities to strategic social investment. It outlines five ways companies can achieve this, and emphasizes how a strategic approach to CSR not only drives genuine social change but also offers measurable business benefits like increased employee engagement, stronger brand reputation, and improved customer loyalty, especially as millennials and Gen Z increasingly expect authentic corporate responsibility.

The Altruous Team

Why Playing It Safe is the Riskiest Strategy (And What Institutional Foundations Should Do Instead)

Institutional foundations have optimized themselves for avoiding mistakes instead of making impact. Committee structures and risk-averse processes create institutions that excel at avoiding embarrassment but fail at solving problems. While nonprofits struggle with declining funding, many foundations continue operating like it's 2005—funding safe, established organizations doing incremental work rather than taking calculated risks on innovative approaches. This article outlines five strategies for transforming institutional grantmaking from risk management to impact management, including funding capacity over programs, making genuine multi-year commitments, supporting policy advocacy, and embracing high-integrity philanthropy that balances trust with accountability.

The Altruous Team

The $18 Trillion Opportunity You're Probably Ignoring (And Why Your Clients Are Going Elsewhere for Philanthropy Advice)

While wealth managers focus on optimizing portfolios, their clients are making crucial philanthropic decisions—involving $18 trillion over the next 25 years—without professional guidance. Despite 91% of high-net-worth households making charitable donations, only 5% of wealth managers feel confident discussing philanthropy with clients. This disconnect creates the largest missed opportunity in wealth management today. This article outlines five strategic approaches for wealth managers to integrate philanthropic planning into their practice, from starting with client values rather than vehicles to mastering planned giving conversations. Advisors who offer charitable planning report 6x the median assets and 3x the organic growth of those who don't, while also creating deeper client relationships that transcend investment performance.

The Altruous Team

Beyond Greenwashing: How Corporate CSR Can Actually Move the Needle

Many corporate social responsibility (CSR) programs are essentially "greenwashing," offering minimal impact despite significant funding. This article, part of a series on philanthropic impact, argues for transforming CSR from mere feel-good activities to strategic social investment. It outlines five ways companies can achieve this, and emphasizes how a strategic approach to CSR not only drives genuine social change but also offers measurable business benefits like increased employee engagement, stronger brand reputation, and improved customer loyalty, especially as millennials and Gen Z increasingly expect authentic corporate responsibility.

The Altruous Team

Why Playing It Safe is the Riskiest Strategy (And What Institutional Foundations Should Do Instead)

Institutional foundations have optimized themselves for avoiding mistakes instead of making impact. Committee structures and risk-averse processes create institutions that excel at avoiding embarrassment but fail at solving problems. While nonprofits struggle with declining funding, many foundations continue operating like it's 2005—funding safe, established organizations doing incremental work rather than taking calculated risks on innovative approaches. This article outlines five strategies for transforming institutional grantmaking from risk management to impact management, including funding capacity over programs, making genuine multi-year commitments, supporting policy advocacy, and embracing high-integrity philanthropy that balances trust with accountability.

The Altruous Team

Why $250 Billion is Sitting in Accounts While Nonprofits Struggle (And How DAF Holders Can Fix It)

There is currently more than $250 billion sitting dormant in Donor Advised Funds, while the nonprofit organizations in need of that funding continue to struggle. DAF holders have a unique opportunity to massively accelerate social change in a tremendously tumultuous era. It’s time to start viewing their funds as charitable resources meant for immediate impact, not just investment growth. Here are some practical steps…

The Altruous Team