Community-Centered Evaluation: Shift focus, Shift power, Center the Community
Co-authored by Julie Halverson, Artemis Research and Consulting & Heather Hiscox, Pause for Change
Have you ever sat staring at your draft grant proposal, trying to specify exactly how you will promise to measure outcomes and impact?
How does it feel to have percentages, timelines, and promises placed upon your work, your team, and those you intend to serve?
Have you ever had a project that did not meet its goals?
We’ve been there too.
People give money to see change. We accept that it makes logical sense to say, “If I give you this money, I want to know that you will use it as intended, and that it will reach the outcomes you promised.” This makes sense if you have never actually worked with a community to address a social problem. The simplicity of this transactional view of support and impact is not how it works in real life, a majority of the time.
How it looks most often is that you spend 30+ hours writing a grant that you will never get, but if you are funded, it will often resemble these main scenarios.
Scenario 1: You don’t want to do the work you promised to do.
The notice arrives that you have been awarded and you first have to re-read what you put in the proposal months ago. You now have to direct new attention to your team to begin working on making those promises a reality. It has happened to us and countless others, that the proposal we wrote months ago, no longer matches the shifts that have occurred in our staffing, programs, community, and their needs, we can not get buy-in from our team, it creates tension internally (and at times externally), results in additional and seemingly misaligned work, and often, as a result, is unsuccessful. However, the funding still helps the organization so you try and make it work.
Scenario 2: You got it wrong.
This is a scenario where the grant still fits into your organizations’ needs and the needs of the community, but something about “the plan”, your methods, the program design, etc. is off and you can not reach your intended goal. It is not widely recognized, but grant writing, while an amazing craft and skill, is also a form of fiction writing. At times programs that are described perfectly, designed with the strongest of logic models, and that include finely curated timelines, just do. not. work. However, the funding still helps the organization so you try and make it work.
Scenario 3: It is a little of both #1 and #2. You want to do the work but you are seeing small results.
For example, say a foundation is funding youth violence prevention interventions. They only fund programs that have a rigorous evaluation. Your organization is not ready to deliver on a rigorous evaluation, but you have to promise this in order to stay funded and provide badly needed services in the community. Your organization is spending precious resources to try and collect data or be a conduit for data collection. However, the funding still helps the organization so you try and make it work (see a trend here?).
From the foundation’s perspective, the results don’t match their expectations. They want the programs they fund to be replicable and scalable, but the programs they are funding for 1-3 years are showing only small effects. The attempts to scale them aren’t resulting in the systemic change that the foundation or its Board want to see. The funder is faced with a decision to continue or discontinue funding.
Some funders will admit that it is extremely difficult to rigorously measure social impact, that they do not give organizations adequate funds to hire trained evaluators, that most nonprofits do not have in-house evaluation staff, and that a majority of the programs they fund do not reach their intended target and outcomes.
What these scenarios have in common is that they result in terrifying conversations, and they reinforce power imbalances and lack of transparency in the sector. It is a very stressful, frustrating, and powerless feeling to know you have to go back to a funder and have that difficult conversation that something is not working. There is a very real risk that these conversations will lead to backlash, in the form of lost future funding or a spread of negative sentiments about your organization to other funders in the community. The funder may feel as though they were misled or have now lost faith in your ability to steward funds and carry out your mission. Unfortunately, many foundations will never be aware of these scenarios because nonprofit organizations are too afraid to share and speak up. An organization may be in turmoil but all the foundation sees is a shiny report, effusive thanks, and promising updates. These scenarios reinforce the lack of transparency and authentic connection between funders and their funded “partners”.
These scenarios occur over and over, in part because we, as a sector, have impact metrics wrong; nonprofits know it, foundations know it, and the community feels it.
We have had front row seats to the conundrum of what the funder wants (systems change), what the grantee has promised (systems change with 1-2 small programs), and what the community wants (resources that meet their needs).
The mismatch between expectations and feasibility set projects up for failure. This is particularly true if community needs are not driving funded programs.
Disequilibrium between expectations of systems change and feasible measurement is a missed opportunity to identify impact and center the community.
Couple this with a nonprofit’s need to stay funded and demonstrate progress to a predefined set of metrics the funder has created. You are unlikely to see systems change. You are unlikely to measure actual impact. You are unlikely to be happy with the outcomes. Most of the time there are areas of overlap between the funder, organization, and the community, but often the overlap is not big enough. The power dynamics weight the importance of outcomes toward the funder instead of the community.
If we know that the status quo of how we measure outcomes is not working, why don’t we change it?
What we offer is that there are four major shifts that need to occur with how we think about impact metrics.
1. We need to disrupt the power divide between foundations and nonprofits.
There must be communication, trust, transparency, and authentic relationship building. This must occur so that our conversations are less terrifying, and both sides can have open and honest communication focused on learning and the uncertainty that’s embedded within all of this work that we do in social change.
2. We need to disrupt the power divide between nonprofits and the community.
Nonprofit organizations, with the best of intentions, often create tremendous waste. In many cases executive leadership teams, board members, and program development staff think they understand, listen, and serve, but often nonprofits create interventions, programs, and design services, without co-creating with and empowering the community.
This siloed approach of “we know” vs “they know” reinforces the flawed dynamic of “givers” and “receivers” and ignores the abilities, assets, and abundance that live within communities.
We must honor and recognize the labor that a community provides to make nonprofit work possible. Community members show up at events, complete surveys, attend focus groups, prepare and provide meeting spaces, spread the word to friends and neighbors, etc. This work is often unrecognised, undervalued, and uncompensated. This work is especially demanding when many communities have been asked to participate in programs over and over again, many of which are short-term and never lead to a direct result in their daily lives.
3. We need to entirely reframe the way that we think about metrics and the results that we can accomplish with the type of funding that is given.
Why? Because the premise is flawed. A funder setting priorities, goals, and metrics that don’t track to reality, forces organizations to fit what could work into an often unworkable structure. For example, funding a project for one year, with $35,000, with a mission of “ending poverty” is exactly as ludicrous as it reads.
Pile onto this the work quarterly or bi-annual reports requiring that you demonstrate change or progress toward change. When you have to show impact at the six month midpoint, not only is this an unrealistic expectation that highlights even further how goals aren’t calibrated to reality or community, but organizations have to invest resources to try and evaluate something, even when they know change in that tight timeframe will be small.
We also know that there are significant dynamics created by funders that want collaboration. They know full well that even they as funders in one community, have a hard time collaborating, and that it’s very difficult for organizations to do so in the competitive reality of a huge amount of nonprofits vying for a scarce amount of foundation dollars.
We cannot make large, sustainable, and systemic changes with small amounts of funding that are just given for 1-3 years. Even large amounts of funding require that the funder understand what will work in a community. In particular, what the community cares about and how to measure it respectfully.
4. Who makes the goals makes a difference.
Often a foundation has goals they would like to see, a nonprofit organization has goals they would like to see, and these two entities come to an agreement of what should work and create impact. Once this agreement is formed, then the program, service, etc. is released into the community.
The community is expected to partner, show up, invest, attend, share, participate, and ultimately change their behavior in some way. They are seen as recipients for the program and in order for the program to be “successful”, the community must complete or comply with some expectation.
What is often true is that the community had zero input in creating the program goals for which they are held accountable. The most egregious part of this dynamic is that the community is often blamed for not participating in and receiving the benefits being “so kindly provided.”
What if the community's desired outcomes are completely different than what funders and nonprofits envision? What if the community's desired outcomes far exceed what is traditionally imagined? For example, what if, while the sector offers programs and services, the community wants policy change and structural redesign of systemic and oppressive practices?
Figure 1. Traditional Approach
How to Work Differently - Center the Community
Continuing with the youth violence prevention example, consider the typical funding pattern. A youth focused prevention program is funded. Funds are dedicated to staff supporting the program, evaluation is required, technology is not supported, infrastructure is not supported, unrestricted funds are not allowed, and systemic change is expected. The problem this program intends to and is funded to solve is youth violence in a community. The intervention is for high school aged kids. After one year, the evaluation is burdensome and the effect is small. The funder wants to see systemic change, the nonprofit offers youth programs, and the community has a set of needs. Figure 1 represents the power dynamic, overlap of interests, and likelihood of seeing impact. The spheres have little overlap. The funder and its expectations have primacy. The nonprofit has a medium sphere of influence and little impact measurement capability. The community has little influence and its needs are not met or are superficially met.
In order to understand the impact you are making (or missing), nonprofits and foundations need to get specific about the problem to solve, the groups directly and indirectly affected, and what is feasible in a given timeframe. Aspiring to systemic change is necessary. Getting there needs 1) time, 2) money, 3) calibration of community and funding goals, and 4) recognition of who the experts are, the community.
1. Engage the Community First and Ongoing.
The programs that work the best, yield the most informative data, and have the most buy-in, are those that center the specific community for whom the intervention or program is meant to help.
What if for the youth violence intervention example, parents and community members were asked first about their greatest needs. Let’s say you learn the following:
Parents value having a safe place for their kids to go after school.
Kids need a safe place to play, to be with friends, to have some dinner or snacks.
Parents need a little bit of extra financial help every month.
Kids are hungry and feel better when they reliably have food in their house.
With this information, you could better frame the problem, correctly identify those directly and indirectly affected, and tailor outcomes appropriately.
By engaging the community in co-creation, you are alsobetter positioned to have community participation and input, and you will better understand the barriers to and facilitators for change in the funding timeframe. Even programs that drift from the changes or outcomes the funder wants to see, if the community has an opportunity to weigh in, you get awesome data and results.
Community engagement does not end with the project kickoff. Meaningful communication with the community is ongoing and always critical, before, during, and after the grant. The work of community participation in funded programs is often overlooked. They show up, give feedback, and frequently see nothing change. Yet, they continue to show up. Regularly give updates on the status and progress of the funded work. Provide updates in ways that are easy for the community to receive and communicate back to the nonprofit, evaluator, and funder. (Read: not in a conference room far from where they live or only in a presentation given to donors.)
2. Set realistic goals and set the table for systemic change.
You can design measures of short- and long-term impact based on what the community cares about and what the nonprofit provides. Only looking at a program and expecting systemic change misses the other areas of impact.
You can have rigor and get results. You can start by understanding the key impacts the community wants to see, crafting a process of getting there, and identifying measures that are low-burden and yield results. The community should also be a driver of the evaluation design, and not all designs are appropriate in all settings.
For example, randomized controlled trials (RCTs) are known to be the gold standard in testing effectiveness of interventions. Foundations often want to fund RCTs. They want to identify interventions that “work” and scale them up in other places. In this example, a youth violence prevention program would be tested for effectiveness in reducing or improving a number of individual-level outcomes that theoretically tie to systemic change. In a resource poor community setting, RCTs aren’t always feasible or ethical. The design forces organizations or individuals to compete for badly needed resources. The burden of randomization and data collection is high. This burden is often placed on nonprofits to collect data and ensure fidelity to the evaluation design. If few people participate, then you have unreliable data to demonstrate “impact”. If the community is centered they may tell you who is likely to participate in a study design and the best way to collect data. Moreover, it misses the key impacts the community values most. Evaluation designs should be linked to the questions and outcomes valued by the community.
3. Be flexible and expand the definition of impact.
As we discussed in our initial scenarios of grant execution, we need to releaseconstraints placed upon nonprofits by the funder in directing how the money is used. Shift power to the nonprofit and allow them to decide how to best deploy resources in support of the community.
Assuming the funds are used to support community defined needs and goals, restricting funding and placing undue evaluation burdens impedes progress.
But, back to our example, how does all of this decrease youth violence? And what about rigorous evaluation? The values outlined in our example are tied to risk factors for violence and other outcomes including mental health, food insecurity, and school performance. Shifting the power dynamic to the community as the driver of funding and evaluation goals affords an expanded understanding of the impact of funding.
An alternative to the tried model in Figure 1, is to center the community and its needs (Figure 2). The spheres of influence show the community having primacy.
Figure 2. Community-centered approach
The nonprofit and funder are in supportive roles that help the community achieve impact. The nonprofit has flexibility and funds can support infrastructure, staff, outreach, and services. Nonprofits may offer the same lines of service or tailor them appropriately. The funder is in a position of support and partnership. Rather than the nonprofit and community proving their worth to a disconnected funder, we have a funder supporting critical community-identified needs to reduce youth violence and enhance the resources available to a community.
This sets the table for systemic change in violence prevention to happen (Figure 3). With the goals and metrics tied to community values and needs, the funder is better positioned to show impact and move the needle on a particular outcome. Figure 3 shows interconnected spheres of influence. It shows the individual, interpersonal, community, and societal levels where the funds are deployed and the potential to use valid data to advocate for change. The valid data hinges on goals, metrics, and evaluation design in which the community invests and values.
Figure 3. Setting the Table for Systemic Change
Communities are invested in their well-being. They want to see change happen for their families and neighbors. Instead of assuming what the outcomes and objectives are and shooting monied arrows into the measurement abyss, get some guidance from those who are impacted directly.
Value their input. Help them craft objectives that can be achieved in a given timeframe. If those goals or objectives don’t look systemic enough for you, you may need to take a step back and examine your place in the power dynamic. You may need to change your mindset and support the change, small or large, the community is demanding.
Community-identified needs and outcomes should be funded. Rather than asking organizations to fit their projects (round peg) into a prescribed set of outcomes (square hole), foundations have an opportunity to ask what is feasible and help craft metrics around what is already happening. This way, you can demonstrate impact.
In community-centered evaluation the funder can align their funding to what is actually needed, the community can experience impact, and nonprofit organizations can better serve their missions and their communities.
About the Authors
Julie Halverson, Founder of Artemis Research and Consulting, helps foundations, nonprofits, and public sector organizations demonstrate the effectiveness of social impact programs by making evaluation easy, feasible, and attuned to the community.
Heather Hiscox, Founder and CEO of Pause for Change, helps nonprofit, local government, philanthropic organizations address challenges in less time, for fewer resources, while achieving greater impact.