When Reach for Change had finished the Annual Impact Report for 2012, we had it reviewed by the Department of Children and Youth in Society at the University of Malmö, Sweden. The head of the department and a PhD professor where impressed by the extensive work. In particular, they acknowledged our ambition to include children’s own voices in our operations. Furthermore, they noted that the report generates one crucial question about the 600,000 children we reported as supported in 2012: Have their lives improved?
This was not the first time we have been asked this question. The question “Are you actually able to affect children?” is mentioned in almost all our stakeholder interaction. So, we decided to put some extra work into answering it. During 2013 Reach for Change have intensified the efforts to measure impact.
Reach for Change have a favorable point of departure since we are a young, and relatively small, organization that right from the start have had a strong ambition to measure impact. Our key strengths regarding impact measurement are:
- Developed monitoring of investments. As a social investor Reach for Change have developed a reporting system that allows us to gather large amount of information from the people we expect to deliver impact: the Change Leaders.
- Developed stakeholder perspective. Reach for Change is a corporate initiative with strong ties to the non-profit sector. This gives us the ability to apply a strong stakeholder perspective on everything we do and provides us with the tools we need to maintain a dialogue with our number one stakeholder: the children.
- Commitment to the cause. The vision of Reach for Change is to make a difference. And we are committed to finding out if we succeed in this endeavor or not.
We have identified a number of challenges to impact measurement. All NGO’s and social initiatives are struggling to find ways for evaluating the impact of social innovation on society. There are a number of challenges that characterize this sector, such as the lack of tradition to measure and report results, and difficulties in quantifying the information. Because social change always takes time, it is often necessary to wait several years before trying to make impact measurements. As an additional obstacle, it is also difficult for us to isolate our impact on society, to adjust estimates in relation to developments that likely would have happened anyway, and in relation to events that should be attributed to the work of others on the same field.
Apart from these field wide challenges, we have identified a few short time challenges specific for Reach for Change that we have addressed during 2013 and will continue to work with during 2014. They are:
- Completeness. We need to gather and store data in a way that makes it accessible to our stakeholders in order to be more transparent. We have also made information about our methods more accessible.
- Comparability. The point of measuring impact is to track change over time. In order to do that we need to repeat the same kind of data collection and analyses over time. Since Reach for Change was founded only in 2010, we have yet to develop (and test for durability) scalable methods to measure impact in a systematic way.
- Balance. To make a balanced assessment of our performance we need to consider the negative aspects as well as positive. We need to strive for a neutral viewpoint and find evidence rather than arguments to strengthen our reporting and avoid bias. By a vivid community engagement we can make sure that the impact is evaluated through the eyes of external stakeholders. Since our primary stakeholders are children we also need to consider the interpretative privilege that adults have.
During 2014 we will take the next step. This means that we will move on from measuring impact to create a management system based on impact data. By analyzing and make key decisions from collected data we can create more value for our investors and make even more difference for children.
Reach for Change have a method based on the idea that each Change Leader needs individual impact indicators. This allows us to understand the impact of each Change Leader and analyze the aggregated results.
We use the Theory of Change methodology, a widely adopted approach to social change, that allows operations to define the building blocks of social impact and to create a visual map of the steps required to fulfill a vision of social improvement, a so called Pathway of Change. This is a method to break down the huge commitment made by the Change Leaders, who promise to improve the lives of children, that allows us monitor their progress.
Change Leaders’ individual Pathway of Change
Each Incubator-year starts with a Take-Off Session, a workshop where Reach for Change staff help the Change Leaders to set goals and objectives, develop indicators and discuss methods of documentation. The result of this session is what we call the Pathway of Change. The Pathway of Change is the base in the reporting. Each quarter Change Leaders report on their own outputs and indicators.
- On their own customized Effect Indicators
- Number of children they have supported – Key Outputs
- Alongside this we also use Most Significant Change Technic – a qualitative method to measure social change using stories from the field
During the Take off Session the Change Leader identifies effect indicators. Effect indicators are made to capture the impact Change Leaders have on children’s situation. For the Change Leader this means that she or he have to reflect on what impact on society she or he want to make. There are different kind of effect indicators:
- Focus on change: the increase, maintenance, or decrease in behavior, skill, knowledge or attitude; e.g. increase immunization among young children
- Focus on targets: stating specific levels of achievement; e.g. immunize 80% of 2 year old children in the community according to recommended public health schedules.
- Focus on benchmarks: including comparative targets, generally related to other time periods or organizations; e.g. increase the current 70% immunization rate for children aged 0-24 months to 90% by the year 2015.
It takes some effort to create a good indicator but there are some rules that help creating a good indicator. A good indicator is:
- SMART: specific, measurable, achievable, relevant and time-bound
- Clear: clearly defined so that they can be reliably measured, and ideally, comparable with those used by others so that performance can be better benchmarked and understood in a broader context
- Factual: should reflect fact rather than subjective impression
- Obtainable data: based on data that is ready available, or can be collected with a reasonable effort
- Quantitative indicators are numerical.
- Qualitative indicators are based on individual perceptions, for example responses to interview questions.
Children Supported – Key outputs
During the Take off Session Change Leaders also set targets on their Key output: number of children supported. Reach for Change also aggregate the number of children supported and report it as an own KPI. In 2013 for the first time, we supported more than a million children.
By children supported we mean children who had qualified support through the programs and operations of our Change Leaders. Children supported are children, who, as a direct or indirect consequence of the activities of the Change Leaders have had increased opportunities to have their rights fulfilled in accordance with the UN Convention of the Rights of the Child. We define children as being below the age of 18. The number of children refers to this calendar year only. The number of children supported does not include children being reached by marketing or other visibility activities (interviews, social media etc).
The children supported could be supported face-to-face, through parents, through professionals working with children, through a web based tool or through media. Regarding web tools and media we are careful only to include such support if the results could be evidenced. We never include marketing or information in our reports.
Examples of Pathways of Change and reported results
All Change Leaders have several customized impact indicators that allows us to understand their unique impact. The Take-off session allows us a deep understanding of each Change Leaders potential impact and indicators to show progress. We follow up each Change Leader’s individual progress in relationship to impact each quarter.
From this rich picture of social change, provided by all Change Leaders through their reporting, we can aggregate results and understand our global impact on children’s situation.
To further illustrate, and give concrete examples, of this multi layered understanding of our impact, we share two examples below. These show some of the activities, key outputs and indicators that two of the Change leaders use to reach their desired outcome. Please remember that these are just a few examples from a Pathway of Change, and that each of our almost 50 Change Leaders have a full individual map. All of these together, make up the fabric of impact of Reach for Change.
James Kofi Annan founded Challenging Heights in Ghana to rescue, rehabilitate and reintegrate children from slavery back into the community. When he made the Pathway of Change he indentified several Key outputs and Effect indicators. One of the Key Outputs where ”Children rescued” and one of his Effect indicators was ”Number children successfully reintegrated in their communities”. In the end of 2013 James had rescued 58 children and reintegrated 96 children. The children who were reintegrated had been rescued a few years back and James continued to work with them in a reintegrating program.
Click to zoom.
Jean Tazinda produces low-cost solar-power lights and making them available in the rural and poorest areas of Rwanda. Access to electricity is very limited in these parts of the world and most children, especially in rural Rwanda, only have a kerosene lamp to provide them with the light they need to complete their homework. The children have to sit close to the lamp and inhale the fumes which may lead to severe health problems. One of Jeans Key outputs was the number of children benefiting from solar lights by being able to read and do homework. One of his Effect indicators was the Percentage pass rate of Primary School National Exams per district. In 2013 Jean reports that 5,671 children benefitted from the lamps and that 90 % passed the national exam.
Click to zoom.
Most Significant Change Technique– the Story Approach
Along with the quarterly reporting on impact indicators and number of children supported we collect stories from children within the Change Leaders target groups. The purpose of collecting stories is to perform case studies of changes in the life quality of an individual child. It gives rich information and a vivid picture of the impact of the everydays lives of individual children. These life stories describes what actually changed through the program.
To write a story we ask the entrepreneur for a testimony of their own results. We ask them: Looking back over the last quarter, what do you think was the most significant change in the quality of a child’s life in your program/by your operation? We want the entrepreneur to explain the significance of the story from their point of view and describe the difference made or lessons learned.
Each entrepreneur submits one story per quarter. Each region submit their 2-3 best stories to Global who choose 1-2 stories that demonstrates the concept of Reach for Change. This process is an evidence based method to make valuable insights about what kind of change Reach for Change contribute to. The selection process also gives us a good picture of what kind of change is valued internally.
To validate the stories and verify the impact and minimize the risk of misunderstandings and/or exaggerations different methods can be used. One method is field visits (different levels of the Reach for Change organization can do the visits). Another one is investigation through interviews for deeper knowledge and a clearer picture; is there enough information about when/where/who/how to be able to determine the truth value of a story.