Programme evaluation, often misrepresented as ‘impact assessment’, has become an essential component of many development initiatives undertaken by companies, foundations, social enterprises and NGOs. The increasing focus on measuring impact of various programmes on beneficiaries is welcome, but with programmes lasting between a month and several years, the definition of impact by stakeholders can be varied and sometimes misleading. Can impact be measured within a month of implementing the programme? What are we actually measuring, if not impact? Are outcomes less important than impact? And what exactly is impact?
In the rush to evaluate programmes, it is important to take a step back and attempt to answer some of these questions in order that companies and their implementation partners develop a common understanding of what they are actually measuring and why it matters.
Understanding impact using the theory of change model The theory of change is a helpful model used to measure change as it clearly differentiates between the intermediate and long-term goals of programmes. It helps implementation partners and companies identify key outcomes and the expected impact of those outcomes. To illustrate, a Water, Sanitation and Hygiene (WASH) programme is designed with the assumption that the construction of toilets will lead to reduced open defecation. If this does not occur, the theory of change approach would highlight gaps in the programme and include aspects such as behaviour change and community practices in the design of the next phase or for future programmes.
Using the theory of change model, this article will attempt to clearly define different evaluation metrics and clarify the difference between outcomes and impact.
Outcomes and impact: What’s the difference?
Outcomes and impact are induced by behaviour change of the targeted beneficiaries. Outcome measurement helps chart the effectiveness with which programme targets were met. Impact measurement, on the other hand, is a way to attribute changes in the lives of the target group to a programme.
Let’s assume that a company has constructed toilets for a school and has to report the results of the programme. One major differentiating factor between the outcome and impact of a programme is the length of time needed to measure results. If the evaluation period occurs six months after the construction, the company will be able to report on the “outcome”, i.e., whether the toilets were used by children in the school. If the period of evaluation is a year or more, impact can be measured, i.e, the reduction of open defecation within the school.
An outcome can be measured relatively quickly – usually within a few months (3-6 months) of programme implementation.
On the other hand, measuring the impact of a programme needs more time. It can take anywhere from a year or more to assess the impact a programme has had on the target beneficiaries. Measuring the reduced incidence of open defecation can take several years and is dependent on many factors apart from the construction of toilets, such as the intensity of the behaviour change curriculum, the willingness of the community to absorb and practice lessons taught to them, and others.
As both outcomes and impact indicate behaviour-oriented changes and help in measuring the effectiveness of a programme, impact is usually confused with and wrongly attributed to outcomes.
Measuring impact is further complicated by the possible existence of external factors, which one must take into account during programme evaluation. For instance, an increase in the number of people who wash their hands with soap can also be attributed to the income levels of families, an individual’s willingness to use soap and existing cultural beliefs - factors which are primarily beyond the control of a basic WASH program.
Is measuring outcomes enough?
A key reason for assessing impact is to understand the effectiveness of programmes and the resultant change in the lives of beneficiaries. Programmes of a shorter duration may not have resulted in any impact in the lives of the intended beneficiaries in which case, outcome measurement is an equally valid marker of the success or failure of a programme. It can also act as a stepping stone for future ‘impact’ assessments.
Since impact occurs over a much longer period of time, in some cases, evaluators treat outcomes as the end goal of the programme – further blurring the line between impact and outcomes.
However, treating outcomes as the end goal is useful for companies reporting their CSR programmes as it measures the change in behaviour that is directly related to the activities of a programme, happens over a much shorter period of time, and can be seen as a marker of initial success.
That said, measuring impact is crucial, as it helps in attribution. At the end of the day, there is no greater validation of the success of a programme, than the impact it has created and managed to sustain.
With companies beginning to play an active role in the development space - working with different stakeholders to address pressing social issues in India - understanding the difference between these evaluation metrics becomes critical as it helps companies, foundations, NGOs, social enterprises and other relevant stakeholders in the development space align their thinking, define common goals and develop a consensus on how to assess impact (or outcomes, as the case may be).
- By Rasikha Venkat, Assistant Manager - Research
The thoughts and opinions shared here are of the author.
Check out our end of season subscription discounts with a Moneycontrol pro subscription absolutely free. Use code EOSO2021. Click here for details.