At the end of November, I attended a SCVO one-day training event on demonstrating impact and value, which was led by Lindsay Linton of Impact Generation. It was a very informative and useful day during which I had to time to reflect upon the evaluation we do of the community health projects. I have written up my notes from the day as I thought you might find them interesting and may make you rethink some of your own evaluation. If you have any comments or questions then please do not hesitate to get in touch with me.
Impact = additionality=’bump’ factor : what is the additionality your project has achieved/improved ?
Project funding should be all about accelerating the ‘bump’ or in other words , using the funding to increase from your current levels of activity to your new levels of activity.
It is really important to understand the funder’s values or charter and demonstrate your understanding in your funding application to them. In your evidence always start with your ‘user profiles’ , who you are targeting, using facts and figures and case studies. Pen portraits can be helpful as you can describe how an individual/service/user/project will change/develop over time.
Feedback MUST be routinely gathered (at least on a monthly basis). You need to do this to be able to track the impact. 12% of your budget should be spent on resourcing the evaluation .
What is evidence of impact?
Quantitative information: seeks to measure, normally large numbers of respondents, how many , how often, consistent structure, predefined questionnaire, methods ; sample surveys, databases.
Qualitative information: seeks to understand- normally fairly small numbers of respondents, how, why, flexible, response orientated discussion guide, methods: focus groups, deliberative, observation.
Four key stages of Social Impact( often described as a logic model)
Inputs Outputs Outcomes Impact
Inputs = resources( staff, funding, materials, time ) that were invested in the activity
Outputs = tangible indicators that demonstrate objectives have been progressed e.g. no of people volunteering and no of people that self report
Outcomes= changes to people that have occurred as a result of the intervention
Impact= outcomes less the amount that would have occurred anyway or without the intervention. This is the additionality- the difference your project has made.
Sometimes you need to make a judgement call as to where to categorise things in inputs/outputs/impact etc.. There are no right answers.
Remember that ‘outputs’ are NOT ‘the change’ from a social impact perspective as outputs are mainly quantitative and not qualitative. It’s the qualitative measures that give the social change and the additonality. It’s the change that funders want to be able to evidence. If you are applying for repeat funding, you still need to demonstrate that you are increasing your levels of activity /levels of change , year on year. Some funders now specifically look for evidence around theory of change. Do not be afraid to borrow benchmarking data/evidence from elsewhere/other sources and then reference it. Some useful sources are: Scottish Health Survey, Rowntree Foundation.
Ideal Structure of an evaluation report :
Executive Summary – summary of key points of all the report
Introduction : your organisation and partners, policy context that the project is being delivered within, summary aims and activities of the project at the outset, what measurement methods were used to provide evidence to funders, how the report is structured
Findings : outputs delivered were and how these were measured, outcomes delivered and how these were measured, impact delivered and how these were measured
Conclusions: what was learnt during the project, how could these be amplified ( if good) or minimised ( if bad) and the value for money ( if you have calculated this)
Recommendations: for future project delivery
Remember : programmes need clear and measurable objectives; some objectives are not achievable – be realistic. Interventions should be based on social impact design from the outset; model of change is what to aspire to i.e. a logic model; before you start think about what sort of analysis you will be doing and can afford. Understand your target group and audience before you commit to any research. Don’t lose sight of the target group. If you are targeting hard to reach groups make sure the research concentrates on them and recruits them into your sample for your research- as assumptions that you make weaken your credibility with funders.
Outcomes are the changes to people that have occurred as a result of your service/project. What you have promised to do is what you should be reporting on. When equating a value to outcomes/impact , you can use a credible source- Google is your friend. In 2008, NICE said that £43 per person per hour of wellbeing contributed to the economy. Remember to always declare your assumptions, sources, data /research. Don’t over claim or overstate the outcomes/impacts. Be clear about your limitations.
Choosing your process of evidence gathering is key , needs to be congregant, robust and routine, from the beginning to the end of the project , and sustainable. Sometimes it is good to think about a new stand out idea for evaluation – which might just attract a funder’s attention. Involve your target group early in the process.
Make sure you are not widening the gap in health inequalities by the intervention/service you are seeking to provide.
I hope you found this helpful. If you have any questions or thoughts, please get in touch.
Stephanie-Anne (stephanie-anne@lchif.org.uk)
