The solution to the challenge of evaluating place-based systems change
Today Renaisi launches a new model for evaluating place-based systems change. Lily O’Flynn, Principal Consultant for Place-based Evaluation & Learning, describes the model and why it solves the problem of evaluating change in places for funders, commissioners, and practitioners.
Place-based systems change programmes are vital to addressing complex issues like deprivation, homelessness, education or youth safety. Yet many people have told us they struggle to conclusively prove it works.
We’ve also observed and been part of evaluations that have attempted to use traditional ‘gold standard’ methods (such as quasi experimental design or randomised control trails) to evaluate place-based systems change but the link between the programme of work and the impact on a community always feel inadequate and it raises big questions around attribution.
Read this blog for more on the challenges of evaluating place-based work.
At Renaisi we are fans of a developmental evaluation approach because it focuses so fervently on the process for change, using in-depth qualitative methods. But not everyone considers developmental evaluation to be ‘rigorous’ enough because it is less good at demonstrating large scale quantitative measures of change. This prevents some – especially public sector commissioners – from investing in place-based and systems change approaches.
And we desperately want their investment in place-based and systems change approaches.
We need a new solution. Something that sits in the middle of traditional and developmental evaluation. Something that acknowledges the very real accountability challenge for the public sector, while allowing evaluators to capture, document and respond to emerging change, and ultimately prove the impact of an initiative on a place, a system, or the population at large.
The place-based systems change journey
To understand cause and effect, you need to understand the starting point, the process and the change. Using our learning from the 70 sites we’ve worked in over the past three years, we think it looks like this:
This arrow is useful to show to prospective place-based systems change investors because it shows what we’ll measure, when, and how long each phase will take – or how and when their money will be spent.
But how does that place-based systems change journey translate into an evaluation method?
Combining experience, flexibility and rigour
Our new model combines Renaisi’s wealth of experience in developmental learning and evaluation approaches with ‘realist evaluation’ methods such as contribution analysis to assess the causal links between a programme and the outcomes – adding more rigour to the overall method, without losing the critical learning about the process.
In addition to this our new model embeds methods that are participatory and inclusive. For example, by involving local people in collective visioning, analysis and learning to ensure that research is culturally sensitive, relevant and meaningful for the people who live there. It is also proportionate, by placing a greater focus on outcomes for the place, the system and the population rather than the individuals supported by specific interventions.
We suggest using a whole range of methods to collect this data, including:
- Reviewing local data and statistics around the particular area of focus
- Root cause mapping with local people to determine the biggest issues
- Systems mapping with stakeholders to understand how effectively people in places and systems are working together
- Interviewing local residents and local stakeholders about their experiences
- Observing changes to relationships / power dynamics
- Collective learning sessions
- Value for money assessments
- Measuring public perceptions
- Identifying policy changes
We’ve often drawn on Australian consultancy Clear Horizon’s place-based measurement cube as a useful template for capturing and charting this data. With their blessing, over the past couple of years we’ve tweaked it to better meet the needs of the UK-based programmes we’re working on and to align it with our new model of evaluation.
We’re delighted to share the adapted version today.
Our model illustrates five key stages of change (on the front of the cube) aligned to the timeframe (along the top), and three cross-cutting process related data points (on the right-hand side).
We believe this model provides the solution we need to test and understand both place-based and systems change. Allowing for emergence, incorporating learning to support real-time change, bringing in the voice of lived experience and being able to rigorously demonstrate both micro and macro level change.
This blog provides a light touch summary of our direction of travel and shares some key tools. You can also watch a short video of me explaining our model here and we would be delighted to talk this through in more depth. Our contact details are at the bottom of this page.
Our experience of evaluating place-based systems change work
You can find out more about Renaisi’s considerable experience delivering and evaluating place- and systems-based change in our timeline of place-based work.
Our experience includes small and large-scale programmes such as Big Local’s Wick Award project in Hackney, the DCMS and Lottery-funded Place Based Social Action programme, Power to Change’s Empowering Places programme, and the Health Foundation’s ‘Economies for Healthier Lives’ programme.
We are excited to be working in a new partnership with Homerton Hospital to evaluate their City and Hackney Neighbourhoods programme using our new solution for evaluating place-based systems change. Read more about that here.
The two place-based evaluations we completed most recently principally use developmental methods, and incorporate elements of contribution analysis to understand large scale change, these reports can be found here:
We would love to hear your feedback or talk to you about your place-based and systems change evaluations.
If you want to know more about the new model or just have a chat, get in touch with one of the team.
Meet our team
Principal Consultant for Place-based Evaluation & Learning (on parental leave mid-November 23 – September 24)
