Impact measurement must work better for services and communities

In recent years, measuring and demonstrating impact has risen up the agendas of organisations that aim to do social good. Alongside this, interest in growing the evidence base for effective practice has spread. Scientific evaluative approaches and research techniques are increasingly being applied in not-for-profit service settings.

Competition for scarce funding has been a key driver, with funders and commissioners placing a greater emphasis on demonstrating the impact and social value of investments. The attainment of standards of evidence is now sought by many organisations to demonstrate the robustness of their delivery.

One effect of this shift has been the growth of an ‘impact industry’. Its hallmark is the proliferation of providers who offer a set of purchasable knowledge and skills in impact measurement. These providers, of which Renaisi is one, offer not-for-profit organisations consultancy support, including the development of impact measurement frameworks and evaluations, to demonstrate, understand and improve their impact. Often, this includes supporting organisations to attain standards of evidence.

At best, the impact industry is able to support the development and where appropriate, growth of effective services – those that create genuine value for the communities they serve. However, there is growing concern among many of us involved in the impact sector that it is not serving the interests of not-for-profit service providers, their staff and beneficiaries as well as it should – and indeed, may even be detrimental in some cases.

In November 2016, Renaisi and the Centre for Youth Impact brought together funders, not-for-profit delivery organisations, and impact service providers to discuss this concern. A summary of the findings is set out below.

The full report of the discussion can be downloaded here.

Our aim was to create a space for honest dialogue and to build a foundation for constructive, inclusive action. Therefore, the views described below are not necessarily shared by Renaisi or the Centre for Youth Impact, and instead represent the reflections shared. However, we believe that the debate must continue – and we welcome all those who wish to contribute to it. If you would like to do so, get in touch.

What are the challenges?

Marketing, accountability and improvement often get conflated

We noted that impact measurement practice can be used to achieve three distinct purposes: accountability to services users and other stakeholders; for marketing and PR; and for learning and improvement. These purposes are often conflated, and as a consequence, approaches to impact measurement that depend on purpose are rarely differentiated. This risks watering down the effectiveness of attempts to understand and improve impact, and undermines the value of impact measurement.

Marketing: the enemy of improvement?

We agreed that organisational imperatives to maintain positive reputations and increase investment disincentivise honest discussion about impact. This hinders the improvement of services and the growth of effective practice.

The language of research

We noted that certain language about impact is perpetuated as ‘expert’ and specialised. This can alienate groups – such as frontline teams – on whose cooperation and participation effective impact measurement approaches often depend. It also serves to maintain an unhelpful separation between practice and impact measurement.

Is ‘measurement’ a useful term?

We differed in our views as to the usefulness of the term ‘measurement’ to describe the activity of understanding an organisation’s impact. Some felt that it contributes to the removal of this activity from the reality of service delivery. However, others advocated for it as a widely understood term describing a useful and even normal activity.

The inappropriate proliferation of impact measurement

We noted that negative perceptions of impact measurement may have arisen in response to its application in inappropriate settings. However, the response should not be to downplay its significance, but rather to talk more openly about appropriateness.

Funders’ requirements can be unhelpful distractions

Funders sometimes impose data collection requirements onto organisations they fund in order to support their own missions or wider policy trends. We agreed that this places an unnecessary burden on organisations, and impedes understanding of their impact.

Separating organisations from service delivery

We identified a problem in the separation of programmes or interventions that have been ‘proven’ to be effective through impact measurement approaches, from the reality of implementation. It creates the false assumption that effective practice can simply be replicated, and discourages conversations about fidelity, quality and impact management.

Ways forward

Widen the conversation

We acknowledged that these challenges require continued collective engagement and discussion if they are to be addressed. This includes involving more funders and commissioners, such as local authorities.

Make impact measurement achievable and common-sense

We want to advance ways in which impact measurement can be embedded more effectively into organisations’ day-to-day delivery.

Explore and clarify when impact measurement is appropriate

We felt that more work is needed to clarify when and how impact measurement creates genuine value for organisations and their beneficiaries.

Funders should stop asking for additional outcomes measures

We agreed that funders should not ask organisations that they fund to report on outcomes that are well beyond the scope or influence of organisations’ programmes or provision.

Funders should encourage honest dialogue with organisations

We felt that funders have a key role to play in moving towards a culture in which honesty is encouraged, and should be more ready and able to challenge applicants’ impact claims, and encourage more appropriate approaches to impact measurement.

John Hitchin is Renaisi’s Chief Executive and Bethia McNeil is Director of the Centre for Youth Impact. To get involved in the discussion contact John or Bethia.