How evidence-informed is cultural learning?

In August and September this year, we surveyed just over 100 cultural organisations working in schools to share their views on evaluating cultural learning and this is what they told us…

Our survey results show that cultural learning organisations see that evaluating their work benefits their participants and this motivates them to evaluate more and more. However, few cultural learning providers are using the most robust evaluation methods, and it is hard to find evidence from other sources to support their work. Therefore, significant support is needed to collect and use evidence effectively.

Below is a summary of our top eight findings.

Dowload a full report on the Learning about Culture survey results (PDF, 1.5MB) 

1. Cultural learning organisations see that evaluating their work benefits their participants

62% said that evaluating their work had led to an increased ability to show how they make a difference and an improved experience for participants.

 2. Evaluation is happening more and more often

72% of organisations said that they now do more monitoring and evaluation than five years ago. 37% of organisations say that they now ‘always’ measure the changes that happen as a result of their work, with a further 34% saying that they ‘often’ do so.

Enlarge this graphic

3. Cultural learning charities are particularly motivated by self-improvement

88% of the 46 charities who responded to the survey stated that their desire to improve projects was their main motivation for doing more evaluation. This contrasts starkly with the results of the NPC’s 2012 survey of charities on which this survey was based. That survey found that over half of charities were doing more evaluation because of a change in funder requirements, compared with just 5% who wanted to improve the quality of their service and 4% who wanted to know the difference their services were making.

4. Most cultural learning providers do not have a theory of change

Only 40% of survey respondents have developed a model to explain how the activities they deliver might lead to change. Developing a theory of change can help organisations to have a more strategic approach to evaluation, and respondents who had developed a theory of change recognised broader benefits from the process including ‘the opportunity to reflect on our work and how it makes a difference’ (59% of respondents).

5. Few cultural learning providers are using the most robust evaluation methods

As previous research has suggested, few cultural organisations use the most robust evaluation methods. Only five of the organisations that we surveyed had used studies with comparison groups (comparing the results of children who participated in the project with a group that did not) and none had engaged in randomised control trials (as above, except that children/schools are randomly assigned into the group). Many report using anecdotal evidence and even among those attempting evaluation approaches beyond self-reporting surveys e.g. tests of learning, only 3% used standardised tests, thus negatively impacting the external validity of results and the ability to compare their results with those of other organisations.

Enlarge this graphic

6. The smallest organisations use some of the most robust methods

Three of the five organisations who had used comparison groups had an annual turnover of less than £250,000, demonstrating that organisational size is not a barrier to robust evaluation. In fact, organisations with less overall resource were more likely to have a strategic approach to evaluation. 48% of organisations with less than £500,000 turnover have developed a theory of change, compared with 34% of those bringing in more than £500,000 per annum.

7. Practitioners find it hard to get hold of evidence to support their work

Those organisations that had been able to identify relevant research from academic institutions, strategic bodies and organisations delivering similar activities found that this ‘helped them design new projects’ and ‘compare their work to other projects’. However, less than a third of organisations that we surveyed find it very easy or quite easy to find evidence from other organisations or from academic sources that is relevant to their project. 36% find it ‘quite difficult’ or ‘very difficult’ and 14% have never even tried to do so.

8. More support is needed to collect and use evidence effectively

Over 50% of respondents said that they would benefit from training and guidance about how to evaluate (awareness and understanding of techniques), more appropriate tools for evaluating cultural learning and increased funding for evaluation. In terms of using evidence from elsewhere to support their work, organisations most commonly chose ‘opportunities to develop practice and share findings with other similar organisations’ as the thing that would most support them to do this effectively (35% of organisations).

Over the next two years, the Learning about Culture programme will be looking to respond to these needs through:

• Training and peer learning networks in partnership with the Arts Council Bridge Organisations
• Developing tools to share best practice in evaluation and enable more cultural learning providers to evaluate their work effectively
• Hosting conversations with funders about how they can support evaluation in the sector, in partnership with Paul Hamlyn Foundation and Bank of America Merrill Lynch

To learn more about who responded to our survey and what they told us, read our survey report:

Download a full report on the Learning about Culture survey results (PDF, 1.5MB) 

Download the Learning about Culture Prospectus (PDF, 700KB)

To find out more about how you can get involved in the Learning about Culture programme, please contact culturallearning@rsa.org.uk