More measurement: the route to a more expansive education?

We know that in high stakes accountability systems like that used to track performance in England’s schools, what gets measured tends to get done and what doesn’t, doesn’t. Which is a problem for those of us who believe schools should provide a rich and rounded education – an education of the ‘head, heart and hand’ – that goes well beyond preparing pupils for the limited number of exams on which school performance is judged.

At the launch of The Ideal School Exhibition a couple of weeks ago, the panel was asked whether the answer to the problem of ‘narrowing’ might lie in more assessment; in finding different ways of measuring different things, so schools and students aren’t judged purely on academic attainment. If too few things are getting done because too few things are being measured, the argument goes, the obvious answer is to do more measuring. If, for example, we want schools to prioritise creative thinking, or collaborative problem solving, or the cultivation of character strengths like discipline, perseverance and grit, wouldn’t it make sense to measure the effects of schooling on these things and factor the results into the judgements we make about school, teacher and student performance?

Tempting as this might seem to those who would like to see a more holistic education incentivised by the assessment and accountability regimes, the risks of trying to measure such skills, capabilities and performance-enhancing character strengths, and of using these measures to judge schools and sort students, are significant, if not insuperable.

The first problem relates to transferability. When the skills we are interested in measuring are ‘thinking skills’ like problem solving or critical thinking, assessors may struggle to disentangle them from the student’s knowledge of the subject in which they are being applied. If a student can demonstrate the ability to think critically or solve problems in maths, but not in history, we can infer that they are good at maths and less good at history. But what are we to make of their residual critical thinking or problem solving abilities – that aspect of the skill that is claimed to be generic and transferrable?

The second problem is definitional. When we use the word ‘creativity’ for example, do we know precisely what it is we are describing? And if we can’t come up with a clear, consistent and widely accepted definition, what chance is there of being able to develop a reliable and valid way of measuring it, let alone of making hugely consequential decisions on the basis of those measurements?

The third problem is one of educability. If we are going to measure so-called non-cognitive skills like perseverance, self-control or grit, and if we are going to use these measures to judge teachers and schools, we need to be certain that these are learnable skills rather than heritable traits or, if a mix of the two, we need to understand the nature/nurture balance. I’m not adjudicating on this matter – simply noting that this remains a highly contested issue within a science still in its infancy. Which is why we cannot ignore the risk that, if we get this wrong, we may end up confusing a school’s performance for the innate characteristics of its pupils.

The fourth problem relates to the subjective nature of character assessments, which tend to rely heavily on a mix of teacher assessment and student self-reporting. As a result, there is every risk that they will suffer from all the flaws we know beset teacher assessment generally. As Durham University Professor Rob Coe and others have shown, the validity of teacher assessment is undermined by teachers’ unconscious biases; biases that tend to further disadvantage the most disadvantaged students, including those from low income and ethnic minority households and those with behaviour problems or special educational needs.

Fifth, faced with all these challenges, there is a real risk we end up measuring inputs (the number of character building activities completed) rather than outcomes (changes in students’ attitudes and capabilities), and, in so doing, encourage teachers to engage in a superficial, tick-box exercise focused on what gets done, rather than what is achieved.

Finally, any attempt to hold schools to account for their performance against a range of skill or character metrics, will, as sure as night follows day, prove Goodhart’s law that “a measure that becomes a target ceases to be a good measure” as individuals anticipate the effect of a policy and take actions that alter its outcome. The key danger here is that teachers would ‘teach-to-the-test’, which would not only invalidate the inferences we want to draw from that test, but more importantly, would hollow out the very thing we are trying to promote as character education morphs into character exam prep.

For all these reasons, even the most high profile proponents of character education, like Professor Angela Duckworth, the world’s leading expert on the performance-enhancing power of ‘grit’, tend to argue that such assessments should not be used for high stakes comparisons of students, teachers and schools.

So, on the basis that it’s too difficult or too risky to judge schools and sort students on the basis of skill and character development, but suspecting that the ‘what gets measured gets done’ adage still holds, how can we make sure that character ‘gets done’?  Some educationalists argue that the best way of ensuring character development remains on the agenda is by improving our understanding of its impact on academic attainment and other easier to measure outcomes.

As cognitive psychologist Dan Willingham points out, however, establishing a clear causal link between character and attainment isn’t easy: 

"Suppose I'm trying to improve student achievement by increasing students' resilience in the face of failure… My real interest is student outcomes like grades, attendance, dropout, completion of assignments, class participation and so on… The disadvantage is that there are surely many factors that contribute to each outcome, not just resilience. So there will be more noise in my outcome measure and consequently I'll be more likely to conclude that my intervention does nothing when in fact it's helping. 

The advantage is that I'm measuring the outcome I actually care about. Indeed, there would not be much point in crowing about my ability to improve my psychometrically sound measure of resilience if such improvement meant nothing to education." 

 Learning about Culture

All of which brings me, in a roundabout way, to the RSA’s recently launched Learning About Culture project, at the centre of which sits a set of five, large-scale randomised control trials (RCTs) which are designed to measure the impact of arts and cultural education on both academic and character outcomes.

The advantage of RCTs, and the reason many view them as the gold standard in educational research, is that they are designed specifically to mute all the background noise Willingham describes – all the other factors that may affect the outcomes we’re trying to measure. The disadvantage is that the value of arts and cultural education cannot be captured simply by reference to its impact on academic attainment, or even on the character strengths we know boost academic attainment. Why? Because as anyone who has ever had their spirits lifted by a song, a poem, a play or a painting knows, most of the value of art and cultural education is intrinsic and unmeasurable. It is to be found, in the words of Sir Kevan Collins, (the Chief Executive of the Education Endowment Foundation with whom we are running the programme), in “art for art’s sake”.  

It is a point picked up by Geoffrey Crossick and Patrycja Kaszynska, the authors of King’s College London’s Understanding the value of arts and culture report, who ask why it is that advocates of the visual and performing arts often feel obliged to justify their place in the school timetable by reference to the benefits that might ‘spill over’ from their subjects to those academic subjects the government values more – maths and English above all:

"It has become customary to consider what the arts might do for other domains of learning by way of transferable skills and knowledge – to ask, for example, whether music improves mathematical learning. As well as concerns over what might be seen as the instrumentalisation of arts learning, this raises questions about the hierarchy of disciplines and learning outcomes. It would be considered unusual to investigate the effects of mathematical learning on musical abilities and one has to ask why the rarity of that reversal should be the case."

Perhaps the best summary of the evidence of the impact of arts participation on academic attainment – and of what we should make of that evidence – comes from the authors of a 2013 OECD report who also challenge those would have us justify arts by reference to their spill over benefits:

"Ultimately, the impact of arts education on other non-arts skills… should not be the primary justification for arts education in today’s curricula. The arts have been in existence since the earliest human, are part of all cultures, and are a major domain of human experience, just like science, technology, mathematics and humanities. The arts are important in their own rights for education. Students who gain mastery in an art form may discover their life’s work or their life’s passion. But for all children, the arts allow a different way of understanding than the sciences. Because they are an arena without right and wrong answers, they free students to explore and experiment. They are also a place to introspect and find personal meaning."

So why, as an organisation that believes passionately in the intrinsic, unmeasurable value of art and culture, and in their centrality to a rich and rounded education, is the RSA conducting trials that seek to measure their instrumental value to the effort to raise the literacy levels of deprived students, critically important as that is? 

The answer comes in two parts.

First, we don’t accept the dichotomy that says you either value cultural education for its instrumental value – its positive impact on students’ academic performance and future prospects – or you value it for its intrinsic qualities. We – and more importantly the cultural organisations whose interventions we will be trialling – believe there are significant potential cognitive and non-cognitive benefits for participating students that are likely to feed through to their academic performance. We want to know more about those, and about how best to deliver them. Our interest in art and culture does not imply a lack of interest in literacy and numeracy.

Improve, not prove

But the second, more important point is this: we aren’t seeking to prove the value of cultural education; we’re seeking to improve cultural education.

There is a parallel here with assessment, and the difference between summative assessment (designed to allow outsiders to make meaningful comparisons between students and schools) and formative assessment (designed to provide useful next steps for teachers and students). Our main purpose is formative. This is a research project, the findings from which we hope will be owned, interpreted and applied by educators. We want to expand the evidence base about what works, what doesn’t, and why, and feed that evidence back into schools to improve their commissioning decisions, and into classrooms, to help teachers and arts practitioners improve their practice.

After all, we know that many of the things that Professor John Hattie (who undertook the largest ever analysis of the relative effectiveness of different interventions) suggests are most impactful in education are the very things that occur within the arts. The process of drafting and redrafting that takes place within the visual arts, for example. The feedback and rehearsal at the heart of drama. The deliberate practice and mastery required of a musician. The motor skills developed in the dance studio. The motivation and inspiration drawn from a memorable experience, be it a play, a concert or an exhibition.

Arts and cultural organisations are understandably nervous about the often reductive nature of educational research, with its effect-sizes and cost/benefit analyses and its narrow focus on academic attainment. But there is little to lose, and a good deal to gain, from moving the conversation beyond the question of whether arts participation delivers benefits to learners to one about the nature of those benefits, how they are best captured, and how arts and cultural organisations, working together with teachers, can use evidence to improve their practice.

At the RSA, we believe every child is entitled to a rich and rounded education – an education of the head, heart and hand. But extending the reach of assessment and accountability into every corner of the school system, to measure every aspect of children’s education and development, is not how we will get there.

Instead, we need to loosen the grip of the accountability system and hand the tools of evaluation and measurement over to teachers, in whose hands it should support, rather than distort, the education our children and young people receive.