Is efficacy efficacious?

Earlier this year I wrote a post complimenting the global education company Pearson on its efficacy framework. The corporation had committed to the principle that it should only develop or sell products or services which have ‘a measurable impact on improving people’s lives through learning’.

I was impressed by the seriousness of the initiative (being led by my former Downing Street colleague Sir Michael Barber), the way it takes social responsibility to the heart of Pearson’s business model, and also the openness with which Pearson executives talk about the challenges involved in developing and implementing the framework across a huge, complex global organisation working in ever more competitive markets.

The post elicited a good reaction and we subsequently developed a short seminar series exploring the efficacy framework, the first of which – focussing on youth unemployment - was yesterday morning.

Although Pearson's work was the jumping off point, our discussions were relevant to the wider debate about impact and evaluation. A number of interesting points were raised, but for me certain themes stood out.

Reviewing the history of social democracy the historian Peter Clarke made the distinction between ‘mechanical’ (technocratic, centralising, rule-based) and ‘moral’ (inspirational, decentralised, value-based) models of change. At first sight, efficacy seems to be a weapon in the mechanic’s armoury. Indeed the efficacy framework looks like a typical technocratic toolkit enabling senior managers to model and regulate decisions further down the hierarchy, albeit with the best of intentions.

Interestingly though, Pearson’s Kate Edwards, who has been using the framework in seeking to reform under-performing vocational colleges in South Africa, emphasised the role of efficacy in changing hearts and minds. A willingness to focus on impact, to look for evidence of success or failure and ultimately to be accountable for results is the starting point for a meaningful commitment to change, she argued.

Echoing this, another theme connects policy and organisation to notions of personal efficacy. Julian Alexander, of the personal development consultancy Emergence, described the successful use of positive psychology to enable young unemployed people to boost their sense of agency and hope. Attributes like persistence, responsibility and an ability to combine ambition with realism are important to the success of an organisation’s students or clients. A well applied efficacy framework will see those attributes powerfully mirrored in the working methods and culture of the organisation.

These ideas extend the ambition of efficacy beyond the rather technocratic framework of assessment initially developed by Pearson. But the seminar also identified major barriers to making efficacy the ‘true north’ of policy or organisational practice. Greatest among these is complexity. Competing goals, competing organisational responsibilities (intra and inter) and competing incentives all confound and distract from a focus on the outcomes that most matter (assuming we can agree what those are).

The challenges can be read vertically or horizontally. The champions of efficacy in Pearson have to convince their marketing department not to sell unproven products, while the salesmen in their turn have to persuade the customers who undertake procurement (whose priority may be to spend their budget in a given financial year) not to buy products that may not work while they then have to persuade teachers (who may have only a limited understanding of robust educational research) to change their working practices to make better use of better products.

For those trying to pursue efficacy in youth employment practice in England, the number of agencies, initiatives and levels from which, often conflicting, policy emerges makes it all but impossible to find the clear space needed robustly to test and refine ideas.

The constraints of complexity, hierarchy and regulation - openly recognised by Pearson - put me in mind of a conversation we held here at the RSA last year. The leader of a innovative and growing social enterprise working with at risk young people shared the platform with the heads respectively of a large children's charity and a large private sector children's services provider.

The social entrepreneur explained how every week everyone in her organisation got together to discuss what they had learnt in the previous week and to agree how they would adapt and experiment in the week ahead. She wanted to know from the national CEOs whether it would be possible to learn and adapt in real time in a large organisation: to which the reply, in essence, was 'no'.

It is worth reflecting on what is lost in terms of immediate assessment, learning, collaboration and experimentation the moment this ability for a cross cutting team to work as a single autonomous unit is lost. Of course, such an enterprise - and this is often the problem with these innovative programmes - would not have the scale or resources to undertake the kind of data driven, scientific evaluations involved in a robust efficacy framework. But, as the seminar highlighted, the purity of efficacy is almost inevitably sacrificed in the fractured world of complex organisations and systems.

All of which leads to a conclusion which is simple to state but, no doubt, tough to act upon. To be powerful, efficacy (indeed any framework for assessing impact) and the clever technology platforms that stand behind it should be uncoupled from top down control systems, should be based on simple powerful principles around which different organisations, or parts of organisations, can collaborate and cohere and - most crucially - should enable the work of people at the front line (including service users) to be more rewarding and creative, not less.

In other words the ultimate test of efficacy - possibly more important than its pinpoint accuracy - is that it is not a tool to empower the hierarchy but a tool to increase the autonomy of those who it is hoped will use the framework in their day to day practice. Or to put it even more simply, does measuring efficacy help people be more efficacious?