Back to impact

After an initial post last week I said that I would return to the issue of impact. I know it's not exactly scintilating to folks outside the RSA but perhaps it is an insight into the small and troubled mind of a working CEO...

It may be obvious but it’s worth identifying some of the hard constraints on measuring social impact.

Complexity and perversity: generally the world is just too complex to be sure that any major social change is the consequence of a particular action. In some cases good results come from bad actions and vice versa. For example, a community may become stronger as a result of suffering a bad decision and some – particularly on the right – argue that being ‘too generous’ to some communities has created a damaging culture of dependency.

Opportunity costs: even if a social project looks impressive there is always the question of whether, if the resources and people dedicated to it had been directed elsewhere, a better result might have occurred.

Parallel universes: even where a project looks great and its impact robust compared to other forms of spending you can never be sure of what would have happened had the project not come along but a slightly different response had developed.  To make the point, if you asked anyone in a successful marriage with nice kids how much their happiness depended on their family they would probably say ‘enormously’ and it would be true.  But, of course, it may have been that had they not met their spouse on the day they did they could have met someone completely different the next day who could have gone on to make them even happier.

Having said this I am convinced that the general approach to looking at impact has to be based on a model of tiers, starting with the most obvious and modest and moving outward to the most complex and ambitious. So, for example, New Philanthropy Capital in a recent evaluation of the School for Social Entrepreneurs used a five step theory of change starting with ‘needs’ (what are the needs the project is aiming to address) and moving through ‘actions and activities’, ‘immediate outputs’, ‘medium term outcomes’ and ending with ‘longer term outcomes and impacts’.   This isn't a million miles from the framework suggested by my colleague Jonathan Rowson (in a comment to my previous post):

Another issue is rigor versus flexibility. If every organisation could choose its own preferred metrics for evaluation they could sure enough find a way of showing they had succeeded. But, the value of the kind of frameworks set out above surely lies in trying to apply them to concrete activities.

For example, we might create a six level impact hierarchy for RSA lectures which ran like this:

Of course, the measurement gets harder the higher up the hierarchy one goes. While it is easy to count how many hits RSA lectures get on You Tube, it is harder and more resource intensive to try to find out whether and how people changed their lives by watching a lecture.

Because it is brave to look for these higher impacts I think we should be realistic and willing to make mistakes. The important thing is to work towards a reasonably robust measure (even if it is a proxy of a more complex impact) and to be willing eventually to proffer a testable hypothesis.

So for lectures (and, by the way, I am choosing this part of the Society’s output because it is the most clearly successful) we might end up with things like a survey of people who came to selected events asking them several months after the event if they still remembered it and if it had made them think or act differently. Or perhaps we could try more systematically to record all the teachers and students who say Animate has opened up ideas in a new way.

These stories are difficult to quantify but as long as data on the earlier steps of the impact hierarchy are robust, such anecdotes may be about the best evidence of transformation we could expect. Here are extracts from a couple of emails we have received in the last few weeks  

Your RSA Animate video Empathic Civilization has become the impetus for an entire Youth Summit we at Invisible Children are doing here in San Diego with 700 hand selected kids from the US, UK, Australia, Mexico, and South Africa. 

Or this

First and foremost, I am a huge fan of RSA: 21st Century Enlightenment and especially impressed with RSAnimate. I love the idea of promoting innovative ways of thinking through compelling visual and audial techniques.

Being a freshman in an American college, I feel uncertain about many potential career pathways. RSAnimate has interested me since I first discovered it through Youtube. I enjoy exploring theoretical possibilities for improving education and would like to someday implement my ideas in the real world. Working for RSAnimate or a similar company is a career I could see myself enjoying until my final days.

So I hope that over the next few months we can work towards a more ambitious and interesting way of thinking about the RSA’s impact. Some people might say it’s a distraction from the day to day work of grinding outputs but my feeling is that organisations like the RSA need to attract and retain people who want to change the world, if so, they’ll also want to know how they are changing it.