Last July, and an oxygen cylinder aboard a Qantas plane suffered what could be considered a relatively rare “behaviour change” in that it exploded, ripping a large hole in the fuselage. Following a stringent follow-up investigation (or evaluation), the cause of the exploding cylinder still remains a mystery. The news item reported that investigators even pressure tested the remaining gas bottles and none failed.
What this shows is that no matter how much knowledge we have about something, or no matter how many tests we replicate, some things remain a mystery. The reason behind the exploding cylinder are complex, and understanding the reason cannot be through attempting to replicate the problem by studying other cylinders, but through understanding the emergent properties related to that particular exploding cylinder. If only cylinders could tell a story? Except for the fact that this one is lost somewhere in the ocean!
So, what does that have to do with evaluating behaviour change? Well, people, like oxygen cylinders, are often considered similar and predictable in that what works for one is considered to work for others. But in reality, we are more like the “exploding cylinder” in that we often react unpredictably, or in a complex and unique manners, when placed in different situations. As such, it is hard to know what parameters to evaluate in behaviour change programs, as we cannot necessarily predict the outcome.
In understanding the reason for a change, what is important is asking the “one that changed”. If only investigators could get the story from the most significant (or exploding) cylinder? In evaluating behaviour change programs, we need to be more open to emergent properties, some which may be instantaneous and highly observable (like the exploding cylinder) and others that may occur over a longer term, and less visible.
Archive for the ‘Evaluation’ Category
Exploding cylinders, complexity and evaluating behaviour change
Wednesday, March 10th, 2010Resources for evaluation in complexity
Saturday, March 6th, 2010A group of Canadians are working with the public health system in Nova Scotia to shift the way health care is delivered to a more collaborative model. Along the way they are running capacity building workingshops exploring the Art of Social Innovation. In response to a recent call from these hosts, several resources have been posted on the Art of Hosting email list relating to complexity and evaluation. Here are a few of them:
- Tamarack Institue resources on evaluation for Vibrant Community including a primer on developmental evaluation (.pdf)
- The Most Significant Change methodology.
- A primer on Action Research
All of these methods have applications for participatory initiatives in complex systems.
People matter!
Tuesday, March 2nd, 2010Evaluation of behaviour change. No mention of people there. Yet it’s people who are at the heart of behaviour change and evaluation. This was brought home to me vividly in this post from an aid worker in Haiti where assessments are an integral part of the response process.
As I write this, I’ve been in Haiti one day shy of a full month. And the truth is that I have spent the majority of my time here up to now chained to a desk. Yeah, I’ve done a few field visits and been to a few cluster meetings, but the reality is that what was most needed by my team was a “text bitch.” So that is what I was.
And so I confess that while I was very willing to do it, I was also a tiny bit disappointed two days ago to find that what was most needed from me was data-entry. Not having deep, intense, heart-wrenching conversations with earthquake survivors, not making big decisions about big numbers, not negotiating complex partnerships with diverse stakeholders.
Nope. Data-entry. Someone had to enter assessment data into an Excel spreadsheet. It is incredibly important to do it and do it accurately. And it’s also incredibly unglamorous. Me, my laptop computer, my iPod and a mountain of hand-written rapid assessment surveys.
About two thirds of the form was numerical, and so entering that data got to be pretty mechanical after the first hour or two. But that last third was all qualitative stuff: open-ended interview questions where at times the respondents appeared to have rambled or gone on wild tangents. But it didn’t take long to see obvious patterns emerging in the ways that people in Haiti seem to view their situation.
If those surveys that I entered are representative of the larger sample, more than anything else, people in Haiti are scared and hungry. Scared of another earthquake. Scared to sleep indoors. Those from host communities, are scared of all these people coming in who they don’t know. Some of them are scared of evil spirits. Many are scared of evil people.
There are no jobs. They have no money. Frequently listed coping mechanisms include “begging”, “nothing”, and “wait to see what God will do to us.”
And they’re hungry.
I have been to a few hardcore places in my time, and I now include post-earthquake Haiti in that “hardcore” category. I am generally able to detach emotionally in the moment for the sake of getting through the task at hand. But this one seems different, somehow. Maybe I’m getting soft. For the past three weeks images of people’s limbs protruding from beneath piles of rubble in downtown Port-au-Princehave been coming back to me during the night. But those images, dramatic as they are, were replaced two days ago by a few lines of scrawled Creole (with English translation) on a smudged piece of paper. A description of chronic, always-in-the-back-of-your-mind hunger by someone who’d lost everything:
“The hunger is… a hole beneath our hearts.”
A good reminder, that in any situation, extreme like Haiti or everyday like assessing local community sustainability behaviour, it’s about people. People matter!
You can read the complete Tales of the Hood post here.
Viv McWaters
What do we measure and Why?
Sunday, February 28th, 2010Meg Wheatley on great questions to ask as we think about measurement, especially in complex living systems (like human communities):
Who gets to create the measures? Measures are meaningful and important only when generated by those doing the work. Any group can benefit from others’ experience and from experts, but the final measures need to be their creation. People only support what they create, and those closest to the work know a great deal about what is significant to measure.
How will we measure our measures? How can we keep measures useful and current? What will indicate that they are now obsolete? How will we keep abreast of changes in context that warrant new measures? Who will look for the unintended consequences that accompany any process and feed that information back to us?
Are we designing measures that are permeable rather than rigid? Are they open enough? Do they invite in newness and surprise? Do they encourage people to look in new places, or to see with new eyes?
Will these measures create information that increases our capacity to develop, to grow into the purpose of this organization? Will this particular information help individuals, teams, and the entire organization grow in the right direction? Will this information help us to deepen and expand the meaning of our work?
What measures will inform us about critical capacities: commitment, learning, teamwork, quality and innovation? How will we measure these essential behaviors without destroying them through the assessment process? Do these measures honor and support the relationships and meaning-rich environments that give rise to these behaviors?
via Margaret J. Wheatley: What Do We Measure and Why?.
These are great questions to consider at this Show Me The Change conference as we dive into questions on the implications for complexity on the measurements used to evaluate change in living and complex systems.
Chris Corrigan – SMTC Design Team
Cross posted from Chris’ Parking Lot here
Taming the lizard brain
Monday, February 8th, 2010I once worked with a young woman who wanted to know, at every turn, what she should do, how she should do it. She was smart, passionate and able – yet she was gripped by fear. Gripped by the fear of not doing it ‘right’. The problem was, and is, that there is no manual – there is no ‘right’ way. As Seth Godin would put it – she was in the grip of her lizard brain, that primitive part of our brain that is either hungry, scared, angry or horny. It’s the reason we are afraid. I heard that she’d recently had a baby. I hope she’s worked out how to tame that lizard brain because I’m pretty sure there’s no manual for raising a child either.
This is the premise of Seth Godin’s latest book, Linchpin. We have a choice to stay stuck, or we can embrace the fear and create some momentum. That’s the good news. The bad news is that our conditioning, and that damn lizard brain, might stop us. We’re conditioned to fit in, not stand out. We’re conditioned to deny our own genius, our art – whatever it is – because we might fail and then the lizard brain can say ‘told you so!’. We fear failure to the point where we don’t even try. Prototyping is all about trying and discarding. Accepting failure. Our lizard brain doesn’t like failure. It once meant we were probably dead, a tasty meal for some predator.
The predators today are no less fearful – it’s just that they are harder to recognise. Security, compensation for our labour, following the rules. These are the things that prevent us from embracing our art and sharing it with the world. Not because we want to get paid, but because there’s nothing else we CAN do, but share our art. Share our passion. We have to accept that it might not work and do it anyway.
Generosity is at the heart of Linchpin, gifting our art to others, not for something in return, not for a later transaction, but for the human to human connection. And for movement. If you’re stuck there’s no movement. It’s hard to be generous if you’re stuck.
There’s no ‘how to’ in this book. It’s a description of what the world needs, and Godin suggests each of us needs to find our own way, create our own map, forge our own future, share our own art, find others who will share the passion and momentum rather than hold us back with the threat of ‘not safe, not secure, not wise’. It’s not a bad description of how to navigate a complex world where even if there was a manual, it would be out of date before you finished reading it.
Discomfort brings engagement and change. Discomfort means you’re doing something that others were unlikely to do, because they’re busy hiding out in the comfortable zone. When your uncomfortable actions lead to success, the organisation rewards you and brings you back for more.
What generosity can you bring to the Show Me The Change conference? What are you doing around behaviour change that is uncomfortable, untried and hard to measure? What do you fear?
Viv
