“Unimagined possibilities came to the surface. Imagine if there was a celebration of the natural world as a part of this conference. Wasn’t this why we were really here anyway? Wasn’t this the real reason for our work? Or was this a totally incorrect assumption? Was this just a part of my imagination? This conference had created a doubt.”
“My learning at this stage had not been great, day one had been difficult and today more enjoyable, but as to the value of attending this conference I was very unsure. Nothing life changing, no great learning in this process, not that I was expecting anything anyway…until.”
Ellen Regos, from the Metropolitan Waste Management Group, has taken the narrative thread that weaved its way through the Show Me the Change Conference to heart and written her story. She has captured the essence of sitting with uncertainty, disappointment and fear and the moment of transformation – a moment where everything is the same, yet everything is different.
For Ellen, this moment came during the conference. For others it may come at another time and place – or it may never come. The sum of our experiences is what makes us who we are, and some experiences resonate more than others.
As a group responsible for designing this conference we are grateful to Ellen for capturing the essence of our approach – opening up to complexity by immersion in complexity; conversations, relationship building then transactions; providing space to share our stories and questions; and a narrative thread held together by our individual and collective experiences.
I recently watched this TED Talk by Daniel Kahneman – he won the Nobel Prize in Economics for his pioneering work in behavioral economics.
The key takeaway for me was the disconnect between ‘experience’ and ‘memory’. This got me thinking about all sorts of contexts, from own happiness to the implications for evaluating behaviour change. At the TED website they describe his presentation as …
“Using examples from vacations to colonoscopies, Nobel laureate and founder of behavioral economics Daniel Kahneman reveals how our “experiencing selves” and our “remembering selves” perceive happiness differently. This new insight has profound implications for economics, public policy — and our own self-awareness.“
After I watched the presentation, I sat down to write this blog post and struggled to document what I was thinking. This post was destined to live a long life in the ‘draft’ folder … then I emailed the link to Irene Guijt (Evaluation Revisited Conference) … who passed it onto Dave Snowden (Cognitive Edge) … and then Dave wrote this post – Between Memory & Experience. Dave writes …
“Seeing the future as anticipated memories is an interesting side idea from the talk. Now there is nothing new here (although its well presented) but its an important reminder in relationship to issues of narrative research.”
Here’s the video – see what connections you can make between the findings and the real world.
I recently watched this TED Talk by Daniel Kahneman – he won the Nobel Prize in Economics for his pioneering work in behavioral economics.
The key takeaway for me was the disconnect between ‘experience’ and ‘memory’. This got me thinking about all sorts of contexts, from own happiness to the implications for evaluating behaviour change. At the TED website they describe his presentation as …
“Using examples from vacations to colonoscopies, Nobel laureate and founder of behavioral economics Daniel Kahneman reveals how our “experiencing selves” and our “remembering selves” perceive happiness differently. This new insight has profound implications for economics, public policy — and our own self-awareness.“
After I watched the presentation, I sat down to write this blog post and struggled to document what I was thinking. This post was destined to live a long life in the ‘draft’ folder … then I emailed the link to Irene Guijt (Evaluation Revisited Conference) … who passed it onto Dave Snowden (Cognitive Edge) … and then Dave wrote this post – Between Memory & Experience. Dave writes …
“Seeing the future as anticipated memories is an interesting side idea from the talk. Now there is nothing new here (although its well presented) but its an important reminder in relationship to issues of narrative research.”
Here’s the video – see what connections you can make between the findings and the real world.
I recently watched this TED Talk by Daniel Kahneman – he won the Nobel Prize in Economics for his pioneering work in behavioral economics.
The key takeaway for me was the disconnect between ‘experience’ and ‘memory’. This got me thinking about all sorts of contexts, from own happiness to the implications for evaluating behaviour change. At the TED website they describe his presentation as …
“Using examples from vacations to colonoscopies, Nobel laureate and founder of behavioral economics Daniel Kahneman reveals how our “experiencing selves” and our “remembering selves” perceive happiness differently. This new insight has profound implications for economics, public policy — and our own self-awareness.“
After I watched the presentation, I sat down to write this blog post and struggled to document what I was thinking. This post was destined to live a long life in the ‘draft’ folder … then I emailed the link to Irene Guijt (Evaluation Revisited Conference) … who passed it onto Dave Snowden (Cognitive Edge) … and then Dave wrote this post – Between Memory & Experience. Dave writes …
“Seeing the future as anticipated memories is an interesting side idea from the talk. Now there is nothing new here (although its well presented) but its an important reminder in relationship to issues of narrative research.”
Here’s the video – see what connections you can make between the findings and the real world.
There is general agreement that word of mouth marketing is a critical element of changing behaviour. Whether it is a family member, colleague, neighbour, or friend, we are more likely to take on the advice and behaviours that are modelled by those we trust. This is the basis of effective communication (think also of the 6 degrees of separation experiment). Mark Earls, the author of Herd: how to change mass behaviour by harnessing our true nature provides great examples of how social networks are key to changing mass behaviour.
Mark recently posted a blog about how important it is to understand social networks.
Mark notes: Social networks are not channels for advertisers or for the adverts/memes you, your clients or any of your so-called “influentials” create, social networks are for all of the people who participate in the network.
So if word of mouth is an element of your behaviour change program (as it should be), how can you track its spread, and find out whom the key people are in networks? Well, social network analysis is one way! So what is a social network analysis?
Andrew Rixon, from Babelfish Group, notes in an e-booklet on enhancing collaboration that Social Network Analysis is the technique of analysing roles and social networks…. The outcomes of social network analysis provides surprising and insightful results allowing structure(s) to become visible and discussable.
Making such networks visible should surely be one of the goals of evaluation. In this way, for those who have read Gladwell’s Tipping Point, you can find out who the mavens, connectors and salesmen are.
Interested in finding out more on Social Network Analysis?
Andrew Rixon will be holding a post-conference workshop on this very topic, so check out the program of post conference workshops and register online.
I see a lot of similarities between behaviour change interventions for sustainability and international development assistance. Both fields seek to intervene to change participants’ behaviours, and generally this is done through a linear model of cause and effect, where the intervention is evaluated as the sole agent of change. In a recent post on complexity and development, Ben Ramalingam highlights a recent publication by Olivier Serrat, Head of Knowledge Management at the Asian Development Bank:
Development is a complex, adaptive process but—with exceptions—development work has not been conducted as such… development assistance often follows a linear approach to achieving outputs and outcomes……Any planning process is based on assumptions—some will be predictable, others wishful. If the assumptions are based on invalid theories of change (including cause-and-effect relationships) and on inappropriate tools, methods, and approaches derived from those, development agencies jeopardize the impacts they seek to realize.
In terms of evaluation, the risk is not solely that we jeopardise the impacts, but that we choose evaluation methods that will seek out what we want to show, whether this has actually occurred or not. If we are intent on showing a particular change, it is quite easy to (inadvertently or not) seek out what we (want to) believe actually happened, and by doing this we reinforce the perpetuation of behaviour change interventions that may not be all that successful. And in doing this we reinforce the status quo, rather than move towards better practices that account for complexity.
Here’s a nice quote from Aaron Levenstein to keep in mind:
Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital.
A post on evaluation and complexity on Rick Davies monitoring and evaluation news site has a link to a great powerpoint on evaluation and the science of complexity by Ben Ramalingam. Ben notes that for many organisations, evaluations are at the centre of a vicious circle that includes pressure to show results and impacts, and poor learning and accountability amongst others.
Further, Ben notes that “Evaluations are still largely focused on reports as opposed to changed behaviours, ways of thinking and attitudes”. This seems very true, and I am sure many of us would recollect knowing of reports that have been produced for the report’s sake, and not what is in it.
The image, taken from one of Ben’s slides, encapsulates well the idea that existing process and culture can overshadow the ability to undertake more effective evaluation.
In another post, Ben notes “Some of the issues for evaluation include the tension between learning and accountability, the limits of attribution, how evaluations are or are not used, equality and power, and ideological debates about methodologies, such as the dominance of randomised controlled trials (RCTs)……Although there is a wealth of evaluation methods in theory, in practice they are largely required to conform to scientific management principles……..In contrast, complexity theory (theories) talks about systems that are interconnected, driven by feedback, where the properties of the system are not predictable but emerge from the relationships within that system……..It may be that we need to stop focusing on projects, and look more broadly at the societies that we work in and across sectors and institutions rather than within them. Evaluations may need to be more centred on real-time learning and helping managers adapt what they do.”
This is what Show me the Change is about- discussing how as a community of practice we can overcome the real or imagined culture that can negatively impact on more novel, experimental, and altogether better evaluation practice and processes.
So if you are interested in evaluation, behaviour change and sustainability in a complex workd, take part in the conversations that matter, on 4-6 May in Melbourne.
This is a question that we don’t really ask ourselves, as we are often made to evaluate (to account for the money we received and spent), and to is also in our nature to demonstrate success (and hide failure..??).
A recent post by by Chris Brogan, that we should pursue the goal, not the method (also picked up by David Gurteen, made me think about how this applies to evaluation. Often, we consume ourselves on developing and implementing a methodology to evaluate change, as opposed to actually focussing on achieving the change itself.
This made me recall a conversation with Greg Bruce, from Townsville City Council, where he and his team are achieving some transformative change, across the community, but also throughout the Council and partnering organisations.
Greg wondered whether it was of any use to focus so much effort on evaluation, especially considering the short time frames in which traditional project evaluation is undertaken (that is, during and immediately post-project). Why spend so much time, effort and money, when you could re-direct the focus towards on-ground action and transformation. For Greg, evaluation “needs long timeframes to evaluate impact to effort – in system dynamics in order to show effect”. Greg proposed that equally good indicators of success would include the buy-in of other organisations (whether through official or unofficial partnerships), receiving further grants, as well as recognition from local, national and international delegations (in in Greg’s humorous ways, “they will come and go as equally mystified as when they arrived“).
So, how much effort should we put into evaluating something that will likely occur over a long time frame, way beyond the scope of the project, or interest of funding agencies? Let us know what you think? Post a response, ask the question amongst your colleagues and network partners, and share your conversations at Show me the Change.
This post resonated loudly with me because of the recent work I have been doing as a facilitator with Government agencies. Also, the upcoming Show Me The Change conference in Melbourne this May is also linked.
Here’s a snapshot of Neil’s Agile post:
We are all bombarded by change and it’s accelerating (and will not slow down) … the enterprise of the future will be hungry for change … but our current business/government are stuck in linear systems that are slow, top-down & inflexible …
“Our structures need to be more speedy. Speed used to kill now lack of speed kills. Lets have organizations that can iterate quickly and empower its folks to make decisions. Percolating decisions up and down an organization makes little sense” Rishad Tobaccowala
… being ‘agile’ is not a process or panacea – it’s a philosophy that you either have or you don’t … Agile is:
… welcoming of changing requirements, even late in development, because it is an opportunity to harness change for competitive advantage. Big business creates big projects that take a long time to confirm, implement, and complete. Projects are often stalled by hierarchical management processes.
… focused on frequent deliverables, with a preference for shippable product and shorter cycles, and implemented at a constant pace which is measured and transparent.
… centred around the belief that the best results come from self-organising teams. Teams that reflect regularly on how to become more effective, then adjust behaviours accordingly.
… projects are built around trusted, motivated individuals who are given the environment and support they need.
… documentation is kept to a minimum, with face-to-face communication preferred, and a focus on simplicity – maximising the amount of work not done.
And I really like this paragraph as it sums up my observations of the planning processes that so many of our large institutions are stuck in …
“I don’t claim here that agile development processes are some kind of cure-all. But I do think that business processes in many industries and organisations are woefully out-of-date and hopelessly rigid. Businesses increasingly operate in complex adaptive systems which, as Bud Caddell rightly says, are “characterized by perpetual novelty – talking of equilibrium is pointless, equilibrium in a complex adaptive system is essentially a dead system”. Inflexible, long-term strategic plans are increasingly irrelevant.” Neil Perkin
Show Me The Change
Ok, so what’s this conference got to do with Agility? Everything!!!!
Participants who are coming to Show Me The Change (deep down) know that our current approach to the design, staging and evaluation of behaviour change projects needs a re-think. We need to take stock and challenge our assumptions about behaviour change and the goals that we set. We have tinkered with ‘behaviour change’ tools and tactics for long enough. It’s time to go deeper.
In the context of ‘Behaviour Change projects’, agility is a mindset and NOT a set of processes, strategies and tools. It’s a philosophy that everyone involved adopts.
In practice, I think it involves these types of tangible features (some adapted from Dave Snowden’s post here):
Do lots of little things rather than 1 big thing – Don’t put all your resources into 1 strategy … now matter how much ‘planning’ and research you have done
Don’t be afraid to experiment and some things will fail – We often learn more from failure than success anyway
Design projects that can be ‘monitored’ – By monitor I mean we can quickly notice if things are working or not. It’s no good if you need to wait 12 months for the data you need!
Beware of ‘Best Practice’ and the success of others - Why? Because your context is different. Just becasue a series of home assessments lead to a reduction in household energy use in Town A, it doesn’t mean the same will apply in Town B
Challenge the ‘Goals’ and ‘Evaluation Measures’ imposed on you by funders – Often we get asked to report a whole heap of indicators that are useless. At times, even the underlying goal of a project is unattainable … it’s aspirational at best. Don’t get sucked in to being ‘measured’ against such goals/aims. Have these conversations early.
Challenge the direction and priorities of your project as it emerges – If you pay attention and monitor your project, new insights will emerge. Many will be unexpected and point to new priorities. Don’t be afraid to challenge your project plan and re work it along the way. Again, have these conersations early on.
Collect and share Stories – Stories of failure and success are critical. Stories are memorable and they stick. Analysis of stories can uncover patterns that data analysis misses.
Don’t be fearful of failure – Did I say that already?
Last July, and an oxygen cylinder aboard a Qantas plane suffered what could be considered a relatively rare “behaviour change” in that it exploded, ripping a large hole in the fuselage. Following a stringent follow-up investigation (or evaluation), the cause of the exploding cylinder still remains a mystery. The news item reported that investigators even pressure tested the remaining gas bottles and none failed.
What this shows is that no matter how much knowledge we have about something, or no matter how many tests we replicate, some things remain a mystery. The reason behind the exploding cylinder are complex, and understanding the reason cannot be through attempting to replicate the problem by studying other cylinders, but through understanding the emergent properties related to that particular exploding cylinder. If only cylinders could tell a story? Except for the fact that this one is lost somewhere in the ocean!
So, what does that have to do with evaluating behaviour change? Well, people, like oxygen cylinders, are often considered similar and predictable in that what works for one is considered to work for others. But in reality, we are more like the “exploding cylinder” in that we often react unpredictably, or in a complex and unique manners, when placed in different situations. As such, it is hard to know what parameters to evaluate in behaviour change programs, as we cannot necessarily predict the outcome.
In understanding the reason for a change, what is important is asking the “one that changed”. If only investigators could get the story from the most significant (or exploding) cylinder? In evaluating behaviour change programs, we need to be more open to emergent properties, some which may be instantaneous and highly observable (like the exploding cylinder) and others that may occur over a longer term, and less visible.
