Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Leader: Keren Winterford

Key Points:

We talked about some limits or challenges for learning:
•    No support time for evaluation
•    Evaluation at the end (“tacked on the end”)
•    Busyness
•    Managers not being involved
•    Often strategy/planning (higher level) org processes, different to true frames of projects: learning can’t fit into higher level org objectives – learning can’t influence this level
•    Reporting (formal) eg; Quarterly reports – people don’t like writing them (“creative writing”) and people reading don’t get what they need/want
•    Question: evaluation needs to be able to change projects, (how to do this?)
•    Often evaluators have different language and hard to connect with
•    Cycles of funding
•    Educate managers of the importance of learning

We talked about some great ways to support learning:
•    Action learning
•    Structured time for reflection in project management
•    Lessons learned methodology
•    Using case studies (tight, structured formats) and peer reviews
•    Structured process for doing something of ‘evaluation learning’ (ie. Process of a response and changed practice as a result of evaluation)
•    Engagement of managers in reflection and learning process
•    Having workshop/reflection meetings (everyone in the room)
•    Practice continuous change
•    Feedback loop to senior management
•    Focus learning processes not just on participants (beneficiaries) but on practitioners
•    Trust important to share the full story (including failures)
•    Right balance of measurement and learning (we need both) – make measurement stuff easy and quicker so we can focus on learning
•    There is intuition in learning- ‘trust ourselves’
•    Monitoring is a part of the process
•    Establish process and structure of evaluation from the start
•    Need flexibility for learning
•    Partnerships are becoming increasingly important and important for learning (design as part of)
•    Need to structure/enable staff to have self reflective learning process
•    There is a need to recognise interactive nature of projects and evaluation
•    Be more creative and innovative in “reporting”, ie; not a report, but also videos, pictures, workshops, opportunity for conversation (through workshops)
•    Use of extended evaluators, yes, not necessarily objective, but they offer a neutral, safe place for participants/stakeholders to really share
•    Recognise not an evaluation should be to change a project (next phase) and then to set up an evaluation (at the start of next phase) and then to know as part of the implementation the evaluation will then need to be revised as new knowing it gained, therefore, interactive process of doing and evaluating
•    Its complex and iteractive

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Participants: Scott McKenry (NCS Swinburne), Elle Morrell (MEFL), Beth Hyland, Candyce Presland, Maxine Schache, Diane Nichols, Cathy Wills (Swinburne)

Key Points:     (INSERT PICTURE)

How do we evaluate impact of training programs over time?

Funders and “deliverers” of programs need to have more conversations about long-term evaluations:
-    what do each need?
-    how can they connect?

What partnerships need to be formed? What relationships need to be credited?

Recognise that we need to move beyond evaluation sheets at end of training

If you want to involve participants in evaluations, you need to train them, get them excited by it as a practice. Makes them more amenable to being ‘evaluated’.

Some of us already doing this:
-    DPI (Farmer CD)
-    ACF (GreenHome)
-    NCS (Carbon Accounting ??)

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Leader: Jess Dart

Participants: Julia Davies, Sue Arendt, Scott McKenny, Maxine Schache, Carolyn Kabore, Lisa Goeman

Key Points:

•    Dangers:
1.    Idea is that some programs that are in the early stages of development, working out and trying different options – then predetermined indicators may ‘squash’ innovation. Evaluating a program a whole range of new ideas could stifle creativity, eg; picking winners and losers too early on
2.    Danger of mandating evaluation of everything – mindset that evaluation is critical and essential. But funnily enough, this is often done at the expense of ‘diluting’ evaluation money and getting very poor quality evaluation – not everything needs evaluating
3.    Focus evaluation on where learning can inform decisions – occur – multiple phase
4.    Is it worth evaluating when costs exceed benefits? Eg; calculating attributable impacts to one particular program may cost more than cost of program!
5.    Concept of evaluability assessment
6.    Focus evaluation on what you want to know
7.    Don’t do evaluation when you have nothing to learn from the answers of when you have no ideas of what you want to know. Don’t do evaluation when it is not worth it.
8.    Why do we collect data that is never looked at and never used and of poor quality?
9.    Potentially we could have ‘levels’ of evaluation – from very light to intensive. Focus evaluation resources where they can have most results
10.    Don’t evaluate a project when its been evaluated to death
11.    Need to always ask why we are evaluating – what is the purpose? Eg; when a project has really failed and everyone knows why it has not worked. What’s the point?
12.    Focus more on using evaluation results.

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Participants: (It ebbed and flowed – quite a few bumblebees!)

Key Points:

•    The workshop was about exploring “lessons from the Dancing Guy” (Derek Sivers – YouTube) and the wave theory (AKA diffusion of innovation). I opened discussion with 2 examples – sustainable events work and designing a pilot shopping centre sustainability project; and how I was trying to use the theory with these initiatives.
•    Key questions to the group:
1.    What occurs to you on watching the Derek Sivers YouTube clip?
2.    What does this theory raise for you re monitoring and evaluation?
•    Key points from discussion:
•    Need to understand what motivates the first followers (and the nuts)
•    Government could create spaces and places that foster the “nuts” and the “first followers”
•    The first followers – makes an action simple, easy and public
•    If you want to work with the first followers (as a change aspect) you need to create apps for them to find you
•    What happens if your champions (your nut/first follower) doesn’t want to go public? Need to understand why and evaluate this
•    The discussion turned to suggestions for government change programs to support the first followers and nuts
-    create standards
-    networking spaces
-    recognition programs
-    peer reviewing from industry
-    “on the job” training programs

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Participants: Helga Svendsen, Kirsty Fenton, Michael Baranovic, Cheryl Samarasuphe, Danielle Kennedy, Marcia Hewitt, Ian Grant, Bruce Paton, Jonathan Russell, Rebecca Petit

Key Points:

•    There are many different BC models used today.
•    Models are inherently limited (they are linear, they cannot capture the complexity of people – their motives, their behaviour).
•    A common language could both assist and limit us (eg. Assist discussions but also assist assumption – define “evaluation”).
•    Aware of the above limitations of models and language – it could be useful to have a compilation of models and situations where they have worked and not worked.
•    From this a matrix of factors to consider may assist us in finding the ‘best’ model currently available.
•    This matrix would need to include – timescales, resources, connectedness of community, values focussed on, etc.
•    BC Matrix Model factors:
-    Self interest eg, AIDS
-    Timescale (affect now, yes)
-    Geographic
-    Complexity of message (eg, climate change is highly complex)
-    Completeness of science (eg, climate change – scientists don’t agree)
-    Prior Learning
-    Limitations of models – constraints
-    *Complexity of people difficult to capture in a model*

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Participants: Andrew Rixon, Keren Winterford, David Williamson, Julie Harris, Candyce Presland, Natalie Moxham, Felicity Thomas, Jonathan Russell, Adrienne Fanning, Vera Lubczenico, Murray Irwin

Key Points:

•    We agreed that a lot of pretending goes on around evaluations. It ranges from quietly allowing people to misinterpret evidence when it suits, through being economical with the truth, to pretending project criteria are useful when they aren’t.
•    Evaluation can create pressure to put on a performance and concealing unintended consequences or failures sometimes feels necessary – but results in considerable waste and lost opportunities to learn
•    Creating more informed management relationships supports greater ??, as does negotiating longer term funding. We found not easy solutions – but it may be good to start to at least acknowledge a serious problem exists.

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Topic: Our connection with nature – nature as teacher in behaviour change and evaluation

Participants: Tanya Loos – City of Ballarat (nature@wideband.net.au)
Ellen Regos – Royal Botanic Gardens, Melbourne

Key Points:

•    Connection with nature – isn’t that our motivating factor for so much of our work in environmental and social sustainability?
•    Do we value our experiences in nature? Are nature walks, meditations and sensory activities valid tools for sustainability/BC conferences?
•    This conference has been a lot about the head – sensory experiences and heart based activities, our whole self – this has been missing.
•    However, deep ecology and other “fringe” education and BC methodologies have shown that creating beautiful, fun, sensory experiences in nature we actually highly successful ways of generosity behaviour change.
•    Our conference question – just go outside, sit in the garden and see what answers come.
•    The mainstream does not value nature as a tool as teacher, it is seen as not valid or rational, not part of the built environment.
•    Certainly, we need to frame nature connection experiences appropriately so as to not alienate people. Big fear: “we are not going to hug trees are we?”
•    Can we bring in nature connection methodologies into the more conservative/traditional spaces of workshops, eg, “field visits”.

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Topic: Who actually cares about evaluation? Who is for? Why do they care?

Participants: Jonathon Russel, Greg Bruce, Mark Butz, Julian Denler, Sarah Bartlett, Candyce Presland, Mike Rowell, Rob Catchlove, and more…

Key Points:

•    Different people care about different aspects of evaluation. This can be about accountability and/or learnings. It is important for funders to justify expenditure (in case someone asks about the project).
•    It is important to include mistakes/lessons/failures, to have a balanced report. Lessons need to be carefully phrased so as not to “surprise” or “shock” or “bring fear”.
•    An independent evaluation can bring objectivity, and help to find unexpected consequences.
•    Evaluation reports (especially executive summaries) are useful in review literature to design projects, but often they do not have a summary and clearly outline what worked and what did not.
•    Safe-matching communication (collective social learning) offers a methodology for evaluation.
•    Remember:
1.    Evaluation is audience specific. Eg, you may only need a 5 minute video/snapshot for a councillor
2.    Do not separate evaluation from project management! Use it as a tool to keep learning along the journey
•    What is needed:
•    A forum/website only evaluation reports that showcase failure
•    Easily accessible evaluation reports for collective learning (not hidden on funder’s shelf)
•    Risk – the media can latch on to bad news and stifle creativity (and lead to unbalanced reporting
•    Remaining Question – “has accountability gone too far?”

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Leader: Jon Kendall

Participants: Anna Straton, Beth Hyland, Anna Lohse, Jonathan Davy, Fran Westmore, Megan Hughes, Mark Butz

Key Points:

•    We focussed all the challenges of the two worlds of change
•    ME     – satisfying the needs of “funders”
– doing things to create sustainable change
•    Tempting to think that we have to pretend in various ways in the direction of funders, and tempting to think something huge has to happen to move the situation on. BUT we concluded that small things are where the power and opportunity live – what small things can we do to probe/test what seem like fixed boundaries? The small steps often create big ripples!
•    And the apparent paradox of funded change! In order to make a difference we decided to maintain the status quo.

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Topic: Our “Mindset” and Worldview as we Evaluate Behaviour Change

Leader: Geoff Brown

Participants: Murray Miller, Ross Egleton, Chris Corrigan, Jan Smith, Jon Keudau, Maria Eliadis, Fran Westmore, Maude Lecourt, Brian Hardy, Diane Nichols, Kathryn McCallum, Sarah Bartlett, Sue Xavis, Andrew Rixon, Debbie Coffey, Stephen Kelly, Felicity Thomas, Lisa Keedle, Jonathan Dacy, Tanya Loos, Martin Hausenblas

Key Points:

•    Willingness to be a learner as opposed to an expert – a key mindset needed across the field
•    Sam Ham’s work – as project leaders we do things in the way “we think” it should work
– A Healthy Assumption we should take is this: “we can learn from our participants”
•    Mindset – we still assume a causal link between providing people with information/experience and the behaviour change we want to see
•    Should we even be focussing on the “set of behaviours” we want to change? Instead should we be focussing on other things like political processes, infrastructural change, social networks/relationships?
•    The notion change “hurts” – how can we support people through this?
–    Re: Biodiversity – help people to notice the things around them (respect the things around them)
–    Maybe our role is to help people simply notice the things around them rather than trying hard to change them?
•    As an evaluator it can be a trap to also be the Architect of the Program because the design is therefore “my baby” – too much attachment and bias
•    Relationships are critical between people in our projects – we often assume that the relationship b/w us (ie, our project) and the participant is the most important one to enable change, maybe we should do more to support the relationships and conversations between participants (”we do what we do under the influence of others)
•    Is evaluation part of the project? Look for opportunities for participants to get involved in evaluation activities – participatory evaluation techniques like Most Significant Change (MSC)
•    Having different “Roles” and different expertise in the design and evaluation of a project is critical
•    A Mindset of “Flexibility” is critical – that is our targets and goals may have to change as a project unfolds – this conversation with the funder can build trust and relationships
•    We need to be able to work and link up what happens @ a Distance (strategic level) @Close-Range: The Actions
•    Giving people (participants and partners) a sense of ownership is critical also
•    Story of Change: Narrative Posters (Andrew Rixon – see www.babelfishgroup.com.au)
1.    “You have all the resources you require” – 1st Challenge and the Mindset needed
2.    “We are doing the best we can”
3.    “Being flexible and hanging loose”
Dilemma raised is the discussion: “WE ARE UP AGAINST IT AND IT’S URGENT!! We don’t have time for all of this!”
-    Refer to the Book: “Time Paradox” Phil Zimbardo – our perception of time can bring about much anxiety
•    Maybe the anxiety around “time pressures” stops us being effective and actually slows us down

Healthiest Time Perspective – refer to TED video (http://ca.ted.com/talks/lang/eng/philip_zimbardo_prescribes_a_healthy_take_on_time.html)
1. Past Positive: appreciation of the past is critical
•    Mindset that our participants have the resources
•    Inquiry Mindset = Listening & Respect the Community – spending time learning the “trigger points”
•    The “Reality” of our participants is important to understand
•    Identify your “behaviour change” journey? – Understand it in ourselves first … walk the talk
•    We need to be living the change we are bringing about to others
•    Being comfortable in saying “I don’t know” is difficult for experts – we need to break this Tyranny!
•    Mindset – Can I even expect to change anybody in my work?

References

1. Dave Snowden’s (Cognitive Edge) work on Complexity helps us to gain a healthy mindset when working with behaviour change projects – refer to this video by Shawn Callhan at Anecdote (http://www.youtube.com/watch?v=5mqNcs8mp74)

2. The ‘failures’ of the Castlemaine 500 Project inspired this topic and their is a chapter on the ‘mindset’ needed at the back of the report – http://www.cvga.net.au/main/index.php?option=com_docman&task=cat_view&gid=15&&Itemid=72