Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Topic: How to reconcile competing values with a shared future?

Leader: Michelle Howard

Key Points:
•    We discussed the universality of many values and how the same value can manifest in different behaviours. There is a risk that we judge others (“Us & Them”) when we are all the change. So how can we support more sustainable behaviours with good in G??, real choices, examples of alternative ways so that we can all take the risk to let of the old trapeze and grab hold of a new one!
•    Values ?? true behaviours different.
•    We are all the change. Un??
•    With nature of ch?? other peoples behaviour.
•    We need ?? and options to be part of change towards a more sustainable future.
•    Organic/voluntary change or “enforced” change – “It’s a bit of a wicked problem”
•    Opportunity to work with positive examples and to sow on fertile ground.

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Kathryn McCallum & Ross Egleton

Participants: Keren Winterford, Rob Catchlove, Deanne Wijey,
Jacqui Boreham, Alison Wallace, Candia Bruce
Hayley Giachin, Greg Bruce, Sarah Bartlett,
Gillian Paxton, Julian Donlen, Mike Dodd,
Sue Arndt, Christina Ting, Adrianne Fanning
Ross Egleton

Key Points:

Surfcoast Energy Group, Port Phillip Bay “Human Sign” where parents we transformed to become more sustainable as their children influenced them to join the sign.  WorkSafe Health checks that encouraged story telling within organisations to talk about health and de-stigmatise health issues, GET UP projects.

Key Themes

Design
Include baseline, decide on outcome and work backwards, decide to evaluate ripple effect or simply the activity, embedded evaluation at different levels and cater for unexpected outcomes.  Unexpected outcomes can become recommendations and can also influence expected outcomes.  If evaluating ripple effects then need mechanism to collect information.

Networking
Determine functional groups where people can report back to ensure community is holistic eg the school community goes beyond students and parents.  Need cross sector community group, use strength in networks, it’s a fallacy that we can control, allow for waves of change s actions may be immediate or dormant.

Tools
Storytelling was crucial and provide forum for this and for people, as much as possible to relate effort (good and bad) back to the functional group.  Learning forums such as reunions or ½ day sessions to people to share their stories which can be treated as a closure or celebration of “where to from here”? – collective social learning with action plans.  Education has limitations and the “nag factor”.  Discussed incremental change v’s massive change and important the policy is v. powerful and that research informs policy.

Digital networks – count internet views or hits relating to original email of social pressuring and data in database non-linear – see social network analysis or June Holly – “Network Weaving”.

Copyright approval – www – creative commons.com

Resourcing emerging properties / groups / successes relate reporting back to the activity / group.

Evaluation can be clear cut, especially in research, if well designed though harder with community action.

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Bridge Wetherall

Participants:

Key Points:

•    Making it relevant

•    When they are ready

•    Want to change

•    Imposed change & moved along by peer pressure

•    Reward & recognition – certificate etc.

•    Minimising assumptions

•    Tapping into existing groups – “green parenting” – churches

•    Non-english speaking backgrounds

•    Shift to “their” timelines

•    Research to understand various stages of a project to accommodate “early adoptors” and “strugglers”

•    Use interim evaluation data to prove a point

•    Evaluating regularly

•    Network to know what is going on in your area – local Gov’t network exists

•    ICLEI – run its course – local Gov’t taken over role

•    Greenhouse Alliance – SEW to join

•    Roadmap of what is going on around you

•    Buy-in of program for middle management

•    Scaleable data – different data provided to different levels – mngt, stakeholders

•    Peer mentoring to get middle mngt to buy-in

•    Choose not to work with certain groups in “this” time and context

I 3 model         |    involvement
|
———————–    intervention
|
|
|

•    Media involvement

•    School pick up and drop of times to influence change ie mother with children

•    Understand what else is on in the area and when so no conflicting “messages”

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Felix Aeber

Participants: Sarah Gorman, Max, Millicent Burke, Sarah Partlett
Stefan Koufman, Bruce Paton, John Harvey,
Bridget Wetherall, Jim Curtis, Catherine Doran
Ian Blain

Key Points:

•    Evaluating the independent contribution of a project is inherently difficult.  A first step is to identify and catalogue co-variants.

•    Asking members of the target population about attribution might be a viable option, but is suggestive and misses social diffusion effects.  As evaluation is not the same as measurement, data that is collected require an appropriate interpretative context.

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Tani

Participants:

Key Points:

Two year program, community garden, start mid end evaluation.

•    They used nickname to provide stats for confidentiality
•    Completed mid project workshop for evaluation and sharing
•    Interviews at end, one on one
•    30 participants
•    Used consultancy for survey / evaluation
•    Easy to get feedback due to already formed group.

“Living 4 our future”

City of Boroondara – sustainability, water, food, waste program pre & post evaluation, measured footprint, 3 workshops, incentives provided saw concrete outcomes by com parity water bills etc.

Good tool – global footprint tool, EU states most creditable tool.

A reduction was seen – it resource use.

“Most significant change” – method = story telling methodology.

On-going indepth long-term interviews highly regarded.

Key = acting, reflecting, learning.

“Participationary approach” – google for websites.

“Action Research”

Outcome mapping.

Spider diagrams – set criteria to evaluate work – shape changes over time.

Radar graph – participatory method.

It is about the people involved creating the data to be measured – visual immediate.

Role modelling method – stakeholders take role of group / individual.  Empowering as it goes.

Assists in seeing how the individual is effected.

Only suitable on a small scale.

Also suitable on a small scale.

Also suitable for team members in acting out the issues.

Useful to ask “critical” friends to look at your evaluation tech.

Valuable way to progress knowledge is to keep talking, meeting, discussing – challenges research to stay relevant.

Make evaluation embedded in process and project plan.

“Program logic” design tool for planning clarification evaluation.

You can complete an internal evaluation for your team for learning and internal use only.  Then when completing the external evaluation this should be an easier task.

Very valuable to verbally have a conversation with participate.

QSR – Doncaster.

“MVIVO” program – can record conversations have them typed up then placed in “MVIVO” program to make it rigorous = code issues to sort collected data.

Two questions to ask:
1.    What would you say to others to attend XYZ?
2.    What would you say to others as to reasons not to attend?

All good evaluations should use a variety of tools to get different angles.

•    Book – the trianglisation design
•    Have / use multiple evaluation tools to ensure you have the right data / answers somewhere along the way
•    “Plan-do-act-check” – around since 30’s
•    The risk analysis of your project plan will guide you on where you need to focus and guide your evaluation
•    If evaluating your own program – a diary is handy to keep
•    Debrief with others to get learnings.

Allow all learning styles to communicate in their style:
1.    amiable
2.    kinaesthetic
3.    verbal
4.    auditory

Don’t just evaluate in the style you prefer when collecting evaluation !!

Tool – photo – learning cards.

“Gallery technique” – people work on walls and present / flipchart.

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Tricia Hiley
tricia@potentialspace.com

Participants:

Key Points:

We had a rich and flowing sharing of stories, people working with projects in regional or rural areas.

Some of the themes included:

1.    Lack of access to services increases isolation.
For instance -

•    No local insulation installer and no local relationship with one from “away”
•    Home sustainability assessors not funded to travel to rural locations (ie travel costs paid from post office closest to clients house)
•    Don’t have local trades people
•    Not eligible for water tank rebate because they are not on town waters
•    Regional project staff may only be able to do one visit a day because of distance compared to 6 or 8 in the city

How do we take this “systemic” consequence into account when planning and evaluating ∆ projects?

2.    Enthusiasm and involvement of whole communities.
Examples include:

•    Heyfield, where the whole of the community has embraced a sustainable town project.  It includes a “game” where people can, through their sustainability efforts, “attain” a white then green then blue flag to place prominently on their roof for all to see and celebrate.

•    Mildura, which has benefited by being a small, discrete community which has an inherent focus such as a city community may find it difficult as residents are members of so many different “communities” (ie living, working, school, shops etc.).

3.    There was a reasonable amount of sharing of the experience of vulnerability in farm families, particularly where there is a single income in which leads to a perceived lack of robustness – which can lead to resistance to change.  One group member introduced “crunch theory” which considers a community’s vulnerability with respect to natural hazards.  The example of the consequences of a similar sized earthquake to Haiti and Chile was given as an example.  In Haiti over 200,000 died.  In Chile it was in low hundreds.

4.    A couple of images that linger for me are  (I NEED THE ARTIST HERE)
a)    “the value of one tree in the paddock”
b)    a small town with a white, green or blue flag on the roof of every building
c)    “the country wave” as a young woman waves to or says hi to everyone she passes
d)    a set of scales with “natural hazard” on one side and “community vulnerability” on the other
e)    a project worker in the city – a project worker in the country

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Chad Foulkes

Participants: Many

Key Points:

Liminal space is where I discover what is and what could become

•    Teaching people to juggle (let go)

•    Questioning

•    Reflection – silence – structured – personal

•    Case study – see others experience & relate it to ones own experience

•    Loosen patterns & allow new / other things to connect to it

•    Simulations

•    Make a statement, stop talking & wait for others to contribute

•    When there is no stated “activity” yet the group “agrees” to do what it does – this can bring about change

•    Disrupting habits is our challenge

•    Humour as a way to get to limited space

•    Facilitate self evaluation rather than evaluate what happened

•    The report is the artefact, the work has been done then

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Candyce Presland

Participants:    Many

Key Points:

•    Making the information relevant to the audience know the audience

•    Using interim evaluation data to evaluate throughout the program

•    Collaboration

•    Consistent messaging (not having too many messages or mixing them)

•    Sharing information about network

•    If a program doesn’t get “buy-in” learn from it and move onto something that will

•    Name and shame if necessary (as a last resort)

•    Look at the ithree methodology

•    Sell it – media – photo opportunity

•    Have a champion or influence

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Candice Feuerring

Participants:

Key Points:

•    Indigenous Sustainability Festival – St Kilda

•    WTC – could seek to tie in sustainability messages

•    Strength of faith communities – focus on volunteering

•    Linking sustainability messages into religion

•    Reconnecting the 1st generation back to their culture

•    Issues with engaging older people from CALD more likely to engage

•    Issues with translating into languages and whether 1st generation use their language to extend their ……..  ?? / older generation

•    Difference between tailoring messages v’s generic communications

•    Difference between how you approach sustainability for different groups

•    Accessing community leaders from CALD groups to inform programs and provide advice on messages = these community leaders are generally happy to be active and are already doing this

•    Remembering that CALD groups are diverse within themselves, not all of one community has same priorities

•    Visiting community groups and going to them and where they are already, meeting = connect with existing networks.  Outreach is vital!  “Don’t ask communities to come to you, go to them”.

•    Talking stick / message stick = idea of having conversation and that being passed onto others

•    Issues:  involving diverse groups

•    Need for a process of enquiry = we should be teaching each other

•    Mainstream needs to learn and embark on a process of enquiry and provide space for everyone to be teachers

•    1st generation creates community then the 2nd generation lives bi-culturally

•    2nd generation may have more choices because of their ability to live bi-culturally

•    The knowledge of the 1st generation in creating community is something that should be learn by mainstream

•    Existing networks spread the message much quicker than recreating new networks = building on existing networks / structures is undervalued

•    Ethics of asking people to do something that you will step away from = how do you go in and work with a community as a paid person and then ask others to do it voluntarily

•    How do we turn a specific non-mainstream message into mainstream and why do we need to do it?

•    Why do we focus on difference?

•    What has happened in our lives that was difficult and how we respond to it?

•    People from other cultures come with experience and adversity that brings insight and resilience

•    Vulnerability brings strength

•    Last 15 years = problem with identity / politics

SHOW  ME  THE  CHANGE
Complexity and the Art of Evaluation – Reporting Sheet

Leader:    Stephanie Madeley

Participants: Danielle Kennedy    Victoria Johnson
Merryl Whyte    Simon Kreeure
Kerry Edgecombe    Jon Wapainakefe
Felicity Wardlaw    Viv Beufon
Karen Parissien

Key Points:

Inability, disinterest, not usually have a correlation – its more that (low socio-economic) groups want to change but there are too many barrier to change.

Access – go out, don’t expect them to com to you.  Organisational flexibility is very important and ask them what they want.  Community often has the answer.  The role of schools in a community can be profound.  This is part of the importance of relationships.

Should we place greater importance on social change at a societal level?  Would this make things easier?

What is the relevance to people (eg economic hardship)?