Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Leader: Keren Winterford

Key Points:

We talked about some limits or challenges for learning:
•    No support time for evaluation
•    Evaluation at the end (“tacked on the end”)
•    Busyness
•    Managers not being involved
•    Often strategy/planning (higher level) org processes, different to true frames of projects: learning can’t fit into higher level org objectives – learning can’t influence this level
•    Reporting (formal) eg; Quarterly reports – people don’t like writing them (“creative writing”) and people reading don’t get what they need/want
•    Question: evaluation needs to be able to change projects, (how to do this?)
•    Often evaluators have different language and hard to connect with
•    Cycles of funding
•    Educate managers of the importance of learning

We talked about some great ways to support learning:
•    Action learning
•    Structured time for reflection in project management
•    Lessons learned methodology
•    Using case studies (tight, structured formats) and peer reviews
•    Structured process for doing something of ‘evaluation learning’ (ie. Process of a response and changed practice as a result of evaluation)
•    Engagement of managers in reflection and learning process
•    Having workshop/reflection meetings (everyone in the room)
•    Practice continuous change
•    Feedback loop to senior management
•    Focus learning processes not just on participants (beneficiaries) but on practitioners
•    Trust important to share the full story (including failures)
•    Right balance of measurement and learning (we need both) – make measurement stuff easy and quicker so we can focus on learning
•    There is intuition in learning- ‘trust ourselves’
•    Monitoring is a part of the process
•    Establish process and structure of evaluation from the start
•    Need flexibility for learning
•    Partnerships are becoming increasingly important and important for learning (design as part of)
•    Need to structure/enable staff to have self reflective learning process
•    There is a need to recognise interactive nature of projects and evaluation
•    Be more creative and innovative in “reporting”, ie; not a report, but also videos, pictures, workshops, opportunity for conversation (through workshops)
•    Use of extended evaluators, yes, not necessarily objective, but they offer a neutral, safe place for participants/stakeholders to really share
•    Recognise not an evaluation should be to change a project (next phase) and then to set up an evaluation (at the start of next phase) and then to know as part of the implementation the evaluation will then need to be revised as new knowing it gained, therefore, interactive process of doing and evaluating
•    Its complex and iteractive

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Participants: Scott McKenry (NCS Swinburne), Elle Morrell (MEFL), Beth Hyland, Candyce Presland, Maxine Schache, Diane Nichols, Cathy Wills (Swinburne)

Key Points:     (INSERT PICTURE)

How do we evaluate impact of training programs over time?

Funders and “deliverers” of programs need to have more conversations about long-term evaluations:
-    what do each need?
-    how can they connect?

What partnerships need to be formed? What relationships need to be credited?

Recognise that we need to move beyond evaluation sheets at end of training

If you want to involve participants in evaluations, you need to train them, get them excited by it as a practice. Makes them more amenable to being ‘evaluated’.

Some of us already doing this:
-    DPI (Farmer CD)
-    ACF (GreenHome)
-    NCS (Carbon Accounting ??)

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Leader: Jess Dart

Participants: Julia Davies, Sue Arendt, Scott McKenny, Maxine Schache, Carolyn Kabore, Lisa Goeman

Key Points:

•    Dangers:
1.    Idea is that some programs that are in the early stages of development, working out and trying different options – then predetermined indicators may ‘squash’ innovation. Evaluating a program a whole range of new ideas could stifle creativity, eg; picking winners and losers too early on
2.    Danger of mandating evaluation of everything – mindset that evaluation is critical and essential. But funnily enough, this is often done at the expense of ‘diluting’ evaluation money and getting very poor quality evaluation – not everything needs evaluating
3.    Focus evaluation on where learning can inform decisions – occur – multiple phase
4.    Is it worth evaluating when costs exceed benefits? Eg; calculating attributable impacts to one particular program may cost more than cost of program!
5.    Concept of evaluability assessment
6.    Focus evaluation on what you want to know
7.    Don’t do evaluation when you have nothing to learn from the answers of when you have no ideas of what you want to know. Don’t do evaluation when it is not worth it.
8.    Why do we collect data that is never looked at and never used and of poor quality?
9.    Potentially we could have ‘levels’ of evaluation – from very light to intensive. Focus evaluation resources where they can have most results
10.    Don’t evaluate a project when its been evaluated to death
11.    Need to always ask why we are evaluating – what is the purpose? Eg; when a project has really failed and everyone knows why it has not worked. What’s the point?
12.    Focus more on using evaluation results.

Show me the Change
Complexity and the Art of Evaluation – Reporting Sheet

Participants: (It ebbed and flowed – quite a few bumblebees!)

Key Points:

•    The workshop was about exploring “lessons from the Dancing Guy” (Derek Sivers – YouTube) and the wave theory (AKA diffusion of innovation). I opened discussion with 2 examples – sustainable events work and designing a pilot shopping centre sustainability project; and how I was trying to use the theory with these initiatives.
•    Key questions to the group:
1.    What occurs to you on watching the Derek Sivers YouTube clip?
2.    What does this theory raise for you re monitoring and evaluation?
•    Key points from discussion:
•    Need to understand what motivates the first followers (and the nuts)
•    Government could create spaces and places that foster the “nuts” and the “first followers”
•    The first followers – makes an action simple, easy and public
•    If you want to work with the first followers (as a change aspect) you need to create apps for them to find you
•    What happens if your champions (your nut/first follower) doesn’t want to go public? Need to understand why and evaluate this
•    The discussion turned to suggestions for government change programs to support the first followers and nuts
-    create standards
-    networking spaces
-    recognition programs
-    peer reviewing from industry
-    “on the job” training programs