Event Reflection: Establishing a whole of organisation Monitoring and Evaluation system – Mission Australia (AES Seminar session)

Speakers: Rachel Christie (National Manager, MA Centre for Evidence and Insights) and Cherie Pereth (Impact Measurement and Evaluation Manager),Mission Australia

Date and time: Monday 22nd August 2022 5:30 – 7:00pm AEST

Format: Recording available below

Overview: This seminar provided a comprehensive overview of Mission Australia’s journey over 5-years to establish an integrated approach to evidence, insights and learning across their our national organisation operating 465 services that serves over 150,000 people a year.

The session provided two presentations including:

• Rachel Christie, National Manager, Centre for Evidence and Insights – Building a foundation for organisation monitoring and evaluation learning system in Mission Australia – An overview of the key steps taken by Mission Australia in establishing their monitoring, evaluation and improvement system over the last 5 years.
• Cherie Pereth, Impact Measurement and Evaluation Manager – Monitoring, Evaluation and Learning Approach within Mission Australia Homelessness services – A more detailed discussion on the way in which this approach has been used within their Homelessness services to explore what’s working well, what’s not working well, and draw on this data to support the best possible services delivery.

Both speakers outlined the challenges they have faced to operate at scale, the resources and operating models used, and some of the change journey for our staff and leaders.

Overview

Mission Australia (MA) is a large organisation operating across Australia, in rural, remote and metro areas. It has over 2300 employees supporting over 167000 individuals.

MA has been on a long journey to establish its whole of organization Monitoring, Evaluation and Learning (MEL) approach. This involved a series of key steps:

• establishing and rolling out the Impact Measurement program which we gathered like-services together to design a client outcomes pre and post survey tool using a program logic which gave them an idea about outcomes indicators (2017-2019).
• Undertaking a major technology transformation and examining funder requirements and service minimum data sets and what was good practice (2020-2021).
• Now, it’s about the MELs and using all the data we have to better understand the context of services and for different cohorts. Establishing the capability to understand the importance of data and then to get data in (i.e. systems and processes). This has laid the foundation for the MELs, which are all about bringing together, synthesizing and making sense of multiple data sources, and continuing to engage with services at all stages of the process to understand. This is done through the development of our Flagship Service Models and also the MEL cycle process.

The Monitoring, Evaluation and Learning cycle includes:

• A cross services theory of change is the first step, outlining the inputs and core service activities of like services and critical success factors and outcomes. (The example provided referred to developing a framework for 64 homelessness services)
• Next, monitoring questions and signs of progress are used to assess implementation and outcomes. Signs of progress are a user friendly way of understanding how successful the service is.
• Periodic, in-depth analysis to monitor how a service is tracking. Merit, worth and value.
• You can’t conduct a good evaluation without good monitoring data
• Evaluations can be conducted internally or with an external provider

Learning – Use the data to learn and improve

• Mission Australia have piloted ‘Evidence to Action’ cycles. A high level embedded dashboard in sharepoint where people will have easy access, and a more detailed dashboard for program managers.
• Developed in consultation with program managers to identify the most useful data to display on the dashboard.
• Share data with the broader sector, and local data packs to share insights with local communities.
• Mission Australia will be rolling out ‘Evidence to Action’ workshops and reflective practice workshops to services to support capability in understanding data.

Key messages

Six key enablers or foundational pieces for building an organisation M&E system were identified:

• A tight outcome strategy. Focussed and definitive, which sets the tone for everything and weaves its way through everything else. ‘What kind of change do we want to see in the people we serve?’
• Establishment of a multi-disciplinary Centre for Evidence & Insights. This was a national group to inspire curiosity for evidence that leads to learning and action. This team ensured data collection was as easy and automated as possible, to be as useful as possible and demonstrate value.
• Foundational service categories and flagship service models. Each type of service has a theory of change, client journey maps and monitoring, evaluation and learning plans amongst other critical pieces of work to underpin the service.
• Client data collection design and systems. This is where effort, time and resources are prioritised. This is built into core and ‘segmented’ minimum datasets.
• Organisational approach to Impact Measurement. Using a proven methodology, cross organisational validated tools and service specific measures. The implementation was staged for scale with an automated ‘pre/post’ survey built into the Customer Relational Management system. Staff engagement was built through the appointment of champions to support implementation.
• Data infrastructure and reporting capabilities. ‘You’ve got to make friends with your IT team’ to understand how the data is housed and how it flows. Understanding this supports automation of data reporting for funders, and:
• Internal data reporting. The CRM allows for the data to be democratised and used throughout service provision

Tips in a nutshell: 

• Get leadership buy-in through strong links to your organisational outcome strategy
• Build an eco-system for sustainability and automation to support collective impact.
• Plan a staged roadmap to avoid burnout. This is a marathon, not a sprint. Look ahead for this year, 3 years ahead, 5 years.
• Nurture a multi-disciplinary, learning mindset. The focus in the MEL Is the L- ‘learning’.

Additional resources to take a look at –

See the Session’s recording here and powerpoint slides here

This session include a great Q&A session. Notes from the Q&A can be found here.

Other OPEN Forums on embedding evidence into practice can be found here and here