Anglicare Victoria and MCHRI: Evaluating a Rapid Response program to prevent children entering care

The project is ongoing, due to conclude during 2020, so final results are not yet available. However a number of observations can be made about its design and conduct so far:

Evaluation begins from day one, and practitioners are a vital part of the research effort

It is important to de-mystify evaluation for practitioners and stress that simple data collection is useful, and any evaluation always beats no evaluation.

Practitioners in the Rapid Response project were involved in regular discussions which helped to build their skills, interest and confidence, allowed researchers to draw on their experience, and enabled sharing of learning as the project has progressed.

In this project, having an evaluation practice lead has been critical, with the researcher acting as a resource and critical friend.

Data precision is crucial

The project was very strict about including only examples where the family met strict criteria and the program had been delivered exactly to plan. This is important to allow clear conclusions to be drawn about effects and treatment to be replicated by others in future.

Relaxing the rules would have included more cases, but data transparency is more important. Each case excluded from analysis was mapped with the reason recorded (for example if the treatment period extended beyond the nominated four weeks).

The aim is to reliably capture influences and pre-conditions that result in the observed outcomes.

A matched comparison group proved difficult to achieve

A control group, while useful, proved not to be achievable in the context.

A fully randomised control trial could not be attempted as it would not be ethical to deliberately deny families potentially helpful interventions.

To create a ‘natural’ control group, the project tried to identify appropriate comparisons by asking workers to nominate cases that would have been eligible for Rapid Response but could not receive it because at the time there was no available capacity.  This approach did not work well when it came to the point, because of the urgency of time frames (removal of the child must be imminent before use of the program is considered) coupled with a long chain of approval for inclusion in the control group. By the time approval could be received it was usually too late to gather full data for those families that would have qualified.

Implementation is not a linear process

Continuous improvement and iterative learning aligns with practitioners’ aims. The central purpose of evaluation for most NGO staff is to answer the question ‘How can I be sure what I am doing is working? I want to make a real difference through my work.’

This means that many aspects of delivering the program will be tested, adapted and improved while the trial is progressing. A linear model of inputs, treatments, outputs and outcomes as represented at the start of the project is a simplified snapshot of what will and should actually occur.

“A good analogy is designing and building a new car engine. Throughout the process all the parts are subjected to constant testing, looking for how well they work and improvements that could make them better, so that the final engine is perfect.”

David Poynter, General Manager Business Development and Research Based Models, Anglicare Victoria

Evaluating outcomes is complex

The usefulness of the program needs to be considered in the context of the whole organisation (Anglicare’s operations) and the broader system of child protection.

For instance, it is possible that some positive characteristics of this project could usefully be applied in other programs being delivered in the organisation. Similarly, assessment of children’s levels of safety in the home undertaken as screening for entry to Rapid Response can provide critical information to the child protection system about those who most urgently need out of home placement.