18.104.22.168 Realistic Evaluation
Tilley and Pawson (1997) developed a model of theory driven evaluation called 'realistic evaluation' that was centred on finding not only what outcomes were produced from interventions but also 'how they are produced, and what is significant about the varying conditions in the which the interventions take place' (Tilley, 2000). Tilley has been critical of quasi-experimental models of evaluation and suggests they fail to effectively identify why interventions work differently across different contexts. Realistic evaluation seeks to find the contextual conditions that make interventions effective therefore developing lessons about how they produce outcomes to inform policy decisions. Tilley outlined three investigative areas that need to be addressed when evaluating the impact of an intervention within any given context.
Mechanism: what is it about a measure which may lead it to have a particular outcome in a given context?
Context: what conditions are needed for a measure to trigger mechanisms to produce particular outcomes patterns?
Outcomes pattern: what are the practical effects produced by causal mechanisms being triggered in a given context?
(Tilley, 1998: 145)
The model involves developing a 'context mechanism, outcome pattern configurations' (CMOCs) that allow a researcher to understand 'what works for whom in what circumstances' (Tilley, 2000). The generation of CMOCs occurs through consultation with relevant stakeholders responsible for implementing, operating and participating in interventions. The model of evaluation allows the researcher to understand what aspects of an intervention make it effective or ineffective and what contextual factors are needed to replicate the intervention in other areas. Interventions can work in a number of ways within one area and realistic evaluation provides a model to understand the effect being produced by the intervention and crucially for policy development how it can be consistently replicated. Tilley suggested that models need to be developed for replication of interventions and realistic evaluation provides the framework to develop appropriate models:
Realism provided the necessary ingredients for such a model: specification of the crucial contextual conditions for the intervention, the change-inducing mechanisms that will be triggered by the intervention, and the anticipated outcomes pattern that will be generated by triggering these mechanisms. This compromises a 'context mechanism outcome pattern configuration
(Tilley, 2000: 104)
One of the key strengths of realistic evaluation is the ability to take the lessons learnt from one evaluation and apply them across a range of different contexts. Once the CMOCs are developed they need to be tested and they produce very specific data requirements across the context, mechanisms and outcomes and collecting all the relevant data can be very resource intensive (Gill and Turbin, 1999).