Realist Evaluation Methodology in the Research Consulting

In September 2024, Kohlrabi took a bold step by applying for an evaluation contract that required the use of realist evaluation methodology – a sophisticated approach more commonly seen in academic settings. Realist evaluation is a theory-driven method that goes beyond simply asking whether a programme worked, instead exploring how, why, for whom, and under what conditions it was effective.

As the lead on this evaluation, I knew we would need to be creative. Research consulting often demands speed, clarity, and outputs aligned with client needs. Our challenge was to preserve the depth of realist evaluation while adapting it to the fast-paced, impact-driven world of non-profit consulting. The project also incorporated elements of both process and impact evaluation, adding layers of complexity, but also opportunity.

What is Realist Evaluation?

For those unfamiliar with it, realist methodology was introduced by Pawson and Tilley in the 1990s. It is based on the premise that programmes succeed – or fail – due to the interaction between context (C), mechanisms (M), and outcomes (O), commonly known as the CMO configuration. Rather than offering blanket conclusions, realist evaluation generates 'middle-range theories': evidence-based insights into how outcomes are achieved under specific conditions (Pawson & Tilley, 1997; Westhorp, 2014). This makes it particularly valuable for evaluating complex social programmes where multiple factors interact.

What worked?

Clients value consultants who combine the rigour of academic research with the agility of applied consulting. By proposing a realist-informed approach, we demonstrated our commitment to both depth and relevance – an approach that strengthened our bid and contributed to us winning the project. This methodology also enabled us to explore the programme’s outcomes at a deeper level, offering richer and more nuanced answers to the evaluation questions than a more linear method might have allowed.

Importantly, we had the space to be creative. We employed techniques, such as the Teacher–Learner Realist Circles, which is a dialogic approach that helped surface how different stakeholders understood the programme’s mechanisms. These methods not only enriched our findings but also introduced a refreshing, participatory dimension to the evaluation process.

What Was Hard?

Realist evaluation is time-intensive. Its layered data collection and analysis methods often require extended engagement, which can be at odds with the fast-paced timelines of consulting projects. In our case, we had to adapt the methodology to fit within the project’s time and budget constraints without losing its core benefits.

Additionally, realist evaluations often produce theories, not bullet points. Clients, however, usually prefer concise, accessible insights that can inform immediate decision-making or be shared with funders and the public. Translating middle-range theories into digestible, actionable recommendations became a challenge. Some of our more sophisticated insights had to be simplified or dropped entirely from final deliverables.

What We Recommend

While we are unable to share the project’s specific details, here are some general principles we learned about applying realist evaluation in non-profit consulting:

• Be strategic and selective. Don't try to realist-evaluate everything. Identify the specific evaluation questions or programme components where realist methods will add the most value. Use it specifically, not universally.

• Talk to your client early. Clarify how they want the findings presented and what kind of results will be most useful. This will help you avoid spending time on theory-building that may never make it into the final report.

• Translate complexity. Develop the ability to explain complex ideas simply. Clients don't always need the full CMO breakdown they need to understand what’s working, why it matters, and what they should do next.

• Balance theory with practicality. Realist evaluation is a powerful tool, but it needs to be applied with an understanding of real-world constraints. If time is short, consider a realist-informed rather than fully realist approach.

References

• Pawson, R., & Tilley, N. (1997). Realistic Evaluation. SAGE Publications.

• Westhorp, G. (2014). Realist Impact Evaluation: An Introduction. Overseas Development Institute. https://odi.org