Case study one: the DFID–ESRC Growth Research Programme
Why use ROMA?
The DFID–ESRC Growth Research Programme (DEGRP) funds world-class scientific research on inclusive growth and related issues in low-income countries. ODI leads the Evidence and Policy Group (EPG) which supports DEGRP and its diverse projects to maximise the profile, uptake and impact of the programme’s research. All the projects have to develop an impact plan as part of their bid preparation and it was with this in mind that ODI decided to write a short guide on impact – a synthesised version of ROMA specifically targeted towards academic researchers to help the project teams.
The DEGRP programme funds a wide range of research on economic growth in low-income countries covering four themes: finance, innovation, agriculture, and the relationship between China and Africa. Many of the Principal Investigators are renowned researchers with good links to national and international policy-makers. The impact guide needed to be relevant to a wide variety of topics and to researchers with a good deal of experience.
How was ROMA applied?
‘Achieving Policy Impact’ was published in 2013. Based on Developing your engagement strategy, the guide explores several key areas: understanding the different types of impact that academic research could achieve; mapping stakeholders; considering what change is likely; and knowledge brokering. Academic researchers are more likely to be familiar with the concept of communicating their research than with the concept of achieving impact - though the emphasis put on impact by the UK’s Research Excellence Framework in 2014 is gradually changing this.
The guide started with the assumption that achieving impact depends on having a good communications strategy. Good communication depends on understanding what sort of brokering roles the project could fulfil, which in turn depends on developing a realistic theory of change.
A realist theory of change depends on having a comprehensive stakeholder map. The guide reverses this logic to set out four steps: (1) an AIIM matrix, (2) an outcome mapping-based theory of change, (3) brokering roles, and (4) research communications.
Because all projects had developed impact plans as part of their programme plan, the DEGRP impact guidance was not prescriptive. The guidance was intended to help projects refine their understanding of how impact could be achieved, and make their approaches more appropriate to the specific needs of their stakeholders.
The guidance has been used by projects researching very different issues such as irrigation in Tanzania, risk perceptions in Uganda, financial sector development and supply chain management in sub-Saharan Africa and improving productivity in Bangladesh’s garment sector.
It has also been well received in a number of workshops, and to a wider audience of grant holders from the DFID-ESRC Poverty Alleviation Programme. Projects are given the guidance as they begin, but are not expected to refine their pathways to impact until a year later, to give them time to get into the field and get to know their real operating constraints.
The guidance is freely available on the website to others outside of DEGRP and is the most downloaded publication to date on the site.
How was ROMA adapted to the project context?
ROMA principles have been used in different ways. A review of the revised pathways to impact showed that in the main the guidance had been used to make projects’ anticipated achievements a good deal more realistic than what was set out in their initial project plans. One project that had drawn up a very detailed impact plan used the guidance to develop a theory of change to do some detailed differentiation of the needs of individual stakeholders – from procurement agencies needing more reliable and more suitable supplies to policy-makers wanting to find more scope for innovation and better supply chains.
Another used the theory of change to nail down some very specific changes they would expect, like and love to see among both public policy-makers and private sector collaborators. Gratifyingly, by the end of the project the latter project had achieved its ‘like to see’ indicator and developed concrete ideas about how to achieve the changes the project team would ‘love to see’.
The guidance had a major effect on one project, causing it to completely rethink its approach to impact. The PI was initially openly sceptical about the need for academic research to consider issues of impact but having read the guidance, he recognised that the project did not understand enough about the agricultural policy-making process in Uganda to really know whether the recommendations they were developing from their research were locally and nationally relevant. Once initial research recommendations were finalised the team decided to embark on an extensive and highly structured stakeholder engagement exercise to turn them into policy recommendations.
There were four stages to this: (1) small group discussions with the project’s direct collaborators, (2) larger workshops with district extensionists and local government officials, and (3) consultations with 90 national policy-makers, private sector representatives and development partners. At each stage the project’s recommendations were adapted and refined - the final stage (4) was a large multi-stakeholder workshop in Kampala which used group work to look at the wording of specific policy recommendations and discuss what might be needed for them to be implemented.
The project team became so enthused by what they had learned from the ROMA-based approach that they decided to hire ‘policy brokers’ to help them ensure that the messages from the research continue to be heard within agricultural policy processes in Uganda.
Key to the success of the guidance has been that it is an approach rather than a template, and that the ROMA tools can be used in any order. The guidance placed less emphasis on Diagnosing the problem of ROMA or From M&E to monitoring and learning as the context for the research was set out in detail in the proposal documents and there is no requirement for projects to monitor their impact in detail.
However, the DEGRP EPG is working to develop a framework for monitoring programme-level impact for this very diverse portfolio of research.
What was achieved or learned by using ROMA?
The DEGRP example shows that ROMA is highly adaptable, being used by projects in a variety of settings. The fact that it is not necessary to know and use ROMA systematically from start to finish means that a small number of ROMA’s overall toolkit can be selected and presented in a non-prescriptive way. This helps ensure that the overall approach remains relevant to the needs and concerns of academic researchers who previously have not had to think about impact in this way and who might find the full ROMA toolkit overwhelming.
Case study two: a toolkit for employer and business organisations
Why use ROMA?
In 2015, RAPID worked with the Bureau for Employers’ Activities (ACTEMP) of the International Labour office (ILO) to help develop a toolkit for employer and business organisations (EBMOs) to monitor and evaluate their policy engagement work. The project was part of ACTEMP’s overall commitment to strengthen EBMO’s policy capacity to promote an enabling environment for sustainable enterprises. It was funded under the ILO-Sweden Partnership Agreement for 2014-2017 (ILO Programme and Budget for 2014-15).
EBMOs play an important role in generating employment and improving living standards within a society. They help create the conditions necessary for successful enterprise by influencing the policy environment in which they do business.
Through ILO capacity development activities (such as the Enabling Environment for Sustainable Enterprises (EESE) toolkit), EBMOs in Africa, Asia and Latin America have been improving their planning and have reported advances in their capacity to influence policy. However, they have limited capacity to monitor and measure this contribution in their respective countries. They currently only capture media monitoring data and anecdotal stories of change.
Therefore, the project involved developing an additional toolkit for EMBO and ILO staff. The purpose was to introduce a systematic and strategic approach to planning, monitoring and learning from EBMO policy engagement work, and to support staff to use the toolkit. ROMA provided the structure and staged approach, by providing tools at different stages of the policy cycle.
How was ROMA used?
RAPID sought to adapt and tailor the ROMA tools to best meet the needs of the EBMOs. This involved working closely with the organisations to:
- understand their strengths and weaknesses;
- draw on existing expertise and work, rather than brining in entirely new approaches;
- maximise peer support and face-to-face engagement.
A series of workshops and meetings with ACTEMP staff and EBMOs were used to test ROMA tools, and revise them, effectively co-producing the adapted toolkit. RAPID staff presented the approach, by taking the participants through the key steps, tools and exercises. Available policy documents and plans from the EBMOs were used as working resources to tailor the sessions to their needs.
After each workshop the toolkit was revised, using formal and informal feedback to incorporate EBMO experiences into the tool’s examples. This also allowed templates to be developed to guide users in recording key information.
What was achieved and learned by using ROMA?
The application of ROMA tools helped participating EBMOs to think through the roles and skills they need in their organisations to achieve their policy influencing objectives, and how to monitor these achievements. Using existing strategic plans, EBMOs were supported to incorporate monitoring and learning into their work.
Key lessons noted by the EBMOs, ACTEMP, ILO and RAPID were:
- Funding and capacity: The challenge of influencing policy where there is limited staff and funding was a common constraint. Making internal adjustments in response to capacity gaps and high staff turnover was recognised as important. For example, recruiting people or changing job descriptions where there are resource shortfalls. This requires continual awareness of this issue, as well as the EBMO being able to bring in external support when necessary, such as technical assistance to carry out a specific piece of research.
- Policy engagement and planning: A willingness to be flexible, to proactively approach EBMO members for funding, and communicate any policy engagement results were recognised as crucial steps to overcoming constraints.
- Communication: The application of ROMA highlighted the importance of effective communication, particularly being proactive and involving member organisations at all stages. For example, having EBMO members chair task forces encouraged them to engage in the work of the EBMO.
- Co-production and tailoring: Working together to co-produce the toolkit allowed it to be tailored, prioritising aspects that are most important and incorporating relevant examples.
- Audience challenges: Developing the toolkit involved the challenge of dual audiences as it was intended for use by both ACTEMP staff and EBMOs. Ideally, it should have been aimed chiefly at EBMOs and a separate trainers’ guide developed for ACTEMP ILO staff to guide them in delivering the toolkit training and supporting the EBMOs.
Case study three: Save the Children UK
Why use ROMA?
Measuring political will is challenging and tactics for influencing change need to be flexible. In 2016, Save the Children UK engaged ODI’s Research and Policy in Development (RAPID) Programme to help the organisation measure its contribution to generating political change for the benefit of children around the world. The project involved designing a tool for its country offices to monitor the impact of their advocacy activities. This resulted in the development of the political will monitoring tool. The tool determines whether there is increased motivation among stakeholders to take decisions or pursue activities that support the organisation’s programme objectives.
ROMA tools were adapted, and the principles of Outcome Mapping were used, to develop a systematic approach to measure how advocacy efforts contribute to behaviour changes. ROMA encourages users to use the tools like a compass, rather than a map; guiding practitioners through challenges and allowing space to navigate and react, rather than being too prescriptive. In this way, science and art work together.
The tool comprises:
- a guidance note, outlining a step-by-step approach to how to use it;
- the tool presented as an Excel file.
How was ROMA applied?
RAPID tailored two of ROMA’s tools, in response to the needs of Save the Children:
The Alignment Interest and Influence Matrix (AIIM) – a stakeholder engagement tool;
Progress markers – a graduated set of statements describing changed behaviours, thoughts, feelings or relationships of an actor over time. They articulate the complexity of the change process and tend to be more informative than a single indicator.
AIIM was used to identify and prioritise three key stakeholder groups. The team carried out a mapping (considering how aligned each stakeholder was with the policy objectives, how interested they were in the policy issue and how much influence they had upon achieving the desired policy change), to help better understand and determine how best to work with them.
Progress markers were used to monitor how the behaviour of the three priority stakeholders changed over time. They were modified to reflect how the organisation’s influence on political will is linked to the extent to which stakeholders ‘understand’, ‘support’ and ‘engage with’ the issue over time. For example, you can judge if you have been successful in influencing a policy-maker’s perception by exploring how they act in relation to, or talk about the issue.
Excel templates were used to record the data, which linked to an activity log and results template. The templates facilitated the reflection stage, on which guidance was also provided.
The following principles and approaches drove the development of the tool methodology:
- Embedded: The project team aimed to understand the needs of the relevant Save the Children country offices to ensure the tool was appropriate, rather than bringing in entirely new approaches. It also sought to minimise the burden for country office staff, to allow learning from what had previously worked and to facilitate uptake of the tool.
- Participatory: The Save the Children UK team played a facilitating role, ensuring that processes were guided by the needs of their staff in the country offices, and that practical exercises were participatory.
- Simple and innovative methods: this allows for the easy collection of data without necessitating a large investment in data collection resources.
- Recognising complexity: based on RAPID’s best practices, the project explored plausible contributory links between interventions and behavioural changes.
How was ROMA adapted to the project context?
The first phase of the work involved a review of existing tools used by Save the Children and discussions to understand the needs and challenges faced by the country office teams. A pilot version of the tool was designed and then tested in ODI-led workshops with teams in Nigeria, Ethiopia and Indonesia. The tool was then revised, incorporating country office team experiences into practical examples. Templates were also specifically developed to guide the tool’s users in recording key information. The revised tool continues to be piloted ahead of its full roll out.
Developing progress markers to adequately measure behaviour change required the most time when working with the country office teams. As progress markers can reflect nuanced and complex changes in behaviour, developing concise markers took several iterations. Reflecting upon the work of different groups in the plenary sessions was an effective way to gather feedback and improve the progress markers.
What was achieved or learned by using ROMA?
ROMA provides a systematic approach to monitoring advocacy efforts. When linked to budgetary information it supports an assessment of cost effectiveness. The approach is iterative; therefore, the reflection step is emphasised as the most important stage and is required to fully understand the results. In response, changes to strategies can be noted and made for the next planning and reporting period. Therefore, reflection is where learning takes place.
The successful application of the tool will depend upon the motivation of the country offices to systematically apply the approach and to integrate it in their routine practices. It presents a substantial change in practice for the advocacy community.
The tool demands practitioners to be more intentional with their advocacy by documenting their actions and observed behaviour change. This requires advocacy to be implemented in a systematic manner. As advocacy is a heavily relational activity that relies upon people talking to each other, most advocates are not used to systematically recording their activities and linking them to observed changes. Instead they may retroactively find evidence to demonstrate contribution ex post, which increases the risk of bias (see Buffardi, Hearn and Tilley; 2017 for further reflections on assessing advocacy in practice).