Speakers:
Fred Carden – Director, Evaluation, International Development Research Centre
Annette Boaz - Lecturer in Translational Research, Division of Health and Social Care Research, Kings College London

Chair:
John Young - Director of Programmes, Research and Policy in Development Group, Overseas Development Institute

Fred Carden, started by introducing his new book, Knowledge to policy: Making the Most of Development Research.The book focuses on the impact of development research on public policy and decision-making. Based on a review of the literature and an evaluation of 23 research projects funded by Canada’s International Development Research Centre (IDRC), it shows how research can contribute to better governance. And in order to depict the influence of research on policy, he began his presentation by expanding on three categories that describe how research can affect policy.

·         Expanding Policy Capacities: Knowledge and research can have a positive impact on building the capacities of policy-makers in different contexts, enabling them to be better informed while making policies.

·         Broadening Policy Horizons: Expanding the scope of concepts covered that will invariably broaden the agenda by bringing in new ideas and knowledge previously not considered. Policy-makers will then have a much broader outlook and be encouraged to think outside the box to address problems.

·         Affecting Policy Regimes: Research can impact overarching policy regimes by opening up policy processes and operational procedures of the government.

Returning to the study, Carden explained two fundamental factors upon which research’s ability to translate effectively into policy hinges: context and contingencies.

One of his main arguments was that context matters, noting five probable scenarios where context played an important role, with examples of cases that fit each:

1.      Policy-maker Demand: In this scenario, there is adequate demand from the policy realm and policy-maker for research. Therefore, the researcher needs to be vigilant and ready to seize opportunities, making timing the most important variable.

a.       Examples: Micro Impacts of Macroeconomic and Adjustment Policies (MIMAP-S), Nepal Information and Communication Technologies (ICTs), Viet Nam, Acacia (South Africa, Mozambique, Senegal, Uganda)

2.      Policy-maker Interest; Leadership Gap: The policy environment is not really conducive for use of data and knowledge due to an institutional and leadership gap despite adequate interest from policy-makers. There is a need to build leadership and institutional infrastructure within this context in order to enable research to have an impact. Implementation problems also exist in this context.

a.       Tanzania Essential Health Interventions Project (TEHIP), MIMAP-B, MIMAP-P, Latin American Trade Network (LATN), G-24

3.      Policy-maker Interest; Resources Gap: Policy-makers are interested in research but there are insufficient resources in order for research to seep into policy.

a.       Ukraine

4.      Policy-maker Neutral, Research Interest: In this context, policy-makers are neutral to research, which makes it challenging for research to have a tangible influence. Communication of research is more important in this scenario given the lukewarm attitude towards the research community.

a.       High Altitude Mining, Eastern and Central Africa Programme for Agricultural Policy Analysis (ECAPAPA), Arsaal (local), Asian Fisheries Social Science Research Network (AFSSRN), Society for Research and Initiatives for Sustainable Technologies and Institutions (SRISTI), Jordan, Copper Mining

5.      Policy-maker Disinterest; Research Interest: Policy-makers are largely disinterested and even hostile towards research within this context. In this situation, policy influencing using evidence-based approaches can break down quickly. Patience and persistence is critical in order to bring salient issues to the table.

a.       Syria, Arsaal (national)

Beyond context, Carden also expressed the need to actively exploit contingencies that emerge, in order to ensure that research can have a role within policy processes. He breaks down the contingencies into five situations that arise in countries and acknowledges that these contingencies evolve, constantly presenting and preventing opportunities for research-based findings to play a role in policy processes.

1.      Stability of decision-making institutions: Stability of decision-making institutions is imperative. Rapid turnover of governments is not very conducive for research to translate into policy and for researchers to develop relationships with policy-makers. Continuity is important in this regard.

2.      Capacity of decision-makers to use research: It is important to educate and build capacity of policy-makers in order to enable them to engage effectively with research findings. When policy-makers are educated and well informed, policy processes are enhanced and there is a greater chance for research to have a positive impact.

3.      Decentralisation vs centralisation: Research can have an influence in highly centralised and decentralised systems of governance alike. The cases of Philippines (highly decentralised) and Vietnam (highly centralised) are elaborated to underscore that both systems are open to research if presented well. Aligning a project’s plan with prevailing political structures is more important.

4.      Transitional countries: Countries in political, economic and social transition remain sites for policy innovation and experimentation, thereby increasing the need and call for research.

5.      Economic pressures: Nations embroiled in economic crises/pressures are more open towards research since policy-makers are in dire need of ideas to turn the tide. Crises represent an opportunity for the policy research community.

In his conclusions, Carden emphasised that research can lead to better governance in three ways: by opening up new avenues of inquiry and debate, by empowering citizens with knowledge to hold their governments accountable, and by expanding the menu of policy options available for policy-makers as they seek to address salient public policy issues.

Annette Boaz’s presentation and comments focused on the methodological aspects of evaluating the impact of development research on policy outcomes. She expanded on the report, ‘Assessing the impact of research on policy: A review of the literature for a project on bridging research and policy through outcome evaluation’ commissioned by the UK Department for Food and Rural Affairs (DEFRA). The presentation focused on the importance of evaluating the impact of research on policy.

Dr Boaz first presented the guiding research question of her talk:

·         How do you (best) evaluate the impact of research programmes on policy?

She also highlighted relevant sub-questions, such as:

·         How do people evaluate the impact of research programmes?

·         What are the advantages and disadvantages of different approaches?

·         Which methods are most effective?

·         Which methods offer value for money?

·         Are there any specific issues to consider when evaluating the impact of research on environmental policy?

Mapping the Literature Review

The literature review primarily focused on evaluations of the impact of research on policy and emerged from a review of 156 studies. Dr. Boaz’s methodology to evaluate the influence of research efforts on policy covered a wide ranging literature from an array of countries – United States, UK, Canada, European Union, etc. The literature also emanated from a range of disciplines from Evaluation Studies, Health, International Development, Environment, Sciences, Utilisation literature, etc. Of the studies reviewed, a majority of them focused on the relationship between research and policy and included specific evaluations and reflections on the methods used to evaluate the impact of research on policy. And finally, Boaz also referred to several conceptual frameworks to study the evaluative impact -- specifically, RAPID Outcome Assessment, the HERG Payback Model and economic analysis are often applied in empirical studies.

Methodological Approaches Utilised

Returning to the driving question, “How do people evaluate the impact of research programmes?” Boaz alluded to a wide range of methods and approaches:

·         Qualitative methods: semi-structured interviews, documentary analysis, field visits and observations

·         Quantitative methods: surveys, bibliometrics and patent/new technology tracking

·         Panels and peer review

·         Workshops and focus groups

·         Process tracking

Is there an ideal research method?

In terms of the advantages and disadvantages of using a set of approaches, Boaz mentioned that it depended on the context where the methods are being employed,r and that it usually best for researchers to use a mixed-method approach to assess evaluative approaches.

Which methods offer the best value for money?

Analysing the various methods utilised, Boaz said that is difficult to pinpoint a method that can be considered best value for money since there is very little data present to underscore the cost effectiveness underpinning these methodological approaches. And in many cases, the costs of conducting these studies are not reported at all. Approaches that are deemed to be more cost effective seldom yield relevant insights into the process of research utilisation. Telephone surveys might present a cost-effective ‘middle ground’ between postal surveys and in-depth interviews.

Key Questions While Designing Research Impact Evaluations?

Boaz concluded by elaborating on a list of questions that researchers need to keep in mind as they engage in evaluating the impact of research. They are:

1.      What is your conceptual framework?

2.      What are the outcomes of interest?

3.      What methods will best explore the outcomes of interest?

4.      How do you address attribution?

5.      What is the direction of travel for the evaluation?

6.      Is this a mixed method approach, providing scope for triangulation?

7.      Will the methods selected capture context and the complexity?

8.      When might be the best time to conduct the evaluation?

Group Discussion

Comments and questions raised in the discussion included:

·         Can researchers double up as effective communicators of their work? How can they deliver in this regard? Does this research-policy-maker conceptualisation represent an old fashioned understanding?

·         What types of research methodologies offer the best results? Qualitative or Quantitative Methods?

Carden mentioned that researchers can be multifaceted and possess the ability to become effective conveyors of their research findings. He added that researchers have a genuine interest in identifying the results of their research, thus elevating the importance of communication.

He mentioned an example from the mining sector Peru to illustrate that researchers can wear multiple hats at the same time. The Peruvian case involved researchers that are actively collaborating with other stakeholders to ensure that their research is being put to good use. Boaz also acknowledged the importance of communication and for researchers to become more adept at relaying the fruits of their research. But communication is also contingent on the structural context and the prevailing culture since it would determine whether a transfer of ideas does occur. A culture of research is imperative. In addition, she also referred to the importance of knowledge management systems in this day and age to enable effective communication of research to happen. She advocated for a greater advocacy role for researchers since they represent conduits of knowledge present to enhance policy-making.

Moving onto the methodological queries, Boaz pointed out that a majority of the methods in use are qualitative in nature but there are instances such as in medical research where quantitative methods are employed. She also stated that researchers are employing both methods in a complementary manner. And bibliometrics is attaining more importance as a method. Carden viewed this question not as a methodological one but as a ‘data one’, and the purpose and intent of the research is more important and should dictate the choice of method researchers employ. The objective and what is being studied is more important in this regard.

After both speakers completed their Q&A portions, John Young summed up the session by stating three important takeaway points from the content covered:

·         Intent and clarity driving the research is important

·         Context matters and determines the scope and ability of research to impact policy

·         An overarching framework is preferred to a specific or singular method in order to maximise the impact of research on policy