Discussants:

Louise Shaxson, Delta/DEFRA

Enrique Mendizabal, ODI

Lines of Argument

Louise Shaxson talked about her experience working with DEFRA to develop and implement a tool called Line of Argument to help incorporate evidence into the environmental policy making process.

The following is a summary of her presentation:

• Situating Line of Argument in the policy making process
  • Evidence is key to the policy development stage, where policy makers need to take time to scope the parameters of an evidence base.
• The question then became how to organise and situate evidence to aid policy development recognising there are various types of evidence and different stakeholder interests.
• The goal, she argued, is to engage with stakeholders in a structured way to establish a ‘line of argument’ between:
  • their particular goal definition
  • the values inherent in how they define the goal
  • the evidence they believe will validate their conviction that this is the path policy should take
• The Line of Argument approach asks stakeholders (usually at a workshop) to go through the following steps:
  • Stakeholders answer the following questions in this order:
    • Why is this issue important?
    • Why is change happening?
    • Why do we need to intervene in the change process (to change the rate or scale of change)?
    • Why should Government intervene in the change process?
  • Stakeholders are then asked to summarise these responses into:
    • Why do we need a policy on this issue?
  • These responses are then used to underpin the question:
    • What evidence do we need to collect to inform policy development on this issue?
  • She then cited an example from DEFRA which allowed them to evaluate the process, concluding that it is a cost-effective yet powerful method of scoping the evidence base for policy way
• Workshop participants then tried the technique themselves structured around a specific example.

During the comments and feedback section, some of the issues that were raised and explained include:

• The fact that Line of Argument is a tool that is not about making judgements, but rather about creating a map. It is the policy maker’s job to evaluate the lines of argument and choose which one(s) to pursue.
• Louise explained that it was not a goal-setting process, but rather an opportunity to establish what evidence was needed in support of these goals.
• It was noted that the full process requires heavy facilitation, but also suggestions that the ‘five whys’ could be used on their own as an aid to evaluation planning, for example.
• There was a suggestion that re-grouping of participants might create more lines of argument and could be helpful if time permitted.
• In Louise’s experience, when looking at the evidence that policy makers thought they needed before the workshop, and comparing that with the outcomes of the workshop she was able to see where the big holes were in terms of where more evidence was required.

Outcome Mapping

Enrique Mendizabal then presented on a technique that was originally developed at the IDRC but that is commonly used and promoted at ODI, outcome mapping. Key points from his presentation include:

• Outcome mapping is not an M&E tool, it’s a planning, monitoring, learning, evaluation methodology
• In the three key stages of the project cycle: planning, implementation, and evaluation, there is a lot of focus given to the former and the latter, but systematic learning during the implementation stage is often overlooked. Outcome mapping was designed to fill that gap, but it can be helpful in all three stages
• In the planning stage, outcome mapping suggests the following techniques:
  • Establish a strong vision and mission
  • Recognise the “spheres of influence” and target boundary partners with whom it is possible to work directly and who can then use their influence to reach the intended beneficiaries of the work
  • Define progress markers, though be open to the idea that they may change as the project evolves
  • Create strategy maps outlining how change might be achieved
• During the implementation stage:
  • The focus is on behaviour change and particularly on the point where a programme’s influence is surpassed by community capacity and ownership
  • Because change is complex, non-linear, not controllable, and multidirectional monitoring of this stage is very difficult. Outcome mapping suggests tools like outcome journals, strategy journals, and performance journals to monitor change
• During the evaluation stage:
  • Refocuses evaluation from monitoring “impact” to monitoring “learning” especially with regards to boundary partners.
• Enrique then elaborated on how outcome mapping is actually practiced citing several examples
  • It was noted that as a strategy, it is often best used stealthily and that it tends to work best in downstream projects
  • He also noted several concerns with the strategy including time restraints (and therefore the necessity of prioritisation) , that it requires changes in upward accountability

During the comments and feedback section, some of the issues that were raised and explained include:

• A concern about the timescale. It was noted that change often takes much longer than one project cycle. Enrique responded by saying that in the planning stages, a theory of change should be established that it explicitly goes beyond the life of a project and in the evaluation stage scope for follow up should be addressed.
• How does this model take into account non-linear progress markers?
  • Outcome mapping is a very fluid procedure that should outline a theory of change but should recognise that the steps might not be achieved in the order predicted.
• It was noted that the procedure is not very facilitation heavy but changing the mindset of “higher ups” can be a difficult task.
• What happens when you do it badly?
  • One big critique of the outcome mapping process is a fundamental question of why we should be trying to change someone else’s behaviour in the first place. If done badly, this can greatly exacerbate this problem.
• How do you persuade people to invest time and effort into progress markers?
  • It’s not just about contributing resources, it’s about painting a more complex picture and still can quantify things
  • Breaking down step by step helps set realistic targets! Also, if you miss a target, you or your organisation will get blamed, but if you have the evidence to support why you missed the target then it might not be as bad.
• A lot of resources are required for outcome mapping and it’s a complex process, therefore it might be difficult to apply to very large projects. However, it might help to see what key changes might be required and focus on those.