Speakers:

Paul Clarke - Director, Oxford Change Management

Margie Buchanan-Smith - Independent Consultant

Maurice Herson - Co-editor, Forced Migration Review

Discussant:

James Darcy - Programme Director, Humanitarian Policy Group (HPG), ODI

Chair:

John Mitchell - Head, ALNAP

John Mitchell, chair of the event briefed the audience about the rationale and content of the annual Review of Humanitarian Action (RHA) publication. Main aim of the publication is to draw out key themes and issues related to system-wide performance in the humanitarian sector. The current RHA includes a synthesis of evaluations of Pakistan earthquake response, a Meta analysis of the joint evaluations and the first ever study on organisational change in the humanitarian sector. Presentations were made by co-authors of each chapter.

Maurice Herson

The study on Pakistan earthquake can be used as a proxy to judge the response of the humanitarian system to a crisis. The scale of the disaster demanded a comprehensive global response and a synthesis offers a policy level approach rather than adapt a narrow programme-wise analysis.

Herson identified the following as the main highlights of the study;

• A quality/time trade off is seen in conducting assessments.  Priority at the initial stages of a disaster response is biased towards fulfilling high needs, thereby resulting in low quality of assessments. However, the quality of assessments increases with time when demand of needs decline.  
• With regards to funds, a large proportion of donor money is channelled through the government administration. However, remittances by the Pakistan expatriate community are significant. Herson believes the later mechanism to be an effective mode of response as long as one can identify specific details of the flow of funds and their use.
• In terms of surge capacity, Pakistan response can be noted as an improvement in the humanitarian system. However, the ‘gain’ is Pakistan was a ‘loss’ elsewhere, thus echoing the fact that ‘humanitarianism is the management of dilemmas’.
• Although individual learning is in general far greater than institutional learning, rolling out the UN cluster approach in a humanitarian response for the first time provides a wealth of knowledge on coordination.
• With reference to ownership of the response effort, Herson highlighted that communities are not consulted systematically to obtain their views.

Who should judge the quality of the response?

• Herson said it’s more than merely asking the people affected by the crisis about the relief efforts. A balance between the view of the evaluations on which the synthesis is based, i.e the agency views looked at through their mandate and operations, and the perspective of the people who are affected by the disaster could be adapted to determine the quality of response. He argues for the use of the standard quality criteria accepted within the general humanitarian system.

Margie Buchanan Smith

ALNAP’s 6th meta-evaluation focuses on Joint Evaluations (JE). Smith considers it as an attempt to stand back and look at the characteristics of a whole range of evaluations against ALNAP’s quality pro-forma with the intention of improving the evaluation practice.

To understand the quality of this JE study, a comparison was made against a set of single agency evaluations (from 2001) analysed in a previous RHA.

According to Smith, JE in the humanitarian sector has in the past been the domain of the donors. However at present, it’s more frequently taken up by NGOs and UN organisations and the sample used for the RHA this year confirms it. The sample includes evaluations from the Tsunami Evaluation Coalition, Evaluation Capacity Building Project, Inter Agency Standing Committee and International Health Evaluation initiatives. 

A typology in terms of focus of the scope of evaluations and the type of collaborations between actors is used to categorise the JEs and Smith presents the findings according to the ten hypotheses developed in the study.

• JEs help to build trust and social capital. It is rarely stated as an objective at the outset of the JE planning process; only a positive by-product. More focus on the process itself will have direct bearing on the uptake of the findings.
• Involvement of national governments is weak. Smith notes it as a surprise finding especially given that most evaluations of the sample are on natural disasters, unlike conflict related emergencies where the government may be seen as part to the problem.
• JEs are done with more consultation with beneficiaries compared to single evaluations. But it is not clear how the communities are consulted. NGOs hold a better record than UN organisations in this aspect. JEs pay much more attention to the views of other stakeholders, i.e Agency and government staff.
• Rigorous methodologies are adopted in JEs although little methodological innovation is observed. Recurring issue is the challenge of data analysis; at times   due to inadequate skills or under utilisation of the data collected.
• A striking difference between individual evaluations and JEs is observed in terms of attention paid to international standards. JEs scored much higher compared to individual evaluations. 
• JEs are poor in addressing cross cutting issues; protection, advocacy and gender. Evaluators are uncertain about how to incorporate protection and advocacy issues in JE.  In terms of gender inequality, Smith comments on the lack of knowledge in incorporating it to evaluations although it’s a well researched area.
• With regards to using JE findings to move forward the thinking and practice of the sector, the results are mixed. On the positive side, IASC real time evaluations can be related to the UN humanitarian reform process. TEC reports however, come out as a missed opportunity for not situating their findings in the wider debate. If evaluation findings are to address policy issues, it should be included in the TORs and team should comprise of people with the correct skills, eg Policy analysts.

Finally Smith summarises a few learning points from the meta evaluations; Do not underestimate the time necessary to carry out a good evaluation. Time spent initially in getting all stakeholders together pays off well in the process. Smith also highlights the shortage of skilled evaluators. An evaluation team leader should not only possess good technical skills, but also have political and interpersonal skills.

In conclusion, Smith says the meta evaluation showed the overall quality of JEs to be better than single agency evaluations.


Paul Clarke

Clarke says the first step towards making change within an organisation is to understand the characteristics of the subject itself, i.e what is an organisation? Traditionally, organisations are compared to machines and the terminology used in both circumstances is similar, ie. units, inputs, outputs, reengineering etc. Hence leading to a very mechanistic view of an ‘organisation’.

• So can we change an organisation in a similar manner to machines?

He argues that organisations are a complex community of people, built on relationships involving motives as well as rational components. By extension, then effective change in these institutions comes down to change in human behaviour from within the organisation, where it is likely to encounter emotional responses and resistance to change by taking a holistic approach.

• The methods and tools for organisational change are largely private sector initiated; are they applicable to the Humanitarian Organisations (HO)?

Study shows that HOs are different. However, the association of organisations to a human community stands true in the case of HOs. The experiences from the private sector could be adapted and they should hypothetically work although use would be difficult.

Clarke suggests a three stage approach for organisation change in HOs. First create awareness of need (bring the organisation on board with the case for change) then plan (deciding on the nature of the change) and finally implement.

Clarke notes that at each of the three stages successful change will address emotional as well as the value element of the change, bring different parts of the organisation together, ensure high levels of communication within the organisation, be open to conflict and disagreement and provide good leadership during the transition period.

• How can sustainable change within HO be achieved?

People in the humanitarian community already are aware of the necessary changes; the question is about their application. Humanitarian workers come across many aspects of organisation change in terms of leadership, participation, encouraging dissent and disagreement during their work in the field, but often its not carried back to initiate change within their own systems. 

Clarke also points out that organisation change comes at a high price in terms of time, money and the confusion it initially creates within the organisation. Therefore, he makes five suggestions; Have a clear vision - think about difference of the organisation’s impact after the change. What would the organisation look like after the change (e.g use OECD DAC criteria) ?; Provide meaningful incentives which match the values upheld by the organisation and individuals; Follow specific leadership styles; Promote collective, system-wide organisation change and finally but most importantly start change within oneself.