Speaker:
Jonathon Beynon – European Comission, DG Development

Stephen Lister - Principal Consultant, Mokoro
Oliver Morrissey - Professor of Development Economics, School of Economics, University of Nottingham
Paul Mosley - Professor of Economics, Department of Economics, The University of Sheffield
Jasmine Burnley - Policy Adviser, Development Finance and Public Services, Oxfam GB

Chair:
Marcus Manuel – Head of Centre for Aid and Public Expenditure, ODI

The chair, MarcusManuel (Head of the Centre for Aid and Public Expenditure at ODI), welcomedeverybody to the meeting and introduced the panellists.

JonathanBeynon started by outlining the context from the European Commission perspective of:

• Rising commitments
• Increasing scrutiny from a wide range of actors
• Difficulty in collecting, disseminating and communicating evidence around GBS

He then discussed the characteristics of the Commissions MDGContract:

• Longer term commitments
• More predictable form of BS
• Focused on MDG results and targeted at stronger performers

Jonathan noted that these innovations in provision aredesigned to improve BS as a modality, in doing so the Commission is trying tomove away from a focus on GBS inputs towards focussing on GBS outputs andoutcomes for the MDG contract. The study (jointly authored by Andra Dusu) is an attempt to enhance theevidence base on the association between MDG performance and provision of GBS.

Jonathanhighlighted that we already know some things from the literature about the association:

• The MDGs are globally on track but with large national variations – the current discussion doesn’t include much about the role of aid instruments.
• In terms of BS evaluation – the OECD DAC evaluation is a benchmark with much on the intermediate impacts of BS but the problems of attribution / counterfactual still need addressing.
• The contested nature of the aid and growth literature demonstrates how hard it is to identify the impact of aid in general and also the number/variety of other factors. Again, very few studies differentiate between types of aid instrument.

GBS performance evidence iscurrently limited to individual countries and more anecdotal evidence. Thisstudy aims to be more systematic and comparative, asking the question: Do high BS recipients perform better on MDGindicators?

Jonathan then ran through someof the results:

• For ACP countries, high GBS recipients perform better on Primary Enrolment, Gender Parity in Enrolment and HDI indicators but not statistically significantly so for Child Mortality or Access to Improved Water Sources.
• For all countries, high GBS recipients outperform low GBS recipients for all MDG indicators.

Jonathan was keen to point outthat there are plenty of reasons to anticipate such an association, forexample, firstly GBS is targeted at low income countries whose low startingpoint might lead to better performance, secondly we would expect strengthenedand effective recipient government systems to have a positive association withboth MDG outcomes and with the eligibility criteria. Jonathan outlined how thestudy tried to control for potential influencing factors:

• Within low income countries, high GBS recipients performed better for all except Access to Improved Water.
• Within high CPIA countries, high GBS recipients performed better for all MDG indicators.
• Within aid dependent countries, high BS recipients perform better for all MDG indicators.

In each casestatistical significance of the relationship between GBS and MDG indicators is significantlyweakened and Water and Child Mortality are the least strong indicators.Jonathan outlined suggestions why this might be the case for the water MDGindicator. Suggestions included poor data; greater private provision; dominationof project inputs; less strong donor coordination.

For child mortality suggestions included limits to how muchimprovement we can see in a rate of change indicator and levels of spendingbelow the levels needed to see improvement. Jonathan also noted that thestrength of the association may be influenced by whether outcome or output indicators are used.

Jonathan concluded that studyfinds an association (not causality) which is boosted by controlling forsome of the other key factors. He also identified the need for complementary case studies to investigate the causality and look at why there isstill such variation within the high/lowGBS groups especially given thesmall share of aid currently given as BS.

Marcus thenasked the panellists to comment. Stephencommented that:

·This is still very much a black box – ‘whatever we believe about association and causality,it doesn’t tell us how it is doing it’.

·Stephen cautioned that in looking at MDGs as estimates of performance wewould expect that they would respond differently to inputs e.g. enrolment isnot a long term output and has a lead time very different from health.

·The study responds to the demands to justify aid and BS in particular

·The nature of BS has been changingover time.

·The performance of BS often depends on itsdesign.

·It is artificial to compare BS to othermodalities without looking at the complementarities between them in furtherdepth.

·Broad quantitative analysis and further in depthcase studies are needed to help explain the association between MDGs indicatorsand GBS.

·Some SBS would qualify as GBS in the studydefinitions.

·Design e.g. how SBS is earmarked (general vs.fully traceable) can determine whether SBS looks like GBS or like a big project.

·Does BS really get to grips with quality (themissing middle) between funds in at the top and outcomes at the bottom and focuson what happens in the middle to deliver services effectively.

Oliver then commented that:

·He had never had any difficulty in findingevidence that aid is effective so had no problem to accept the association but thoughtit is an association between aid andperformance indicators which is consistent with evidence that aid does contributeto growth and that aid is associated with improvements in human development.

·He was a lot more sceptical about whether ittells you anything about GBS

·Oliver didn’t agree that outcome indicatorsshould be used to evaluate aid modalities as GBS does not have a direct effecton MDG outcomes – Oliver asked what the channel/ mechanism is that links these two?

·Oliver pointed to the literature on aid and humandevelopment indicators, indicating that it explains how aid affects the level and pattern of public expenditure. Hesuggested that – conditional on the effectiveness of the provision ofgovernment services - that this may be relevant for explaining aid outcomes.

·Oliver stated that although there is a strongassociation between aid and the level of pro poor public expenditure, that spending in health, education and watersectors areas has a very weak relationshipwith outcomes because of inefficiencies and ineffectiveness of governmentexpenditure. Oliver indicated that the challenge is make the spending that isfinanced by that aid, beneficial. He suggested that this is where selectingappropriate modalities such as GBS may be important.

·For Oliver, trying to look for the broadassociation between GBS and MDGs will be very difficult – you have to look inthe middle and explain therelationship contained there.

Paul followed on by raising:

·The study compares averages of high and low BS recipient countries and it would be desirableto disaggregate the countries to see which of the high BS recipients have donewell / badly in order to examine the causation.

·Paul also added another plea to disaggregate by sectors.

·For Paul, the fact that donors behave selectivelyand give aid to countries they already trust and think that are effective atmaking aid funds work well – so BS isbeing given to countries making the most progress – makes the positiverelationship between GBS and MDG indicators unsurprising.

·Paul noted that when looking at the low qualitycountries in terms of the CIPA score, the high BS recipients still outperformthe low BS recipients – so it isn’t just that the already successful countries thatare getting lots of BS aid and the causation is going from the amount of BS tothe aid flows but also some appears to be going from the size of the aid flowsto the effectiveness of those flows – so there is two way causation going on.

Jasmine commented that

·Galvanising the discussion is hugely importantand it is complicated to look at outcomes and MDGs.

·Given the complexity much more research has to be done looking at how BS works in practiceand in the national context.

Marcusthen opened the discussion to questions from the floor. The questions included:

·Reiterating calls for more case studies andlinks to HLF4 / the latest DAC evaluations.

·Agreement on the need for further research intothe ‘missing middle’ including the political context and causality.

·The need for lagged variables, use of output indicatorssuch as immunisation rates, DHS survey data and general data constraints

·Awareness of the hybrid nature of aid supportincluding mixes of modalities, fungibility and GBS in aid dependent countries.

Jonathancommented that this isn’t the last word – the study is trying to fill a gap –additional questions and case studies will improve the way BS is provided.Given BS is still really quite small getting these results at all is quiteencouraging.

Marcus added that knowing whether this instrument producesresults that are value for money in this fiscal environment is incrediblyimportant.