Dr Martin Ravallion, Senior Research Manager, Development Research Group, World Bank
Kate Bird, ODI Research Fellow
Massimiliano Cali, ODI Research Officer
Andrew Shepherd, Director of Programmes, Rural Policy and Governance Group (RPGG), ODI
1) Remarkably little is known about the long term impacts of project aid to lagging poor areas. This paper is a technical assessment of the impacts of a large World Bank-financed rural development project in Southwest China that was initiated in 1995 and on which disbursements ended in 2001. The assessment is made in the context of China's remarkable record in aggregate poverty reduction (from poverty rates of 53% in 1980 to 8% in 2001) but with a persistent problem with geographic disparities - specifically for resource-poor inland lagging regions;
2) The World Bank-supported Southwest Poverty Reduction Project (SWP) was a classic integrated rural development project. It aimed to reduce poverty by targeting poor areas (Guangxi, Guizhou and Yunan with a total population of some 120m people) and investing in poor farm households and social services and rural infrastructure. The four key components of the project were income generating activities; off-farm employment; social services and infrastructure and institution building and poverty monitoring. The US$460m project funds were applied to the following components: agriculture (43%), infrastructure (17%), rural enterprise (11%), labour mobility (10%), education (9%) and health, institution building and project and poverty mapping (all < 5%);
3) The project did not tackle a number of key structural constraints to rural development in China. These include the restrictions on internal migration (the houkou system); the administrative allocation of land undermining the development of a land market and the high level of uninsured risk for the poor - resulting in a high level of precautionary savings;
4) The Government of China traditionally focuses upon village and small-holder investment programmes as the main direct intervention for fighting poverty. The project was implemented through a project office with links to government structures. The SWP project demonstrated a greater extent of integration across sectors, higher levels of participation with beneficiary groups, more resources and a much clearer focus on monitoring the impact of the intervention - compared with conventional government-driven approaches in poor area programmes. Government tends to regard the impact of a project as inevitable if the implementation process is correct;
5) The assessment was very detailed and involved monitoring closely (daily income and consumption diaries for individuals, households and communities with monitoring checks every two or three weeks) over a ten year period 1996 to 2006 on 2000 households in 100 villages within the project area and 2000 households in 100 villages out-with the project area. Cash consumption and income was augmented with imputed values for in-kind benefits. The fact that the baseline assessment was in 1996 - a year after the start of the project - reflected a failure to adequately communicate the project start date between government and the evaluation team;
6) Fundamental to evaluation is the recognition that project impact is the difference between the relevant outcome indicator with the project and that without it (essentially evaluation is a missing data problem - how to compare the actual out-turn with the counterfactual situation in which the project intervention did not take place);
7) Methodologically, impact evaluation should balance any selection bias between the treatment group (100 villages in the project area) and the comparison group outside the project area that is not a result of the project itself. For instance, local political economy and knowledge may result in the spill-over of benefits from the project areas into the comparison group. As long as any bias is additive and time-invariant, 'diff-in-diff' will work. However diff-in-diff hides the true impact when targeted areas have lower growth prospects than the comparison group - which is obviously a serious issue with projects that specifically target lagging areas;
8) During the disbursement period, from 1995 to 2000, the SWP project resulted in a sizeable but transient impact on income in the treatment area. Consumption was not significantly affected - due to very high levels of household saving. Analysis indicates that this saving was precautionary and there is no evidence that the project had changed the returns to savings and that the project area was entering a new growth trajectory (the 'big push' or virtuous cycle growth scenario). The project did not change agricultural productivity significantly and there was some evidence of asset value increases in assets such as livestock and housing. There was a large initial impact on headcount poverty, as measured by income. However, consumption changes were mainly observed in the middle-income groups. There was no sign of any significant impact on overall perceived standard of living or satisfaction indices. The evidence from self-perception assessments indicated that much too much weight is given to current living standards - compared with the project baseline scenario;
9) There is evidence of a quite significant (about 40%) displacement of non-project funds from the project area to non-project areas during the project implementation period. It is also possible that non-project villages also gained knowledge as a result of the project. These 'spill-over' effects would have blurred the impact of the project itself.
The discussion clustered around two sets of issues:Evaluation methods; and
Implication regarding types of project aid.
Questions were raised about:What can be concluded about on the use of qualitative methods alongside quantitative assessments?
Do income and consumption diaries yield systematic under or over-reporting?
Could the project result in more benign long-term impacts where, for instance, savings function to smooth erratic fluctuations in income?
Could trade between project and non-project areas cloud the impact of the project?
If the results of project impacts from qualitative self-assessments were from the same households as those painstakingly filling in income and consumption diaries over extended periods - the recall problems of self-assessment may be even bigger than this study suggests?
Was the high level of saving simply reflecting the lumpy nature of assets in the project area?
Why were community organisations excluded from the evaluation process?
In response the Speaker noted:
Generally a mixed methods approach is appropriate in evaluations, what this exercise has revealed is the dangers of using qualitative methods in isolation. There is no obvious role for civil society organisations in an evaluation of this nature. Generally the use of randomisation methods is not politically feasible - and may be ethically questionable. The stimulation to trade, through increasing demand in the project area, may equalise the benefits between the project and non-project areas. The key tool to identifying good comparison areas is a good quality poverty map. The lumpy nature of investments may, indeed, account for the very high levels of savings observed in this evaluation.
Implications for this type of project aid
Questions were raised about:The extent to which the lack of impact generated by the SWP project reflected its 'enclave' status - with a project office that was separate from government structures?
Do these results not indicate that project aid of this type works well in fast-growing local economies, but has a much smaller marginal impact in slower-growing areas?
Can these findings for a very large project inform evaluations of the much more common, smaller, projects?
Is it possible to move beyond the Sachs-Easterly dichotomy about whether aid works or not and instead focus upon a more nuanced analysis that recognises the importance of local institutions in making external interventions work - or not?
In response the Speaker noted:
The high standard errors and high levels of variance imply very differential project impacts in each local area - which may well support the case that aid effectiveness can be attributed to the quality of local institutions. A key blockage in rural China is the policy framework that prevents the development of a land market - to avoid devaluing the main non-labour asset of poor rural people. Also the population registration system, by reducing internal migration, has a significantly negative impact on poor rural areas. Reform in these two areas would have a much greater impact than external development assistance.
Remarkably little is known about the long-term impacts of project aid to lagging poor areas. The paper re-visits a large World Bank-financed rural development project in southwest China, 10 years after it began and four years after disbursements ended. Survey data were collected on 2,000 households (in both project and non-project villages) who had first been surveyed at project commencement in 1995. A double-difference estimator is used to assess impacts after balancing the initial covariates of placement. We find sizeable gains in mean income during the disbursement period, but much smaller long-term impacts, although there are signs of a lasting reduction in extreme poverty. We also highlight the difficulties in assessing long-term impacts, given spillover effects through spending displacement and knowledge diffusion. Rapid appraisal methods do not reveal any impacts, although recall biases may well be the reason.
At this ODI event, Dr Martin Ravallion, Senior Research Manager in the World Bank's Development Research Group, will present his recent paper which re-visits a large World Bank-financed rural development project in southwest China, ten years after it began and four years after disbursements ended.
Findings include sizeable gains in mean income during the disbursement period compared to much smaller longer-term impacts (although some signs of a lasting reduction in extreme poverty were evident). The paper also covers the difficulties of assessing long-term impacts.